TODO: Enhancing LLM Alignment with Ternary Preferences
Authors: Yuxiang Guo
Lu Yin
Bo Jiang
Jiaqi Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: LLM
Preference alignment
Ternary Preference
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Aligning large language models (LLMs) with human intent is critical for enhancing their performance across a variety of tasks. Standard alignment techniques, such as Direct Preference Optimization (DPO), often rely on the binary Bradley-Terry (BT) model, which can struggle to capture the complexities of human preferences—particularly in the presence of noisy or inconsistent labels and frequent ties. To address these limitations, we introduce the Tie-rank Oriented Bradley-Terry model (TOBT), an extension of the BT model that explicitly incorporates ties, enabling more nuanced preference representation. Building on this, we propose Tie-rank Oriented Direct Preference Optimization (TODO), a novel alignment algorithm that leverages TOBT's ternary ranking system to improve preference alignment. In evaluations on Mistral-7B and Llama 3-8B models, TODO consistently outperforms DPO in modeling preferences across both in-distribution and out-of-distribution datasets. Additional assessments using MT Bench and benchmarks such as Piqa, ARC-c, and MMLU further demonstrate TODO's superior alignment performance. Notably, TODO also shows strong results in binary preference alignment, highlighting its versatility and potential for broader integration into LLM alignment. The code for TODO is made publicly available.

Published as a conference paper at ICLR 2025
TODO: ENHANCING LLM ALIGNMENT
WITH
TERNARY PREFERENCES
Yuxiang Guo Lu Yin 13t Bo Jiang"
Jiaqi Zhang" 't University of Surrey
'Meituan Inc. Beihann University
{irisg, aaaaangoo@uuuaae edu. on 1. l.yinesurrey. ac.uk uk zhangjiaq 390me w
ABSTRACT
Aligning large language models (LLMs) with human intent is critical for enhanc- ing their performance across a variety of tasks. Standard alignment techniques, such as Direct Preference Optimization (DPO), often rely on the binary Bradley- Terry (BT) model, which can struggle to capture the complexities of human pref- crences-pprtieullle
in the presence of noisy ties. To address these limitations, we introduce the Tie-rank Oriented Bradley- Terry model (TOBT), an extension of the BT model that explicitly incorporates ties, enabling more nuanced preference representation. Building on this, we pro- pose Tie-rank Oriented Direct Preference Optimization (TODO), novel align- ment algorithm that leverages TOBT's ternrr ranking system to improve pref- erence alignment. In evaluations on Mistral-7B and Llama 3-8B models, TODO consistently outperforms DPO in modeling preferences across both in-distribution and out-of-distribution datasets. Additional assessments using MT Bench and benchmarks such as Piqa, ARC-c, and MMLU further demonstrate TODO's su- perior alignment performance. Notably, TODO also shows strong results binary preference alignment, highlighting its versatility and potential for broader integra- tion into LLM alignment. The implementation details and datasets can be found in htt / thub. com/ XXa reaaressTOoo /TOD
inconsistent labels and frequent
INTRODUCTION
Large language models (LLMs) demonstrate remarkable potential in various tasks (Huang et al,,
1 Hendrycks et al., 2021 Shi al. 2023), with performance gains linked to better alignment
with human intent (Mishra et al. 2022 Christiano et al., 2023 Wu et al., 2023). The alignment
process typically involves two stages: Supervised
Tuning (SFT) establish instruction follow-
ing abilities (Thoppilan et al. 2022; Sanh et al.] [022] Mishra et al. 2022), followed by preference
fine-tuning to refine the model's 's alignment with human preferences Ziegllrr et al. 2020 Christiano
et al., 2023). This stage typically employs either reinforcement learning based (Schulman
iet al. OpenAl 2023 Ramamurthy et al. 2023) or RL-free methods (Rafailov et al.) Azar et al.)
Sacidi et al.) Meng et al.) 2024). both leveraging the preference datasets. Effective alignment is enhanced by the diversity of training data, enabling LLMs to accurately learn from high-quality
pairwise responses (Cui al. 2023 Song al. 2024 Sacidi al.).
Current alignment methods relying on the Bradley-Terry (BT) (Bradley & Terry
1952) model con-
sider only two preference rankings: preference and dis tisffrreeee,,e which restricts the diversity of learnable information. A notable challenge is the inconsistent quality of the pairwise preference data, often showing minimal discernible differences (Nvidia et al., 2024 Amini et al.) 2024). Table I shows a sample from the Ultrafeedback- binaried datase (Tunstall et al.). which is commonly used in preference alignment procedures (Tunstall et al.) [Hong et al.). In this example, both sponses have identical quality score of 8.5 from GPT- evaluations (OpenAl 2023), differing only
Work done during the internship at Meituan Inc. Work done during the research at Meituan Inc. Corresponding author.
hta w/tp////r/g/grrr ing www...geo.c..ooooo.oo...o....... ace..cc /a aset http://....o.ooaooooo///////a//a
xed

---

Published as a conference paper at ICLR 2025
31
y2
DPO
Signoid fonction
Max(n(y))
Reward diffiresce betwees two responses
LLM Proniet x
Responsc
Preference collection
Maxiort all
Uncetaint profirener caete
Response y2
The Bradley-Tery IBT) model
31
V
TODO
Our TORT model
Figure 1: Comparison of DPO and TODO. DPO relies on the BT model, which is only capable of handling binary preferences. When responses are tied, it either learns incorrect preference infor- mation or discards tied data, preventing learning from such data. In contrast, the proposed TOBT model can directly model ternary preferences. Based on this, TODO can learn more information from tied data and exhibits better robustness against potential noise in binary preference data.
in narrative sequence and text format. In practice, we observe considerable amount of tie data in common preference datasets and chat arenas judged by humans, as detailed in Appendix A. These ties encompass variety of information, necessitating a more nuanced analysis. However, existing preference optimization techniques, such as Direct Preference Optimization (DPO) (Rafailov et al.), are constrained by their reliance on the binary BT model and struggle to effectively manage tie re- lations. How to learn useful information from tie data and achieve nuanced preference modeling in the alignment process remains to be explored, which is the goal of this paper.
Prompt
Q: Ninette de Valois was involved professionally in what activity P(nA:
Ninette de Valois was professionally involved in ballet dancer, choreugrupher, and founder of the Royal Ballet in London. ynConfidence: 95% (score: 8.5)
RI
Ninette de Valois was involved professionally in the field of ballet. She was a choroogra- pher, duncer, and founder of the Royal Ballet in London. (score: 8.5)
R2
Table One pair of responses in Ultrafeedback -innaried Dataset which have identical quality scores evaluated by GPT-4. R1 is treated the preferred response and R2 is treated as the dispreferred one.
Our primary contributions can be unfolded as:
We enhance the existing human preference alignment process by incorporating a "tie" ranking, transcending the traditional binary rankings as depicted in Figure [ We first extend the BT model into the Tic-rank Oriented BT (TOBT) model. The TOBT model incorporates the concept of prefer- ence uncertainty, allowing for the representation of ties alongside "prefer" and "disprefer" rankings. This innovation enables more comprehensive handling of preference relations.
Building on the TOBT model, we introduce the Tie-rank Oriented Direct Preference Optimization (TODO) algorithm. TODO is designed to accommodate ternary ranking relations, offering a nuanced approach to preference alignment. By integrating the tie relation, TODO ca- pable of learning from broader spectrum of preference information, enhancing the adaptability and accuracy of L.LMs.
We use Mistral-7B and Llama 3-8B to conduct experimental validation. First, we evaluate the effectiveness of DPO and TODO in terms of preference modeling accuracy. Our evaluation spans both in-distribution dataset, drawn from the same source as the training data, and oui-of-distribution dataset, notably the Reward Bench (Lambert et al.). Results indicate superior preference modeling by TODO. Additional assessments on MT Bench (Zheng et al., 2023) and other popular bench- marks such Piqa (Bisk et al. 2019). ARC-c, ARC-e (Clark et al. 2018). Hellaswag (Zellers) al. 2019), MMLU (Hendrycks al. (20T) and Winogrande (Sakaguchi et al. 2019) confirms TODO's 's enhanced alignment capabilities. Finally, provide intuitive analysis highlighting TODO's advantages over DPO in two dimensions: enhanced granularity in preference modeling and increased diversity in the acquired information.
TODO can also be directly applied in binary preference alignment process, outperforming DPO with standard binary preference datasets. Furthermore, the proposed TOBT model can be utilized not only in offline policies like DPO but also can be integrated into other online policies or used to train reward model.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORKS
Preference alignment LLMs. Current methods for aligning preferences in LLMs often utilize the BT model (Bradley & Terry] 1952) and the Plackett-Luce ranking model (Plackett 1975) to capture human preferences. RL-based approaches, such as Reinforcement Learning from Human Feedback (RLHF, (Schulman et al.)), require reward model assess generated responses. which is typically trained on pairwise preference data gathered from crowdworkers (Wang et al., (222)) or by utilizing another LLM as an evaluative judge (Bai et al. 2022 Cui et al.) 2023). RL-free algorithms, such as Direct Preference Optimization (DPO) (Rafailov et al.) and its variants (Azar et al. Ethayarajh 2024). can directly optimize human preferences by introducing an implicit reward. DPO offers stable and computationally lightweight method to align with human intent. To address the potential overfitting problem DPO, Identity Preference Optimization (IPO) (Azar et al.) and Reward-aware Preference (RPO) (Nvidia et al.) (024) has been proposed by introducing differences between pairwise responses. Furthermore, Kahneman- Tversky Optimization (KTO) (Ethayarajh) al. 2024), was proposed by directly maximizing the utility of generations instead of maximizing the fog-likelihood of preferences, and Hong et al.] introduced a reference model-free monolithic odds ratio preference optimization algorithm. Some concurrent works attempt to introduce intrinsic knowledge constraints into preference alignment, either by on-the-fly fine-tuning LLMs to obtain relative qualities (Yu et al., 2024) by defining reward distance (Nvidia al., 2024 Amini et al. 2024). These studies do not utilize tied preference to refine alignment, whereas our approach has the potential be integrated with them as discussed in Section 7
Ternary preference models. Ties frequently occur in ranked data, such as sports and examinations. For example, soccer result can be classified simply well-known BT model, which can be derived from the order statistics of the exponential distribution, cannot handle ties. Rao & Kupper (upper([196]] (1967) corrected the BT model by assuming that small probability difference values would be declared ties. Kuk 1995) applied this approach to football. [Dvidsoon (970)) provided an ad hoc correction the BT model for ties. [Dewart & Gillard (2018) applied the BT model to cricket, where draws occur but do not depend strongly on team strengths. |Baker & Scarf (2020) used discrete distributions, principally the geometric distribution, to obtain a modified BT model that allows for tie ranks. These studies lay robust theoretical foundation for addressing ties in ranking results, facilitating the connection between LLM alignment and preference data.
home win, an away win, tie. The
PRELIMINARIES
3.1 THE SRADLEY-TERRE MODEL
19 1952 model is probability model for the outcome of pairwise comparisons between instances, teams, or objects. estimates the probability that the rank order is true, where the symbol > represents preference or rank relation, such as instance i being preferred over j. or beating j. depending on the application.
The Bradley-Tery Terry (BT) Bradley & Terry
The computation of BT model can be represented by Equation where the positive strength owned by two competitors are denoted by X and A2 respectively, and '12 represents the probability that the first competitor obtains higher rank than the second one in comparison.
A1
(1)
X +
Define d12 In A1 In X2, which represents the strength difference of two competitors in logarith- can be written into the form of Equation
mic form. Following |Rao & Kupper (1967), Equation 2. where sech(-) denotes the hyperbolic secant function. This relation shows that the preference probability F12 depends only on d12.
[
sech (y/2())))))) = / sech2 (y/2) dy di2
F12 / -(m -n
3.2 HUMAN PREFERENCE MODELING
Following the BT model, the human preference distribution p" is formulated in Equation 3, where r* is some latent reward model, y* (xx y) denotes the reward for response y given an input prompt
3

---

Published as a conference paper at ICLR 2025
x, and (-) : is the Sigmoid function. The variables æ, V1 and y/2 drawn from preference dataset D = {a', vi. 12. where Y11 is the preferred response and 1/2 is the dispreferred one. The term exp(r" (z, y)) signifies the strength À of responses following Equation [ acknowledging that the reward could be negative,
exp(r"(z,
(vv 32/z)) exp(r"(x; vn)) (x, a("" (2, (a, (zz y2)).
Subsequently, a reward model that mirrors human preferences can be trained using the method of maximum likelihood estimation (Schulman al.).
max ((r.n log (r - (o( ].
(4)
3.3 DIRECT PREFERENCE OPTIMIZATION
In RLHF (Schulman et al.). the goal is to maximize the expectation of rewards under the KL diver- gence constraint,
Te Ta ())
max E D.yro (T) v)l - B log
(5)
(z)()
The optimal solution this problem satisfies the following relationship (Rafallov al.):
r(æ, y) = =logg-
+ 3 log ((x),
(6)
where Z(x) = Eg (ellax) exp (2, v)) only depends on prompt æ.
By integrating this relation into Equation 4 the loss function of DPO can be expressed as shown in Equation 7 which incapable of addressing tied preference data.
(o(
To (322222)
LDPO (To) Taet) E (2. log a (Blog
(
log
Tee ( (x)
Jee (32 (x)
TIE-RANK ORIENTED DIRECT PREFERENCE OPTIMIZATION
To handle ties preferences modeling, we introduce buffer in the integral interval Equation 2 which is inspired by Rao & Kupper (967).. We call the new preference model Tie-rank Oriented BT (TOBT) model. Based on this model, we propose a novel preference alignment algorithm, Tie-rank Oriented Direct Preference Optimization (TODO).
4.1 TIE-RANK ORIENTED BT MODEL
As shown in Equation 2 d12 represents the preference difference between two instances, and d12 0 means the first instance is preferred. To handle ties, we impose higher requirements on the comparison by introducing a positive number and then the preferred relation is determined by d12 > That is, the ranking probability 712 becomes Equation
g
712
sech ( dy
(8)
(In A ln
Then, the probability that two instances share tie relation is 121, which is denoted by T(12), and can expressed in Equation
(In A A1-lo Aa)+a
T(12)
sech sech2 dy
(9)
-In A2)-
Intuitively, the parameter encapsulates the inherent uncertainty and noise in the strength values A1 and A2 which are ubiquitous in LLM alignment. Preference datasets for alignment often rely on human labeling (Chiang et al., 2024), LLM-as-judge assessments (Cui et al. 2023). reward model scoring (Nvidia et al., 2024), introducing label noise due to inconsistencies among human annota- tors the inherent variability of LLMs in approximating human preferences. Accounting for this

---

Published as a conference paper at ICLR 2025
uncertainty and noise in responses evaluation likely contributes to TODO's improved performance on binary preference datasets, as detailed Section
Following Equations 8 and the TOBT model can be represented by Equations 10
and
where
= exp(a) and Φ I. The detailed derivation process is provided in Appendix A.2
F12 = dy + Φ62
(10)
2122( 1) (A1 + ( + X2)
T(22)
(11)
4.2 OBJECTIVE FUNCTION OF TODO
Following Equations 10 and the tie-rank oriented human preference distribution p" can be
expressed by Equation 12 and Equation
(s, p" (vn > yala) = exp(r((x.v1)) + (xxp(r"
(12)
exp(r" (z. exp(r (I, (6
p (y1 = yu(x)
(13)
(xp(r"(xx + expp(r )2)) exp(r" (xx, v2)) + exp(r* un))
Equation 12 represents the possibility of treating y1 as the preferred response and 1/2 as the dispre-
ferred response in pairwise data based the TOBT model. Following Equation [2] the objective CRODO can be written as Equation 14 where μ = ra (2 3/1 - re (z. V22 represents the reward dif- ference of two responses y1 and 1/2- Because the ((x) in implicit reward ro(x:, y) only depends
J, the difference results of μ can be equivalently expressed as μ = 3 log Za( (a)
Tarly [z] log Ka ==/(1=(=))
The superscript in LTODO denotes that this formulation is the objective of TODO when the sponses a pair exhibit a clear preference, rather than being tied. The detailed derivation process is provided in Appendix A..
Crono Fret) = ((z.pr. log o(u ax]
= -B...n log o Blog (o((112))
Boog bogo(Blog Trc((y1lx))
To (V(22)
a))
(14)
Equation 13 represents the possibility of treating pairwise responses y1 and y/ as a tied pair based on the TOBT model. Following Equation 13 the objective CTODO be written as Equation 15 The superscript in LTODO signifies that this represents TODO's objective when the pair is tied. The detailed derivation process provided in Appendix A.5
exp(2a) -
CToDO Tha) - - -B log (1 exp(u a))(1 + exp( a))
(15)
Given preference dataset (x', vi. v I') D, the indicator Il' is determined by Equation 16 Specif- ically, I' = 0 indicates clear preference or quality difference between the two responses to the same prompt z, while I represents that two responses yf and y are tied. Then, the final loss CTODO of TODO can be represented by Equation 17
1. v v2.
r
(16)
Di
v2.
LTODO (1 T) T
(17)
Compared to DPO, TODO introduces margin to shift the decision boundary towards for paired data with a clear preference, while also accommodating ties. In contrast, methods based on the BT model struggle to handle tie data effectively.
5

---

Published as a conference paper at ICLR 2025
4.3 THE EFFECT OF DIFFERENT PREFERENCE RELATIONS ON TODO UPDATES
To elucidate the dynamics of TODO parameter updates, we introduce a gradient-based analysis that distinguishes between scenarios where pairwise responses show a clear preference or are tied. The comprehensive derivation is detailed in Appendices An6land
For the gradient update (8) for pairwise data with a clear preference, the introduction of positive small value results in more substantial weight adjustments when the reward difference misestimated compared to DPO. This refinement mitigates noise from narrow reward margins, allowing TODO to more effectively learn distinct preferences by concurrently enhancing the likeli- hood of the favored response y11 and diminishing that of the unfavored response y/-
Vo (oo - -E(2.11
Ba a
Vo (og((())))))) V Vo (18)
highor weight than DPO
For the gradient update 19 for pairwise tie data, G(p) [1-expl pi-expp--+aar ( (Txxp(((ttttxxt is mono- tonically decreasing with respect to and G(O) = O. When 0, two responses obtain the same rewards, and DPO continues to update policy models as per Equation / which shifts the distribution to reduce the likelihood of the "dispreferred" response y/, potentially discarding valuable informa- tion. contrast, TODO refrains from updating any parameters to maintain the consistent preference alignment of the tied responses.
When the estimated reward difference for tied responses is μ > O, suggesting y1 has a higher reward than y2. TODO's gradient update strategy will reduce the likelihood of yı and increase that of y2- Conversely, if μ < O, the update will elevate the likelihood of y/1 and lower that of y/, ensuring the preference consistency between the tied responses is preserved.
VoLTODO meet) - - B(.. G(u) log(m (y1 (x)) G(- log(m (v2l=)))
(19)
part
part
5 EXPERIMENTAL SETTINGS
5.1 MODELS AND DATASETS
Models. We select two different series of models, Mistral-7B (Jiang et al. 2023 and Llama 3-8B (AI@ Meta 2024), as our experimental backbone models. We select zephyr-sft-fu TTnstaal al.) as the supervised fine-tuming (SFT) version of Mistral model and Ilama3-86 -a---bb---tittii ulltrcttaacht"" as the SFT version model of Llama 3 model. Both zephyr-sft- and samma---b-ftt-ltraee are fine-tuned on Ulrrccat-2000k (Ding et al. 2023) dataset.
Training datasets. For the datasets used the preference alignment process, we construct 20k-size datasets with different tie data proportions from Ultrafeedback (Cui et al. 2023). Responses sharing the same quality score are classified as tied. The quality score for each response a weighted score across multiple assessment metrics, taking into account helpfulness, truthfulness, verbalized calibration and honesty. Each sampled dataset exhibits a tie data ratio that varies within the set {0, 0, 0.2, 0.3). Other details of the training sets are shown in Appendix A.8
Evaluation benchmarks. We first compare the effectiveness of preference modeling ability be- tween DPO and TODO. To this end, we curate an in-distribution test set containing 1500 non-tied samples and select the Reward Bench (Lambert et al.) as an out-of-distribution dataset. Subse- quently, we select a suite of well-established benchmark datasets evaluate the models compre- hensively: 1) MT Bench (Zheng et al.] 2023), which contains open-ended questions designed to assess the multi-turn conversational capabilities and the ability follow instructions of LLMs. 2) A diverse set of benchmarks containing Piqa (Bisk et al. 2019 ARC-c, ARC-e (Clark et al. 2018) Hellaswag (Zellers al. 2019), MMLU (Hendrycks al. 202T) and Winogrande (Sakaguchi al., 2019), which collectively evaluate the language comprehension and reasoning faculties of L.LMs.
attps: httos://huggiingece.co wwwpe//mmmnentneeeee//emm.meeeem haw.mmeen--nnnooooeeeeeeeeee
full
httpat//hogging http://ggggoooooooooo.o// kaykk 1

---

Published as a conference paper at ICLR 2025
Unma 3-10-577
0.0.2
0.72
0.72
Rarlo
1est 560
TODO Sed
. DPO (Reward Bancht
TODO (Roward Bench)
Accuracy Mistral and Llama 3 models aligned with DPO and TODO on non tie prefer- ence test set and Reward Bench. The X-axis denotes the proportion of tie data mixed in train set.
Figure 2:
5.2 TRAINING SETTINGS
All comparative results are derived from training each model for epochs on their respective train- ing datasets. As demonstrated in the analysis presented in Appendix A..3 the model performance is relatively insensitive to variations in a within reasonable range of a E (0.1, 0.8), we set a = 0.5 in TODO. Other hyperparameters are shown Appendix A.9 where we adopt the settings from previ- ous works (Sacidi et al.; Meng et al. 2024). We ensure the consistency of training hyperparameters among experiments for fair comparison.
5.3 EVALUATION SETTINGS
Accuracy of preference modeling. In assessing the efficacy of models fine-tuned with DPO and TODO for preference modeling, we employ prediction accuracy as the primary evaluation metric. For each pair of data where the preference rank y1 favored over 1/2, we calculate the predicted
probabilities for each preference rank for pair data, adhering to Equations 12 and 13 A prediction
deemed accurate if the model assigns the highest probability to the scenario where 1/1 is preferred
over y/2 across all possible ranks.
GPT based scoring in MT Bench. For the evaluation in MT Bench, we use gpt-4-turbo-2024--0000 (OpenAT 2024b) score generated results.
Accuracy of other benchmarks. For the evaluation Piqa, ARC-c, ARC-e, Hellaswag, MMLU and Winogrande, we Opencompass (Contributors) 2023) assess final results, details of prompt
templates and evaluation metrics can be found in Appendix A. 10
6 RESULTS AND ANALYSIS
6.1 TODO IMPROVES HUMAN PREFERENCE MODELING WITH TIE DATA
In this section, we assess the preference modeling capabilities of models trained with DPO and TODO across both in-distribution and out-of-distribution datasets, incorporating varying proportions of tie data in the training regimen. The in-distribution assessment is based on the above mentioned test set, while the distribbtiti assessment utilizes the Reward Bench.
Figure 2 iillustrates the accuracy results of Mistral and Llama 3 models aligned with DPO and TODO on the test set and Reward Bench, using training sets with varying proportions of tie data. Reward Bench contains pairwise responses divided into Chat, Chat hard, Safety, Reasoning and Prior pref- erence data categories. We compute the average accuracy score of all categories to represent the final performance Reward Bench. The detailed scores in each category of two series models Reward Bench are provided in Appendix(A.12 A.
We observe that both Mistral and Llama 3 models aligned with TODO generally achieve better per- formance than those aligned with DPO on both in-distribution and oftttf--iistrbuution data. Overall, when directly modeling human preferences mixing tie data, DPO often leads to sub-optimal results. TODO addresses this issue with more nuanced preference modeling. Experimental results demon- strate the effectiveness of the combinatorial optimization objectives in TODO.

---

Published as a conference paper at ICLR 2025
Marr 18-5mT+B10 Dr with rato ae ceta
Jama 3 32:-STT-EN Ein with of 3-45-5ry
Hitma 18-5rr
MT Sandh
MT Bench
0.20
C21
Ratio 0.0
Rablo D.1
Rabio 0.2
Rotio 0.3
Ratio 0.0
Ratio 0.1
Rario 0.2
Ratio 0.3
Tie Data Ratio
The Data Pattin
(a) Results on Mistral-7B
(b) Results on Llama 3-8B
Figure 3: MT Bench results of Mistral-7B and Llama 3-8B. The models are aligned with DPO and TODO using datasets with varying ratios of tie data.
6.2 TODO ENHANCES ALIGNMENT IN LLMS
In this section, we evaluate the effectiveness of DPO and TODO across different benchmarks to comprehensively demonstrate the alignment capabilities of LLMs.
As illustrated in Figures 3al and 3b the scores the MT Bench for models aligned with DPO and TODO reveal that TODO consistently outperforms DPO across all training sets for both the Mistral and Llama models. Specifically, TODO achieves peak performance when using the binary preference dataset for the Mistral models and incorporating a 20% tie data ratio for the Llama models, respectively.
Table 2 and Table respectively show the results on Piqa, ARC-c, ARC-e, Hellaswag, MMLU and Winogrande benchmarks of the Mistral and Llama 3 models, where highlight the best perfor- mance in underline, and mark the better performance between DPO and TODO with different ratio of tie data bold. As demonstrated in these tables, TODO achieves better performance than DPO across all train sets on both two models, and obtain the best average performance when mixing 20% and 30% of tie data Mistral and Llama 3 serials models, respectively.
Optimal proportion of tie data enhances alignment. Our experimental findings across various benchmarks underscore the critical role and efficacy of TODO in managing non-binary preference data. Notably, optimal performance on the MT Bench and six additional benchmarks is consistently achieved when a specific ratio of tie data is integrated with TODO. This observation indicates that the strategic incorporation of tie data, in conjunction with TODO, can markedly enhance the alignment capabilities of LLMs.
Method
Tie Data Ratio
Piqa
ARC-e
ARC-e
MMLU
Hellaswag
Winogrande
Average
SFT
80.09
51.33
74.97
60.26
75.69
73.64
69.33
DPO
77.15
49.87
69.56
59.44
73.55
67.88
66.24
TODO
81.01
54.59
78.44
60.16
79.01
72.93
71.02
DPO
0.1
79.87
53.82
77.21
60.07
79.04
73.48
70.58
TODO
80.25
$4.68
77.76
60.46
79.27
73.95
71.06
DPO
80.25
79.08
TODO
0.2
81.07
55.11
78.52
59.40
79.14
73,80
71.17
DPO
0.3
75.73
48.15
68. 84
59.41
71.97
69.30
65.57
TODO
77.37
74.36
67.37
Table 2: Results of Mistral-7B aligned with different methods trained with various ratios of tie data.
TODO's superiority binary preference alignment. In addition to enhancing performance by incorporating tie data into the training regimen, TODO can also be effectively employed in scenarios involving purely binary preference alignment. The experimental outcomes, as depicted in Figure for the MT Bench and Tables 2 and 3 for other benchmarks, consistently indicate that models fine- tuned with TODO outperform their counterparts optimized with DPO, even in the absence of any tie data.
8

---

Published as a conference paper at ICLR 2025
Method
Tie Data Ratio
Piqa
ARC-e
ARC-e
MMLU
Hellaswag
Winogrande
Average
SFT
79.65
55.02
78.65
63.89
76.45
72.69
71.06
79.27
76.58
79.05
71.74
70.75
TODO
0.0
79.43
55.45
76.79
63.45
79.27
71.82
71.03
DPO
0.1
79.65
54,68
75.48
63.46
79.08
71.74
70.68
TODO
DPO
79.16
60
78.72
TODO
0.2
79.38
56.22
77.08
63.07
79.04
71.11
70.98
DPO
0.3
79.05
55.54
76.11
63.34
78.17
70.88
70.52
TODO
79.33
77.76
78.83
71.03
Table 3: Results of Llama 3-8B aligned with different methods with various ratios of tie data.
6.3 KEY FACTORS OF TODO HANDLING TIE DATA AND OBTAINING BETTER PERFORMANCE
In this section, we analyze the limitations of DPO and the advantages of TODO in handling tie data. We highlight two pivotal factors that TODO can enhance alignment: nuanced preference modeling and enriched diversity of information learned.
TODO refines DPO by accurately handling tied data. We consider the scenario where responses of equivalent quality are incorrectly labeled as binary preference data, as illustrated in Table | As stated in Section 4.2 represents the implicit reward margin between two responses during the training. For the i-th pair of responses, vi and y which are essentially tied, DPO assigns a small value n². Despite the trivial value of p², DPO unilaterally steers the policy model update by increasing the likelihood of one tied response vi. while concurrently decreasing the likelihood of the other v². This biased update is consequence of the binary BT model, following the gradient computation:
VoLDPO = -E().u1
(a(--))
Volog( (x)) - V log(T(val>))]
(20)
always positive
The gradient update direction shown in Equation 20 is decided by the Sigmoid function and positive 3 value, which makes DPO always update policy model in one direction. Such a gradient computation, when applied tied data, can result erroneous updates and introduce noise into the binary preference modeling process, potentially culminating in a suboptimal policy. Both DPO and TODO assign small reward margin for tied pair responses, as compared to utilizing data with clear preference distinction. However, TODO employs combinatorial optimization objectives for pairwise responses accounting for the uncertainty of preference difference, and effectively rectifying the is- sues DPO faces with tied data. This changes the update direction depending on the exact preference relation, ensuring preference learning consistency between two tied responses. Concurrently, the gradient computation for con-tied responses facilitates the learning of preferred or dispreferred rela- tions. This distinction is crucial for effective preference modeling and can learn "prefer", "tie" and "disprefer" ranks. The exact reward margin changes of two models aligned with DPO and TODO using different ratios of tie data are provided in Appendix A.1l to support our analysis.
Learning from diverse tied responses improves the performance.
Both DPO and TODO are adept ranking pairwise data with clear preference distinctions, dis- tinguishing between "preferred" and "dispreferred" responses. However, when responses exhibit no clear preference or quality disparity, DPO's optimization strategy compels the policy model to skew towards one response at the expense of the other, result- ing in unnecessary information loss for the unfavored response.
Ultrafeedback Ultrafeedback Chatarena Chatarena (tic ratio (tic ratio 0.2)
Methods
(tie ratio 0) (tie ratio 0.17)
DPO
76.40
75.00
77.47
76.33
KTO
73.80
74.27
74.80
74.80
SimPO
77.80
76.35
76.27
76.33
ODPO
76.78
/
TODO
77.20
76.40
76.80
78.47
Table 4: Test-set accuracy Mistral-7B aligned with different methods and datasets.

---

Published as a conference paper at ICLR 2025
TODO mitigates this by incorporating tie rank and novel optimization objective, enabling the policy model to evolve in accordance with the consistent preference trends of both responses. This mechanism allows both the content and format of the two responses be learned simultancously, enabling the model to capture more diverse information from the same amount of data.
6.4 COMPARISON AGAINST OTHER BINARY MODEL BASED APPROACHES.
To comprchensively assess the generalizability and efficacy of TODO, we conduct an exhaus- tive comparative analysis against several prominent binary alignment techniques, including DPO (Rafailov et al.), KTO (Ethayarajh et al. 2024). SimPO (Meng al. 2024), and ODPO (Amini et al.) 2024). Our evaluation transcends the limitations of single dataset by leveraging both the Ultra- feedback dataset and the diverse, human-labeled Chatarena (ims) 2024 dataset, which encompasses multiple language pairs. Our experimental results Mistral-7B, encompassing test set accuracies and MT Bench scores, are provided Tables 4 and 5 In both datasets, TODO consistently outper- forms other baselines when tic data is included. highlighting its advantages. Additionally, TODO with tie data surpasses methods without tie data most cases, demonstrating the effectiveness of incorporating tie data. The experimental settings are provided in Appendix A.13
7 DISCUSSION
Potential integration of TOBT into other methods. For RLHF- based methods, a straightforward integration way is to train a ternary reward model instead of the binary ones. The objective functions for the new reward model are com- posed of (12) and (T)). To this end, a ternary-laeeled preference dataset is needed, which is not dif- ficult to obtain based on common annotation methods.
Ultrafeedback Ultrafeedback Chatarena Chatarena
Methods
(tie ratio 0)
(tie ratio 0.2)
(tie ratio 0) (tie ratio 0.17)
DPO
5.94
5.55
5.50
5.71
KTO
5.63
5.61
5.47
5.53
SimPO
5.56
5.69
5.21
5.28
ODPO
5.95
TODO
5.85
5.96
5.52
5.83
Table 5: MT Bench scores of Mistral-7B aligned with different methods and datasets.
For DPO-like methods, the inte- gration varies by case. We take ODPO (Amini et al.) 2024) as an example which adopts loss function in the form of - log(o the ground-truth scores of the two inputs. By applying (15). we can obtain a new loss incorporating the derivation of which should be similar to TODO.
A,)), where A, introduces a preference margin depending on
tie data log
exp(22,)-1
[1
Extension for other tic-aware preference models. The model's reliance on - rather than and r2 individually crucial, because the reward includes an untractable variable Z(z) (Equation 6) which can be climinated in 72. Some models (Glenn & David 1960) Davidson 1970) share this characteristic, offering potential for adaptation in [.L.M alignments. However, their effectiveness in this context has yet to be explored.
8 CONCLUSION AND FUTURE WORK
This study illuminates limitation in LLM preference alignment: the inability of binary models like DPO to resolve ties in preference data. To overcome this, we integrate tie ranking system into preference modeling, refine the BT model into the more robust TOBT model, and introduce the TODO algorithm. Our experimental results demonstrate that TODO consistently outperforms DPO across a range of evaluations. The success of TODO stems from its nuanced handling of ties and the robust TOBT model. This approach is not only limited to direct preference optimization but is also compatible with various online and offline preference optimization policies and can be employed for reward model training, which are left for future work.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Imsyschaathtarrenacccvvvaaaaoo, arena conversations
2024. URL /tt.htttsss//mggggofocooo////tte ott.aaaaaooaaaaeooooooooerreeroo
A Meta. Llama 3 model card. 2024.
URL https:://githbb.coommeettt
md
Afra Amini, Tim Vieira, and Ryan Cotterell. Direct preference optimization with offset, 2024.
URL wtt.hhttss///xiii org. abs 2402. 10571
Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. general theoretical paradigm understand learning from human preferences. URL atttp://arxia org/
12036
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Ols- son, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran- Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer. Noemi Mer- cado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Con- erly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zae Seffiel-DDodd Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional ai: Harmlessness from feedback, 2022.
Rose Baker and Phil Scarf. Modifying bradley-terry and other ranking models allow ties. IMA Journal of Management Mathematics, 32, 12 2020. doi: 10. 10000//imaa//pppa02....
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language, 2019.
Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 394344) 1952. ISSN 00063444. URL http:
www. jstor.r.......o 1tsttt...r///tabee//3333333
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating lims by human preference, 2024.
Wei-Lin Chiang, Zhuohan Li. Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph Gonzalez adn Ion Stoica, and Eric Xing. Vi- cuna: An open-source chatbot impressing gpt-4 with 90 URL htttps://mssss.rrxt 2023-03-30-viccm
Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences, 2023.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick. and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018.
OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models. 2023.
https: wwm http://///ihuub.cooopeeoeeeeeeee
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high qual-uullty feedback, 2023.
Roger R. Davidson. On extending the bradley-terry model to accommodate ties in paired comparison experiments. Journal of the American Statistical Association, 65(329)3317-328, 1970. ISSN 01621459. WRRLhtpp:2 //www:/wwwww. htwp:wsstorr. jstor. wtttto/.ogg/sttleee//eeae
Neil Dewart and Jonathan Gillard. Using Bradley-Temy models analyse test match cricket. IMA Journal of Management Mathematics, 3012) 2018. ISSN 1471-678X. doi: 10.1093/
imaman/dpx013. URL htttps://daat http://oo.ooro.101 100

---

Published as a conference paper at ICLR 2025
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations, 2023.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela, Kto: Model alignment as prospect theoretic optimization, 2024.
William Alexander Glenn and Herbert A. David. Ties in sarredcomparison experiments us-
ing a modified churstone-mostelle model. Biometrics, 16:86, 1960. URL https://api,
124399691
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021.
Jiwoo Hong, Noah Lee, and James Thorne ORPO: Monolithic preference optimization without reference model.
http://axxvv.oor////////0// 077691
Shengding Hu, Yifan Luo, Huadong Wang, Xingyi Cheng, Zhiyuan Liu, and Maosong Sun. Won't get fooled again: Answering questions with false premises, 2023.
Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. Efficient attentions for long document summarization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pp. 1419-1436, Online, June 2021. Association for Computational Linguistics. doi: 10. 18653v13 2021.naacl-main 12. https:/:: //acl antho org.logg..og/221111 www.0.2..aaacl.... www.caacllaanmmm ma
112
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William Sayed. Mistral 7b, 2023. URL https:
7/arxiv. iv. org/abbs 2311 0...1.00000 22310006220303
Anthony Y. C. Kuk. Modelling paired comparison data with large numbers of draws and large variability of draw percentages among players. Journal of the Royal Statistical Society. Series D (The Statistician), 44(4):523-528 1995. ISSN 00390526, 14679884. URL. http:://www.
jstor. wttp:/orttt/teeee43555333
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, L. J. Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannanch Hajishirzi. RewardBench: Evaluating reward models for language modeling. URL http:
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3214-3252, Dublin, Ireland, May 2022. Association for Computational Linguis- URL htt.Ltpss///c.aattoooo.ooo//////
tics. doi: 10. 10.18666/v112002aac--oooo acl-long: 229
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoe V. Le. Barret Zoph, Jason Wei, and Adam Roberts. The flan collection: Designing data and methods for effective instruction tuning, 2023.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward, 2024.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannanch Hajishirzi. Cross-task general- ization via natural language crowdsourcing instructions. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3470-3487, Dublin, Ireland, May 1ong.244. URL
2022. Association for Computational Linguistics. doi: https: /ttts://aaclat antho wetchologyy.oo www.oog........... aclp:///222ace-eeea0/..cc long. 244
12

---

Published as a conference paper at ICLR 2025
Nvidia, :, Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H. Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper. Bryan Catanzaro, Sharon Clay, Jonathan Cohen, Sirshak Das, Ayush Dattagupta, Olivier Delalleau, Leon Derczynski, Yi Dong, Daniel Egert, Ellie Evans, Alek- sander Ficek, Denys Fridman, Shaona Ghosh, Boris Ginsburg, Igor Gitman, Tomasz Grze- gorzek, Robert Hero, Jining Huang, Vibhu Jawa, Joseph Jennings, Aastha Jhunjhunwala, John Kamalu, Sadaf Khan, Oleksii Kuchaiev, Patrick LeGresley, Hui Li, Jiwei Liu, Zihan Liu, Eileen Long, Ameya Sunil Mahaballshwarkarr Somshubra Majumdar, James Maki, Miguel Martinez, Maer Rodrigues de Melo, Ivan Moshkov, Deepak Narayanan, Sean Narenthiran, Jesus Navarro, Phong Nguyen, Osvald Nitski, Vahid Noroozi, Guruprasad Nutheti, Christopher Parisien, Jupin- der Parmar, Mostofa Patwary, Krzysztof Pawelec, Wei Ping. Shrimai Prabhumoye, Rajarshi Roy, Trisha Saar, Vasanth Rao Naik Sabavat, Sanjeev Satheesh, Jane Polak Scowcroft, Jason Se- wall, Pavel Shamis, Gerald Shen, Mohammad Shoeybi, Dave Sizer, Misha Smelyanskiy, Felipe Soares, Makesh Narsimhan Sreedhar, Dan Su, Sandeep Subramanian, Shengyang Sun, Shub- ham Toshniwal, Hao Wang, Zhilin Wang. Jiaxuan You, Jiaqi Zeng, Jimmy Zhang, Jing Zhang, Vivienne Zhang, Yian Zhang, and Chen Zhu. Nemotron-4 340b technical report, 2024. URL https: /ttp:///aarvv../ org/ abs 2406. 11704
OpenAl. Gpt-4 technical report, 2023.
OpenAL. gpt-4o, 2024a.
URL http:://ppaffamm...rrra.....///p
4gpt-40.
OpenAl. gpt-4turbooand--gt-4 2024b. URL
mode 1s/ 3p/ppt-444 --tt---tttr-oo-------- and -nd-apt-----ad------
R. L. Plackett. The analysis of permutations. Journal of the Royal Statistical Society. Series
(Applied Statistics), 2932)2 1975. ISSN 00359254, 14679876. URL http://www.
jstor.org/stable/224444
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model.
URL http:://p/a/xi..ortt
18290
Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Chris- tian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization, 2023.
P. V. Rao and Lawrence L. Kupper. Ties in paired-comparison experiments: A generalization of the bradley-terry model. Journal of the American Statistical Association, 62:194 02:194-2044 24-200, 1967. URL org httgg//orpuspp 19998924
Amir Saeidi, Shivanshu Verma, and Chitta Baral. Insights into alignment: Evaluating DPO and its variants across multiple tasks. URL BBRl:/tce//arrx..oo///////////
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale, 2019.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler. Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, De- bajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Repre- sentations, 2022.
net http://wwww....eooooooooo//////a
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.
13

---

Published as a conference paper at ICLR 2025
Shuming Shi, Enbo Zhao, Wei Bi, Deng Cai, Leyang Cui, Xinting Huang, Haiyun Jiang, Duyu Tang. Kaiqiang Song, Longyue Wang, Chenyan Huang, Guoping Huang. Yan Wang. and Piji Li. Effidit: An assistant for improving writing efficiency. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguisties (Volume 3: System Demonstrations), pp. 508- 515, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 10.1865.. 18653/1/vv 3023.acl-demo.44 URL / www.hhowoo.......////////// aww.:.....o..................e.. demo. 49
Feifan Song, Bowen Yu, Hao Lang, Haiyang Yu, Fei Huang, Houfeng Wang, and Yongbin Li.
Scaling data diversity for fine-tuning language models in human alignment, 2024. URL https:
//arxiv. abs/ .40..11224 11124
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vin- cent Zhao, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Mare Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoe Le. Lamda: Language models for dialog applications, 2022.
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of LM alignment. URL http:/httt://tt// org/ abs /2310 6944
Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang. Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi. Songyang Gao, Nuo Xu, Yuhao Zhou, Xiaoran Fan, Zhiheng Xi. Jun Zhao, Xiao Wang, Tao Ji, Hang Yan, Lixing Shen, Zhan Chen, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, and Yu-Gang Jiang. Secrets of rlhf in large language models part ii: Reward modeling. 2024.
Tianhao Wu, Banghua Zhu, Ruoyu Zhang, Zhaojin Wen, Kannan Ramchandran, and Jiantao Jiao. Pairwise proximal policy optimization: Harnessing relative feedback for Ilm alignment, 2023.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardIm: Empowering large language models to follow complex instructions, 2023.
Runsheng Yu, Yong Wang, Xiaoqi Jiao, Youzhi Zhang, and James T. Kwok. Direct alignment of language models via quality-aware self-refinement, 2024. URL httphttt////rxivorrr///////
2405 21040
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence?, 2019.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing. Hao Zhang, Joseph Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences, 2020.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1
TTE DATA IN CURRENT PREFERENCE DATASETS
We conduct statistical analysis of existing tic data across common preference data, as presented in 6 Current pairwise preference data are scored by LLMs or labeled by humans to establish preference rankings. Preference data scored by LLMs, commonly rely on the quality score consid- ering multiple aspects to differentiate between preferred and dispreferred responses. Ultrafeedback (Cui et al. 2023), as a popular preference dataset, scored based on GPT-4 feedback, and contains 383k pairwise responses. We find that 17.0% of the data pairs exhibit identical quality scores. The Ultrafeedback. binnaiee (Tunstall et al), a subset of this dataset, also includes 12.1 1 tie data. This analysis underscores the prevalence of ties in preference datasets.
Datasets
Tie data( %))
Uleebbee
17.0
Ulrafeedback binarized
12.1
29.4
Table 6: Statistics on common preference datasets.
Unlike datasets featuring explicit quality scores, some preference datasets include only human- provided preference labels. Figure illustrates two pairs of responses from the Chathot Arena" (Chinng et al. 2024). judged by different users. Specifically, user793 prefers RI and disfavors R2 for prompt 1. Conversely, user1244 exhibits no clear preference between the two responses for prompt 2, categorizing them as tied. The Imsys-chatbot. arena aet.aaaeeeneeeeaaaa conversations dataset, which collects the pairwise data from Chatbot Arena, reveals that 29.4% of the total samples are tied. This statistic highlights the significant presence of tie data in human- labeled preference data.
it Broning with friend agroupine. bi Sharing data stalisiciin / Sharing mazsphore atageent Shanng phatster C Shering videotet. Sharin food of Shurig someting cedling imporanten0,, Sharing know edge education.
bitter than
Emaji for ""sharinght List
10
Rt vians gennested
Promp:
Sharing handsn' Share looer" Group chat konter Sharing dipboarsicorten Social meds Shaiing Inmier Sharing keyboard lcon
kzada groemte
Mount Everaai ia the highest mounsan in the work; with height of 3,545 moters (23) CCU lestt Tibe. mountain nammdd arter George Everstt drtaan gecgrapher surveyox, who led beam that mepped the Himsalayas the mid- century. is populer
heiht
Prompt 2
M2: aioece geranded
Figure 4: Human evaluation of pairwise responses across various prompts in the Chatbot Arena.
A.2 THE TOBT MODEL.
By substituting into 1 sech"(()dg we obtain that
F(t) 1
1 tanh (t),
sech
(21)
15

---

Published as a conference paper at ICLR 2025
where tanh(t) exp(t)-exp(-)) Then r12 4 J ln +a) sech"(y/ dy be written into exppp)+
r12 1/2 Je sech² (t) dt, where t -(lu -In to We can obtain the representation of r12
following Equation
22
lanh(ox) - tanth (41)
12
2 tanh (t1)
exp(tt) -xp f1)
exp(t texpl -h
2 exp(26 )xxpp(
(22)
(xp(2f,) -exp
2
(xp(
exp(2(1) exp( ( 2/1)
1 cxp(In A1 In 22 a)
exp(In A2 In A1 + a) exp(In In - a)
By substituting with In á in the last line of Equation 22 the final presentation of T12 can be
represented by Equation 23
1 - (xp A In A2 - l)
712
exp In A2
In (nd) exp(In 22 In の)
A1
(23)
6dz
adz
\1
\ Ød²
For the T(12) = [- (In An sech² (5) dy, we can rewrite into T(12) = _/2 S sech? (t) dt,
12)
in which t (In Ag)tt and b - da) Then we compute r(12) according to
Equation 24
T(12) = (tanh((t) - tanh(t2))
2
- (xxoo//ooooxxxxxxxxxxx=x==xoooo- (exp)(() (x0p))))ex0(((()))) + exp(-4))
2(exp((11 + + cxp(
+ exp(-tz)
exp(11 - 12) - exp(12 11)
exp((t + +22 + exp(ti - 12) + exp(t2 -1) + exp( -t- - t2)
exp(a) exp(-a)
exp(-(In - In 22)) + exp(a) + exp(o: + exp A In A2)
(24)
By substituting a with In in the last line of Equation 24
we can obtain the final representation of
T(12) given in Equation 25
T(12)
Aa Ai
+ + + X2
A1 - 1) 2 0 X122 t>122 + ΦX²
(25)
20 11
(A1 + 2d2) + +
A.3 SELECTION a TODO
In this section, we elaborate on the process of selecting the optimal value for a in TODO. Initially generate a series of random values emulate the expression 3 log ne(vil2) 0 log Zo i
am((3/)) x>)
16

---

Published as a conference paper at ICLR 2025
which represents the reward difference between two responses in the initial stage of preference alignment. Subsequently, we calculate the corresponding pairwise non-tied and tied losses across a range of values. Figures 5a and [5b present the visualization of the results for two distinct value spans, illustrating the impact of a on the losses.
4.5
35
16: tosc
25
20
15-
1.0
0.0
10
0.2
0.6
9.6
0.8
1.0
1.2
ondferencc losses
and creercc losses
(a) O varies from to
(b) varies from 0.1
Figure 5: The initial preference and tie losses simulated with different a values.
A delicate balance between the non-tie loss and the tie loss is beneficial to the TODO performance. Our observations indicate that the non-tie and tie losses are interconnected through a power-law relation. As depicted in Figure [ the preference loss sharply ascends with increasing values of a, exceeding the concurrent growth in tie loss.
To achieve equilibrium between these two types of losses and to identify appropriate value of that maintains the preference loss near the levels observed in the original DPO (which implies that should not be markedly distant from zero), we establish two thresholds: the initial tie loss in TODO is restricted to a maximum of 1.5, and the initial preference loss is restricted to a maximum of 1.0. These constraints instrumental in guiding the selection of ensuring that the optimization process effectively reconciles the competing objectives of preference and tie loss minimization.
Initially, graph the variations in both preference and tie losses across a spectrum of values, ranging from to 10, as illustrated in Figure 5a This visualization reveals that the preference loss using TODO is consistently higher than desired. Subsequently,
refine approach by focusing on narrower band of values, specifically from 0.01 to 0.1. The detailed outcomes of this refined analysis are shown in Figure 5b
Theoretically, the BT-model and our TOBT model both rely on the assumption that the reward difference μ r(x, vı) r(x, 122 follows the logistic distribution with unit variance. The mean of BT-model is 0 and the mean of TOBT is a > 0. In the early training phase of LLM, μ is typically small in magnitude. Hence, a large may produce very large loss (E log ( a)) and leads to gradient vanishing, where is the Sigmoid function. From this perspective, < 1 is beneficial the other hand, a cannot be too close since
since the gradient of a is less than 0.2 for a > the weights of tie data is propotional exp( 2a.. There is clear tradeoff.
Empirically, we further compare the performance of TODO with different a on our preference testset and MT Bench besides in Tables 7 and g the results on MT Bench are scored by gpt-4o-2024-05-1 OpenAl 2024a). It shows that the performance is relatively not sensitive to the change of a within reasonable range like (0.1, 0.8), but degrade significantly when is too large, which agrees with our theoretical analysis. This behaves like the tuning of learning rate, which can also lead to divergence beyond some threshold.
17

---

Published as a conference paper at ICLR 2025
Ultrafeedback (tie ratio 0)
Ultrafeebback (tie ratio 0.2)
Chatarena (tie ratio 0)
Chatarena (tie ratio 0.17)
Methods
TaDO(a 0.1)
77.20
76.40
76.80
78.47
TODO(a = 0.2)
77.67
76.40
77.00
77.93
TaDO(a 0.5)
77.33
76.87
77.53
77.87
T = 0.8)
Diverge
75.47
77.27
76,40
TODO(a 1.2)
Diverge
75.33
Diverge
Diverge
Table 7: Test-set accuracy in Mistral-7B model with different values set in TODO.
Ultrafeedback Ultrafeedback (tie ratio 0)
Chatarena (tie ratio 0)
Chatarena (tie ratio 0.17)
Methods
(tie ratio 0.2)
TODO(a 0.1)
5.85
5.96
5.52
5.83
T = 0.2)
5,44
5,94
5.53
5.58
TODO(a 0.5)
5.73
5.91
5.69
5.78
TODO(a = 0.8)
Diverge Diverge
5.81
5,48
5.51
TODO(a 1.2)
5,41
Diverge
Diverge
Table 8: MT Bench scores in Mistral-7B model with different a values set in TODO.
A.4 THE OBJECTIVE FUNCTION OF TODO WITH PAIRWISE DISTINGUISHED PREFERENCE DATA
It is straightforward derive the TODO objective in pairwise data with distinct preference differ- ence. Under the TOBT model as Equation 12 we can obtain the possibility of yı being preferred over y2 following Equation 26 by substituting & into exp(a.).
exp(r* (1)
p" (3) > 82|2) exp(r*(z, (1) + + exxp(r U2))
cxp(r"(x; 91))
(z)(r*(z,)))) exp ()) exp(r"(x,1))
(26)
+ exep(a) exp(r" (z, 92) v1))
1 1 exp(r" - r (x) aa)
(r (x, y2) - a))
Recall that the (unavailable) ground-truth reward through is given as follows:
(y(z)
(x, v) = Blgg
+ 3 log Z(x).
(27)
Substituting Equation
27 into Equation
26 we derive the per-instance preference possibility as
shown in Equation
28
x (V(l() - $ Blg
xt' (V2(2)) a)
pz (31 > 32lz) = o (Blog
(28)
(11())
Teat (Vel
A.5 THE OBJECTIVE FUNCTION OF TODO WITH PAIRWISE TIE DATA
For instances of pairwise tie data, by utilizing the TOBT model as defined in Equation 13
we can
compute the possibility that y1 and are tied, shown in Equation 29
exp(r (z, v1)) exp(r' (z. u2))
p" pv( yι|=)
(exp(r*(xx,))) @exp(r* 1 (s, va) (xx)
@exp(r"((, (z,))))
exp(r (z, 91) + (1.
(exp(r*(x, v1)) exp(o)exp((1 exp(r" (co (2 v2))
v2))) exp(a) exp(r" v1))
(29)
18

---

Published as a conference paper at ICLR 2025
Which can be transferred into:
exp(2a)
p p" = va|=) exp(r* v1) (x, v2) + +a) exp(r" (x, - (x, a) + exp(2a)
(30)
by dividing exp(r* (ar, v1) + r" (a, U2)) both the numerator and denominator. Because exp(2n) can be expressed by exp(r" (z. y1 - r" (z. 1/2 a + r" 1/2) r" (a, + a), we can rewrite
the denominator of the last line in Equation 30 into (1 + exp(r (I, - - (2, 32) + a)) +
exp(r" (z. 82) r" (2, (x,81) 91) + a)). Then we can obtain Equation 31
exp(2x) -1
(w/ = 32) (1 exp(1 (1) v1) - (1, 12) + +a)) + exp(r* - v1 a))
(31)
Substituting Equation 27 into Equation
31
we obtain the per ver-inetance tie possibility as shown in
Equation
32 where μ Blog
(vi
log x (valar)
Rea (y1
(12(
exp(2a) (1 + explu +a))())))((1) + exp(-u
(y1 y2(z)
(32)
A.6
THE GRADIENT OF TODO WITH PAIRWISE DISTINGUISHED PREFERENCE DATA
The gradient of TODO with pairwise distinguished preference data can be expressed by following equation:
Ve C (TT; neet) = V (z. log Logo(( - a) ].
(33)
where μ = 3 log
(a log [e(w]lz) Then Equation
33 can be written into following form:
3nd (y (a)
(p a)
V CoCDODO (Toi nee = - -.
a(p-c) vo(ui).
(34)
Using the properties of Sigmoid function o' (x)) = oxx) (x)) and of (--z)= = (z)), we ob- tain the final gradient, VaLTODO (To; Teet) - -0 log
5u (r)
33 log
Te (vn (z) +
Tnd (1r(va
Ferlyn [2]
a)[V, log(=(y) (z)) Vo (1g(((12()))]]
A.7 THE GRADIENT OF TODO WITH PAIRWISE TIE DATA
The gradient of TODO with pairwise tie data can be expressed into following form:
exp(2ai -
V VeLTODo ( meet)
V Fe (a. (1)))))))))))))))))))))))))))))) log(
}}
(35)
f(w)
where f(4) = (1 + exp(-x ax)( and μ = 3 log 50 (m)
nnd (vn (z) - B log =a(u)lz) Then
ma((val)) [m)
Equation 35 can be rewritten into:
f()) vo(())
V VeLToDo (To; )ert) = g2)
(36)
[()
We can derive the final gradient of TODO following Equation exp(=uta)-exx(ptaa
37
where G(μ²
Vo = -B v2) Dl
Vo log(=([>(x))]]
(37)
For any value of p. we have G(μ) G'(n), the first derivative of G(u), is always negative derived from Equation 38 indicating G(u) is monotonically decreasing with respect to μ.
G(-P), indicating G(μ) is an odd function. Additionally,
('(w) -(exp(-p a) + exp(u + a)) + exp(2a) texp(=#+0) a))
(38)
(1 + exp(p + a1) + exp( papl-p1- a))²
19

---

Published as a conference paper at ICLR 2025
A.8 CONSTRUCTION OF THE TRAINING DATASETS
The original Ultrafeedback dataset contains about 64k prompts from diverse resources, including UltrChhat(Ding al. 2023), ShareGPT (Chiang al. March 2023), Evol Instruct (Xu al. 2023),
TruthfulQA (Lin et al. 2022). FalseQA (Hu et al.) (023). and FLAN (Longpre et al.) 2023] Each prompt is used to query multiple LLMs and generate 4 different responses, resulting in total of 383k samples. Each response in pairwise data has its quality score provided by GPT-4 feedback. We sample 20k samples from these 383k samples to construct different training sets. To ensure consistency and fairness in data distribution, each sampling follows the original distribution of Ul- trafeedback dataset, as shown in Table / We then construct datasets with different proportions of tie data, with the tie data ratios varying within the set {0, 0.1, 0.2, 0.2, 0.3).
Data source
Evol instruct
False QA
FLAN
Sharegpt
TruthfulQA
Ultrachat
Percent(%) (%)
15.63
3.66
32.73
31.19
1.27
15.52
Table 9: Data source distribution in each sampled train set.
A.9 TRAINING HYPERPARAMETERS
We use full-parameters fine-tuning when comparing different methods on Mistral-7B and Llama 3-8B models. We use Adam optimizer and the weight decay is set into 0. We use cosine learning rate scheduler, and the other detailed settings DPO and TODO is shown in Table
Model
Learning rate
Batch size
B
Mistral+SFT
5e-7
64
0.01
Llama 3+SFT
le-6
128
0.01
Table 10: Training hyperparameters settings of DPO and TODO.
A. EVALUATION METRICS
For the evaluation based OpenCompass, use the default prompt template, and the specific metries and evaluation mode settings are shown in Table II In this table, the PPL mode is used for multiple-choice tasks, utilizing the perplexity of each choice as the evaluation metric. The LL mode used for the Winogrande task, where log likelihood is employed to evaluate task performance.
Task
Piqa
ARC-c
ARC-e
MMLU
Hellaswag
Winogrande
Mode
PPL
PPL
PPL
PPL
PPL
LL
Metric
0-shot
0-shot
0-shot
5-shot
O-shot
0-shot
Table 11: Evaluation details multi downstream tasks, PPL represents accuracy based perplexity and LL represents accuracy based log likelihood estimation.
A.11 REWARD MARGIN CHANGES DURING THE TRAINING PROCESS
Figure 6 illustrates the reward margin changes of Mistral+SFT and Llama 3+SFT models aligned with DPO and TODO. We observe that the growth of the reward margin in DPO decelerates as the proportion of tie data within the training set increases for both the Mistral model shown in Figure 6al and the Llama model as shown in Figure 6b For models aligned with TODO, a similar deceleration in the growth of the reward margin is noted with an increasing proportion of tie data in the training dataset, as shown in Figure (c for the Mistral model and in Figure (d for the Llama model. Furthermore, models aligned with TODO exhibit more pronounced disparity in
20

---

Published as a conference paper at ICLR 2025
data ratio:
Tie data ratio: 0.1
Tie data ratio: 0.1
data ratio:
Tie data ratio: 0.3
Tie data ratio: 0.3
Reward margin
margin
200
400
600
800
1:00
200
300
400
Step
Step
(a) Mistral series models aligned with DPO
(b) Llama 3 series models aligned with DPO
ratio:
Tie data ratio: 0.1
Tie data ratie: 0.1
ratio:
Tie data ratio: 0.3
Tie data ratie: 0.3
Reward margin
mergin
2
200
400
600 800
1.00
200
300
400
Step
(c) Mistral series models aligned with TODO
(d) Llama series models aligned with TODO
Figure 6: Reward margin changes of Mistral-7B and Llama 3-8B models aligned with DPO and TODO during the training process with varying tie data proportions. The dashed lines, sharing the same color as the solid lines, represent the fit the reward margin.
reward margins when utilizing varying ratios of tie data, indicating higher sensitivity of TODO to preference relations of the pairwise training data than DPO.
A.12 DETAILED SCORES ON REWARD BENCH
In this section, we show the detailed scores of each category Reward Bench. For the results, we highlight the best performance over all results in underline, and mark the performance better aligning with different ratio of tie data in bold. Table 12 and Table [3 show the performance of the Mistral+SFT and Llama 3+SFT models aligned with DPO and TODO on Reward Bench.
A.13 MORE EXPERIMENTAL RESULTS AGAINST OTHER BINARY MODEL BASED METHODS
We compare the proposed TODO method with KTO (Ethayarajh et 2024), SimPO (Meng al., 2024), and ODPO (Amini et al. 2024) on Mistral-7B evaluated on MT Bench and our preference testsets. Besides, we construct another Chatarena d sourced from diverse human-labeled open data (Ims) 2024). Chatarena captures the diversity and complexity of real-world human pref- erences across 96 different languages. It includes pairs with clear preference differences as well as used training set of 20,000 pairs with tie data ratios of 0 and 0.17 (the natural tie ratio of this dataset) and a test set of 1,500 randomly selected samples.
ties. In our experiments,
21

---

Published as a conference paper at ICLR 2025
Method
Tie Data Ratio
Chat
ChatHard
Safety
Reasoning
Prior
Average
SFT
66.48
54.61
35.73
62.94
40.20
51.99
DPO
0.0
91.34
66.45
75.79
72.36
49.23
71.03
TODO
0.0
93.85
64.25
76.88
72.17
61.07
73.64
DPO
0.1
91.9
67.98
73.64
72.18
53.67
71.87
TODO
0.1
94.13
64.69
71.80
73.43
64.25
73.66
DPO
0.2
91.34
64,47
78.27
74.37
49.83
71.66
TODO
0.2
95.81
64.69
78.49
62.30
61.63
72.58
DPO
0.3
85.75
64.04
75.78
75.01
45.56
69.23
TODO
0.3
89.94
65.13
77.13
73.37
57.06
72.53
Table 12: Mistral model results on Reward Bench trained with different ratios of tie data.
Method
Tie Data Ratio
Chat
ChatHlar
Safety
Reasoning
Prior
Average
SFT
X
67.60
55.48
41.89
65.17
43.26
54.68
DPO
0.0
93.3
67.32
75.18
84.23
54.78
74.96
TODO
0.0
93.58
65.79
77.66
83.32
59.72
76.01
DPO
0.1
93.85
63.82
77.14
83.52
53.91
74.45
TODO
0.1
96.37
64.69
79.32
84.08
59,59
76,81
DPO
0.2
91.62
66.45
78.18
84.80
53.70
74.95
TODO
0.2
92.74
66.89
75.50
84.37
60.37
75.97
DPO
0.3
89.39
65.35
78.31
87.02
52.87 60.33
74.59
TODO
0.3
93.02
64.91
74.35
84.64
75.45
Table 13: Llama 3 model results on Reward Bench trained with different ratios of tie data.
The results on MT Bench are scored by 2024-05-13 (OpenA 2024a For ODPO, distinct quality scores are required for the chosen and rejected samples in a pair, condition only met in the Ultrafeedback dataset without tie data. Therefore, we only report ODPO results under the data ratio 0 setting in Ultrafeedback. For each method's specific hyperparameter settings, we follow the configurations used previous work (Saeidi al.) Meng et al.] 2024 2024][miini Amini et al., (2024] Ethayarajh ctt al. (024).. As shown in Table Table TODO achieves the best performance compared with other methods in the presence of tic data. If only binary preference data is available, TODO still delivers competitive performance.
22


Reviews:
[{"review_id": "gsg1QBPsYK", "reviewer": "Reviewer_qtpx", "summary": "The authors present an interesting approach to enhancing the alignment of LLMs through the use of ternary preferences. A limitation of binary models is their inability to effectively address tie situations in preference data, where two responses may be equally preferred or indistinguishable in quality. To address this issue, the authors propose an extension of the BT model, termed the Tie-rank Oriented Bradley-Terry (TOBT) model, which accommodates ties in preference relations. Building on this framework, they introduce a novel alignment algorithm called Tie-rank Oriented Direct Preference Optimization (TODO), specifically designed to manage ternary preference data that includes \"prefer,\" \"tie,\" and \"disprefer.\" They conduct experiments comparing their TODO algorithm with the DPO. The results demonstrate that TODO consistently outperforms DPO in modeling preferences across both in-distribution and out-of-distribution datasets, as well as on several benchmarks. The paper also provides an analysis of the key factors contributing to TODO's improved performance.", "questions": "1. Can TODO be integrated with other alignment methods beyond DPO, such as reinforcement learning from human feedback (RLHF)? If so, how would it interface with existing reward models?\n  \n2. Are there any trade-offs in using the TOBT model compared to traditional binary preference models? For example, does incorporating tie data introduce additional computational complexity or affect convergence?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a novel extension to the Bradley-Terry model by incorporating ties through the Tie-rank Oriented Bradley-Terry (TOBT) model. This allows for more nuanced preference modeling in LLM alignment.\n  \n2. The theoretical foundations of the TOBT model and the TODO algorithm are well-developed. The authors provide detailed derivations and clarify how the tie parameter α is integrated into the model.\n  \n3. The paper is generally well-written and structured. The methodology, including the mathematical formulations and the proposed algorithm, is clearly presented. The inclusion of figures and tables aids in understanding the results.", "weaknesses": "1. The extension from the traditional Bradley-Terry (BT) model to the Tie-rank Oriented BT (TOBT) model is central to the proposed method. However, the paper provides limited theoretical justification for this extension. The choice of the parameter α in the TOBT model appears somewhat arbitrary, and while Appendix A.3 attempts to justify its value based on balancing loss values, there is no rigorous analysis or sensitivity study demonstrating how different values of α affect the model's performance. Additionally, the paper lacks a thorough exploration of the probabilistic interpretation and properties of the TOBT model compared to the original BT model. A deeper theoretical analysis would strengthen the foundation of the proposed method and clarify its advantages and limitations.\n\n2. The experiments primarily focus on datasets derived from Ultrafeedback and a select few benchmarks. While Ultrafeedback is a valuable resource, it may not fully represent the diversity and complexity of real-world human preference data, especially in terms of the prevalence and nature of tie data. The paper does not evaluate the method on more varied datasets, such as those involving different languages, domains, or more complex tasks where tie situations might behave differently. This raises concerns about the generalizability of the proposed method to broader applications and different types of data.\n  \n3. The paper compares the proposed TODO method primarily against Direct Preference Optimization (DPO). However, there are other existing methods in preference alignment and RLHF that could serve as relevant baselines, such as PPO, RPO, IPO), and KTO. By not including these comparisons, it is difficult to assess how TODO stands relative to the other existing methods in preference alignment. Including a broader range of baselines would provide a more comprehensive evaluation and strengthen the empirical claims.\n  \n4. While the paper highlights the prevalence of tie data in preference datasets and proposes methods to utilize it, But ties may often result from annotator uncertainty, ambiguity in prompts or responses, or inconsistencies among annotators. By treating tie data as equally informative as clear preferences without accounting for potential annotation noise or bias, the method might be incorporating noisy signals into the model. A discussion on how to handle annotation noise, perhaps by incorporating methods to assess annotator reliability or confidence, would strengthen the approach. Additionally, exploring strategies to improve data quality or to model uncertainty explicitly could enhance the effectiveness of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PIuTAxwoSg", "reviewer": "Reviewer_iuqB", "summary": "This work aims to directly model the fact that binary preference models such as the standard and well-known Bradley-Terry (BT) model cannot integrate ties and uncertainties when working with preference data. The main idea is thus the 'ternarization' of the original binary BT model that takes into account possible ties, where 'ties' are modeled via an additional parameter alpha. A large part of the paper is then dedicated to motivating and deriving the ternary (i.e., tie-aware) BT model, and then, based on that model on deriving and explaining TODO, a tie-aware extension of the original DPO algorithm for preference optimization. Experiments with two standard models (Mistral and LLaMA) and with a spectrum of standard datasets demonstrate the benefits of the TODO approach over its base binary DPO approach, even in setups without any ties. The authors then aim to provide some theoretical and empirically grounded interpretations on what makes TODO a more expressive and better-performing alternative to DPO.", "questions": "Q1 (related to the first, main point listed under \"Weaknesses\"). Speaking of previous work, how does your tie-aware BT model differ from or improve upon existing multi-class generalizations of the BT model that I provided under weaknesses plus the ones that you also cited? Are there any components, ideas or other aspects of prior approaches to BT generalisation that could be incorporated into or compared against your method?\n\nQ2. It seems that the results are also dependent on the tie data ratio in the full training set of 20k instances. How can a practitioner determine the best (or task-optimal) choice of the tie data ratio? What if tie data is lacking for some specific use-cases?\n\nQ3. How were hyperparameters from Appendix A.9 selected? Is there any substantial variation of the results based on the hyper-parameters?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written, and it is very easy to understand its core motivations and follow how the entire derivation of the tie-aware BT model and the corresponding TODO model unfolds from the main design assumptions.\n- The provided results offer reasonable evidence on the benefits of TODO over DPO. The chosen evaluation data and models are adequate.\n- The paper raises awareness on modeling ties in preference optimizations, where such valuable data often gets discarded under inadequate models or models with too strong assumptions (such as the binary BT) - taking into account finer granularity of preference data might yield further improvements in this line of work, and this paper sheds some new light on this link.", "weaknesses": "- One of the core issues with the work lies in the fact that other researchers in 'less recent times' have already attempted and derived generalised versions of the BT model, which better align with multi-class (i.e., non-binary) problems. Even a cursory search online reveals some very relevant literature which is not cited nor discussed in this work. I would require the authors to provide a thorough overview of that relevant work and discuss why a new (tie-aware) derivation might be needed here, and whether previous solutions could be repurposed for the extension of the original, binary BT model. For instance, I refer to the following relevant work here:\n-- https://dl.acm.org/doi/pdf/10.5555/1248547.1248551\n-- https://proceedings.neurips.cc/paper_files/paper/2004/hash/825f9cd5f0390bc77c1fed3c94885c87-Abstract.html\n-- https://link.springer.com/article/10.1007/s10994-011-5240-0\nSee also Q1 related to this comment.\n\n- Another concern is the lack of generality of TODO as it is derived directly as an extension of one concrete model: DPO. Following that, only DPO is also used as the only relevant baseline, and I see at least two potential issues here:\n(i) While the authors state at the very end of the paper that the approach is \"(...) not only limited to direct preference optimization but is also compatible with various online and offline preference optimization policies and can be employed for reward model training...\", already this work should have coupled the idea with another preference optimization method to show this decoupling. In essence, the authors should both theoretically and empirically discern between making 'ternary decisions' and deriving enhanced preference optimization algorithms that incorporate the possibility to do ternary decisions. \n(ii) DPO is used as the only baseline, while there is a plethora of other online and offline preference optimization algorithms out there, where some of them should have been included as baselines and reference points. For instance, the KTO approach disposes of the need to even collect preference data and model preference decisions as part of the process (unlike DPO or PPO). One core question is then: can we simply use KTO instead of 'ternarizing DPO'? What would be the benefits of TODO versus KTO? \n(iii) The authors very vaguely comment in lines 122-123 that previous preference alignment studies \"...overlook the potential of leveraging tied preference data to refine alignment, whereas our approach is complementary to most of them.\" It is unclear to which studies the proposed approach is complementary, and as mentioned above - it does not offer sufficient empirical validations to demonstrate that the approach is better than KTO, IPO, etc. I would suggest including direct comparisons to these other methods beyond DPO.\n\n- All the experiments are based on the UltraFeedback dataset, and the training set size is limited to 20k data instances. The training set size might be a very important factor in defining the final performance (and benefits of ternary vs binary vs unary modeling as with KTO). Therefore, the paper should investigate how the standard DPO and KTO behave in comparison to TODO also for larger datasets (e.g., using the full UltraFeedback). Ideally, the work should experiment with another preference alignment dataset as well beyond UF, I would strongly suggest running experiments with another dataset, and compare against methods such as DPO and KTO also on larger datasets.\n\nI am willing to reconsider my initial assessment and position if these central weaknesses get addressed by the authors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g9ygksHNyp", "reviewer": "Reviewer_BkLA", "summary": "This paper proposes the inclusion of ties in the DPO framework to be able to model preferences in a more nuanced way. It derives a ternary model for preferences analogously to the binary Bradley-Terry model, and defines a novel preference learning objective (TODO) accordingly. Experiments are conducted by preference-tuning Llama3 and Mistral with preferences from Ultrafeedback with varying proportions of ties, and evaluating them in and out of domain, and on downstream tasks. The new TODO objective outperforms DPO in preference modeling and downstream tasks across the bench.", "questions": "1. What is the intuition for why it works better than DPO even without ties in the data?\n2. What is the naturally occurring frequency of ties in the UltraFeedback data?\n3. How does it compare empirically against related approaches, especially ODPO?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea is well motivated, as ties are relevant especially in human preference elicitation and can be expected to become even more prevalent the stronger generative models get. \n2. The experiments are strongly supporting the new objective, the results are convincing.\n3. The proposed solution is based on a solid intuition and supported by detailed derivation.", "weaknesses": "The only claim that is empirically not supported is that TODO “exhibits better robustness against potential noise in binary preference data.” There is no experiment studying noise in preferences (ties do not need to be caused by noise, they also present adequate equalness). One could easily set up an experiment with artificial noise added to preferences to test this hypothesis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lHi20FDXFm", "reviewer": "Reviewer_CZTq", "summary": "Existing LLM alignment algorithms rely on the binary BradleyTerry model, which is hard to capture ties in preference. To address this issue, this paper proposes Tie-rank Oriented Direct Preference Optimization (TODO), an alignment algorithm that  leverages TOBT’s ternary ranking system. \n\nTODO rewrites the ranking probability in BradleyTerry to add tie instances and derive the loss function of TODO. Experimental results verify the effectiveness of TODO.", "questions": "I am also very curious that if the performance of LLM alignment will improve if you just simply filter out the tie preference pairs out of the training set.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The strengths of this paper can be listed as follows:\n1. The motivation of this paper is clear and reasonable. Starting with the improvement of the BT model, this paper introduces the tie instance. This research idea is consistent with theoretical improvements of BT model.\n2. The paper writing is good. I think it is easy to follow the authors' research ideas.\n3. Experimental settings are detailed and well organized.", "weaknesses": "The weaknesses of this paper can be listed as follows:\n1. **Limited baselines.** This paper only provides the comparison of experimental results between TODO and DPO, which is relatively limited. I think it is necessary to supplement some other algorithms as baselines, e.g. SimPO (https://arxiv.org/abs/2405.14734).\n2. **Limited improvements.** The improvement of TODO over DPO is limited as shown in table 2 and table 3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)