TODO: Enhancing LLM Alignment with Ternary Preferences

## Abstract

Abstract Aligning large language models (LLMs) with human intent is critical for enhancing their performance across a variety of tasks. Standard alignment techniques, such as Direct Preference Optimization (DPO), often rely on the binary Bradley-Terry (BT) model, which can struggle to capture the complexities of human preferences—particularly in the presence of noisy or inconsistent labels and frequent ties. To address these limitations, we introduce the T ie-rank O riented B radley- T erry model (TOBT), an extension of the BT model that explicitly incorporates ties, enabling more nuanced preference representation. Building on this, we propose T ie-rank O riented D irect Preference O ptimization (TODO), a novel alignment algorithm that leverages TOBT’s ternary ranking system to improve preference alignment. In evaluations on Mistral-7B and Llama 3-8B models, TODO consistently outperforms DPO in modeling preferences across both in-distribution and out-of-distribution datasets. Additional assessments using MT Bench and benchmarks such as Piqa, ARC-c, and MMLU further demonstrate TODO’s superior alignment performance. Notably, TODO also shows strong results in binary preference alignment, highlighting its versatility and potential for broader integration into LLM alignment. The implementation details can be found in https://github.com/XXares/TODO .

## 1 Introduction

Large language models (LLMs) demonstrate remarkable potential in various tasks  (Huang et al., ; Hendrycks et al., ; Shi et al., ), with performance gains linked to better alignment with human intent  (Mishra et al., ; Christiano et al., ; Wu et al., ). The alignment process typically involves two stages: Supervised Fine-Tuning (SFT) to establish instruction following abilities (Thoppilan et al., ; Sanh et al., ; Mishra et al., ), followed by preference fine-tuning to refine the model’s alignment with human preferences (Ziegler et al., ; Christiano et al., ). This stage typically employs either reinforcement learning (RL)-based (; OpenAI, ; Ramamurthy et al., ) or RL-free methods (; ; ), both leveraging the preference datasets. Effective alignment is enhanced by the diversity of training data, enabling LLMs to accurately learn from high-quality pairwise responses (Cui et al., ; Song et al., ; ).

Current alignment methods relying on the Bradley-Terry (BT)  (Bradley & Terry, ) model consider only two preference rankings: preference and dis-preference, which restricts the diversity of learnable information. A notable challenge is the inconsistent quality of the pairwise preference data, often showing minimal discernible differences (Nvidia et al., ; Amini et al., ). Table 1 shows a sample from the Ultrafeedback-binaried dataset(^1^11[https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized](https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized)) (), which is commonly used in preference alignment procedures (; ). In this example, both responses have identical quality score of 8.5 from GPT-4 evaluations (OpenAI, ), differing only in narrative sequence and text format. In practice, we observe a considerable amount of tie data in common preference datasets and chat arenas judged by humans, as detailed in Appendix A.1. These ties encompass a variety of information, necessitating a more nuanced analysis. However, existing preference optimization techniques, such as Direct Preference Optimization (DPO) (), are constrained by their reliance on the binary BT model and struggle to effectively manage tie relations. How to learn useful information from tie data and achieve nuanced preference modeling in the alignment process remains to be explored, which is the goal of this paper.

Figure: Figure 1: Comparison of DPO and TODO. DPO relies on the BT model, which is only capable of handling binary preferences. When responses are tied, it either learns incorrect preference information or discards tied data, preventing learning from such data. In contrast, the proposed TOBT model can directly model ternary preferences. Based on this, TODO can learn more information from tied data and exhibits better robustness against potential noise in binary preference data.
Refer to caption: x1.png

**Table 1: One pair of responses in Ultrafeedback-binaried Dataset which have identical quality scores evaluated by GPT-4. R1 is treated as the preferred response and R2 is treated as the dispreferred one.**
| Prompt | Q: Ninette de Valois was involved professionally in what activity?$\backslash$nA: |
| --- | --- |
| R1 | Ninette de Valois was professionally involved in ballet as a dancer, choreographer, and founder of the Royal Ballet in London.$\backslash$nConfidence: 95% (score: 8.5) |
| R2 | Ninette de Valois was involved professionally in the field of ballet. She was a choreographer, dancer, and founder of the Royal Ballet in London. (score: 8.5) |

Our primary contributions can be unfolded as:

- ❶ We enhance the existing human preference alignment process by incorporating a “tie” ranking, transcending the traditional binary rankings as depicted in Figure 1. We first extend the BT model into the Tie-rank Oriented BT (TOBT) model. The TOBT model incorporates the concept of preference uncertainty, allowing for the representation of ties alongside “prefer” and “disprefer” rankings. This innovation enables a more comprehensive handling of preference relations.
- ❷ Building on the TOBT model, we introduce the Tie-rank Oriented Direct Preference Optimization (TODO) algorithm. TODO is designed to accommodate ternary ranking relations, offering a nuanced approach to preference alignment. By integrating the tie relation, TODO is capable of learning from a broader spectrum of preference information, enhancing the adaptability and accuracy of LLMs.
- ❸ We use Mistral-7B and Llama 3-8B to conduct experimental validation. First, we evaluate the effectiveness of DPO and TODO in terms of preference modeling accuracy. Our evaluation spans both in-distribution dataset, drawn from the same source as the training data, and out-of-distribution dataset, notably the Reward Bench (). Results indicate superior preference modeling by TODO. Additional assessments on MT Bench (Zheng et al., ) and other popular benchmarks such as Piqa (Bisk et al., ), ARC-c, ARC-e (Clark et al., ), Hellaswag (Zellers et al., ), MMLU (Hendrycks et al., ) and Winogrande  (Sakaguchi et al., ) confirms TODO’s enhanced alignment capabilities. Finally, we provide an intuitive analysis highlighting TODO’s advantages over DPO in two dimensions: enhanced granularity in preference modeling and increased diversity in the acquired information.
- ❹ TODO can also be directly applied in *binary* preference alignment process, outperforming DPO with standard binary preference datasets. Furthermore, the proposed TOBT model can be utilized not only in offline policies like DPO but also can be integrated into other online policies or used to train a reward model.

## 2 Related works

Preference alignment in LLMs. Current methods for aligning preferences in LLMs often utilize the BT model  (Bradley & Terry, ) and the Plackett-Luce ranking model  (Plackett, ) to capture human preferences. RL-based approaches, such as Reinforcement Learning from Human Feedback (RLHF, ()), require a reward model to assess generated responses, which is typically trained on pairwise preference data gathered from crowdworkers  (Wang et al., ) or by utilizing another LLM as an evaluative judge  (Bai et al., ; Cui et al., ). RL-free algorithms, such as Direct Preference Optimization (DPO) () and its variants  (; Ethayarajh et al., ), can directly optimize human preferences by introducing an implicit reward. DPO offers a stable and computationally lightweight method to align with human intent. To address the potential overfitting problem in DPO, Identity Preference Optimization (IPO)  () and Reward-aware Preference (RPO) (Nvidia et al., ) has been proposed by introducing differences between pairwise responses. Furthermore, Kahneman-Tversky Optimization (KTO)  (Ethayarajh et al., ), was proposed by directly maximizing the utility of generations instead of maximizing the log-likelihood of preferences, and  introduced a reference model-free monolithic odds ratio preference optimization algorithm. Some concurrent works attempt to introduce intrinsic knowledge constraints into preference alignment, either by on-the-fly fine-tuning LLMs to obtain relative qualities  (Yu et al., ) or by defining a reward distance  (Nvidia et al., ; Amini et al., ). These studies overlook the potential of leveraging tied preference data to refine alignment, whereas our approach is complementary to most of them.

Ternary preference models. Ties frequently occur in ranked data, such as sports and examinations. For example, a soccer result can be classified simply as a home win, an away win, or a tie. The well-known BT model, which can be derived from the order statistics of the exponential distribution, cannot handle ties.  Rao & Kupper () corrected the BT model by assuming that small probability difference values would be declared ties.  Kuk () applied this approach to football.  Davidson () provided an ad hoc correction to the BT model for ties.  Dewart & Gillard () applied the BT model to cricket, where draws occur but do not depend strongly on team strengths.  Baker & Scarf () used discrete distributions, principally the geometric distribution, to obtain a modified BT model that allows for tie ranks. These studies lay a robust theoretical foundation for addressing ties in ranking results, facilitating the connection between LLM alignment and preference data.

## 3 Preliminaries

### 3.1 The Bradley-Terry model

The Bradley-Terry (BT) (Bradley & Terry, ) model is a probability model for the outcome of pairwise comparisons between instances, teams, or objects. It estimates the probability that the rank order $i\succ j$ is true, where the symbol $\succ$ represents a preference or rank relation, such as instance $i$ being preferred over $j$, or $i$ beating $j$, depending on the application.

The computation of BT model can be represented by Equation  1, where the positive strength owned by two competitors are denoted by $\lambda_{1}$ and $\lambda_{2}$ respectively, and $r_{12}$ represents the probability that the first competitor obtains a higher rank than the second one in comparison.

$$ $r_{12}=\frac{\lambda_{1}}{\lambda_{1}+\lambda_{2}}$ (1) $$

Define $d_{12}=\ln\lambda_{1}-\ln\lambda_{2}$, which represents the strength difference of two competitors in logarithmic form. Following  Rao & Kupper (), Equation  1 can be written into the form of Equation  2, where $\text{sech}(\cdot)$ denotes the hyperbolic secant function. This relation shows that the preference probability $r_{12}$ depends only on $d_{12}$.

$$ $r_{12}=\frac{1}{4}\int_{-(\ln\lambda_{1}-\ln\lambda_{2})}^{\infty}\text{sech}^ {2}(y/2)\,dy=\frac{1}{4}\int_{-d_{12}}^{\infty}\text{sech}^{2}(y/2)\,dy$ (2) $$

### 3.2 Human preference modeling

Following the BT model, the human preference distribution $p^{*}$ is formulated in Equation  3, where $r^{*}$ is some latent reward model, $r^{*}(x,y)$ denotes the reward for response $y$ given an input prompt $x$, and $\sigma(\cdot)$ is the Sigmoid function. The variables $x$, $y_{1}$ and $y_{2}$ are drawn from a preference dataset $\mathcal{D}=\{x^{i},y_{1}^{i},y_{2}^{i}\}$, where $y_{1}$ is the preferred response and $y_{2}$ is the dispreferred one. The term $\exp(r^{*}(x,y))$ signifies the strength $\lambda$ of responses following Equation 1, acknowledging that the reward could be negative,

$$ $p^{*}(y_{1}\succ y_{2}|x)=\frac{\exp(r^{*}(x,y_{1}))}{\exp(r^{*}(x,y_{1}))+ \exp(r^{*}(x,y_{2}))}=\sigma(r^{*}(x,y_{1})-r^{*}(x,y_{2})).$ (3) $$

Subsequently, a reward model that mirrors human preferences can be trained using the method of maximum likelihood estimation (),

$$ $\max_{\theta}\mathbb{E}_{(x,y_{1},y_{2})\sim\mathcal{D}}\big{[}\ \log\sigma \big{(}r_{\theta}(x,y_{1})-r_{\theta}(x,y_{2})\big{)}\ \big{]}.$ (4) $$

### 3.3 Direct preference optimization

In RLHF (), the goal is to maximize the expectation of rewards under the KL divergence constraint,

$$ $\displaystyle\max_{\pi_{\theta}}\mathbb{E}_{x\sim D,y\sim\pi_{\theta}(y|x)}[r_ {\psi}(x,y)]-\beta\log\frac{\pi_{\theta}(y|x)}{\pi_{\text{ref}}(y|x)}.$ (5) $$

The optimal solution to this problem satisfies the following relationship ():

$$ $r(x,y)=\beta\log\frac{\pi^{*}(y|x)}{\pi_{\text{ref}}(y|x)}+\beta\log Z(x),$ (6) $$

where $Z(x)=\Sigma_{y}\pi_{\text{ref}}(y|x)\exp\big{(}\frac{1}{\beta}r(x,y)\big{)}$ only depends on prompt $x$.
By integrating this relation into Equation 4, the loss function of DPO can be expressed as shown in Equation  7, which is incapable of addressing tied preference data.

$$ $\displaystyle\mathcal{L}_{\mathrm{DPO}}(\pi_{\theta};\pi_{\text{ref}})$ $\displaystyle=-\mathbb{E}_{(x,y_{1},y_{2})\sim\mathcal{D}}\bigg{[}\log\sigma \Big{(}\beta\log\frac{\pi_{\theta}(y_{1}|x)}{\pi_{\text{ref}}(y_{1}|x)}-\beta \log\frac{\pi_{\theta}(y_{2}|x)}{\pi_{\text{ref}}(y_{2}|x)}\Big{)}\bigg{]}.$ (7) $$

## 4 Tie-rank oriented direct preference optimization

To handle ties in preferences modeling, we introduce a buffer in the integral interval of Equation 2, which is inspired by Rao & Kupper (). We call the new preference model Tie-rank Oriented
BT (TOBT) model. Based on this model, we propose a novel preference alignment algorithm, Tie-rank Oriented Direct Preference Optimization (TODO).

### 4.1 Tie-rank oriented BT model

As shown in Equation 2, $d_{12}$ represents the preference difference between two instances, and $d_{12}>0$ means the first instance is preferred. To handle ties, we impose higher requirements on the comparison by introducing a positive number $\alpha$, and then the preferred relation is determined by $d_{12}>\alpha$. That is, the ranking probability $r_{12}$ becomes Equation  8,

$$ $r_{12}=\frac{1}{4}\int_{-(\ln\lambda_{1}-\ln\lambda_{2})+\alpha}^{\infty}\text {sech}^{2}(y/2)\,dy$ (8) $$

Then, the probability that two instances share a tie relation is $1-r_{12}-r_{21}$, which is denoted by $r_{(12)}$, and can be expressed in Equation  9,

$$ $r_{(12)}=\frac{1}{4}\int_{-(\ln\lambda_{1}-\ln\lambda_{2})-\alpha}^{-(\ln \lambda_{1}-\ln\lambda_{2})+\alpha}\text{sech}^{2}(y/2)\,dy$ (9) $$

Intuitively, the parameter $\alpha$ encapsulates the inherent uncertainty and noise in the strength values $\lambda_{1}$ and $\lambda_{2}$, which are ubiquitous in LLM alignment. Preference datasets for alignment often rely on human labeling (Chiang et al., ), LLM-as-judge assessments (Cui et al., ), or reward model scoring (Nvidia et al., ), introducing label noise due to inconsistencies among human annotators or the inherent variability of LLMs in approximating human preferences. Accounting for this uncertainty and noise in responses evaluation likely contributes to TODO’s improved performance on binary preference datasets, as detailed in Section  6.

Following Equation 8 and 9, the TOBT model can be represented by Equation  10 and  11, where $\phi=\exp(\alpha)$ and $\phi>1$. The detailed derivation process is provided in Appendix A.2.

$$ $r_{12}=\frac{\lambda_{1}}{\lambda_{1}+\phi\lambda_{2}}$ (10) $$

$$ $r_{(12)}=\frac{\lambda_{1}\lambda_{2}(\phi^{2}-1)}{(\lambda_{1}+\phi\lambda_{2 })(\phi\lambda_{1}+\lambda_{2})}$ (11) $$

### 4.2 Objective function of TODO

Following Equation  10 and  11, the tie-rank oriented human preference distribution $p^{*}$ can be expressed by Equation  12 and Equation  13.

$$ $p^{*}(y_{1}\succ y_{2}|x)=\frac{\exp(r^{*}(x,y_{1}))}{\exp(r^{*}(x,y_{1}))+ \phi\exp(r^{*}(x,y_{2}))}$ (12) $$

$$ $p^{*}(y_{1}\equiv y_{2}|x)=\frac{\exp(r^{*}(x,y_{1}))\exp(r^{*}(x,y_{2}))(\phi ^{2}-1)}{\Big{(}\exp(r^{*}(x,y_{1}))+\phi\exp(r^{*}(x,y_{2}))\Big{)}\Big{(} \exp(r^{*}(x,y_{2}))+\phi\exp(r^{*}(x,y_{1}))\Big{)}}$ (13) $$

Equation  12 represents the possibility of treating $y_{1}$ as the preferred response and $y_{2}$ as the dispreferred response in pairwise data based on the TOBT model. Following Equation  12, the objective $\mathcal{L}^{p}_{\mathrm{TODO}}$ can be written as Equation  14, where $\mu=r_{\theta}(x,y_{1})-r_{\theta}(x,y_{2})$ represents the reward difference of two responses $y_{1}$ and $y_{2}$. Because the $Z(x)$ in implicit reward $r_{\theta}(x,y)$ only depends on $x$, the difference results of $\mu$ can be equivalently expressed as $\mu=\beta\log\frac{\pi_{\theta}(y_{1}|x)}{\pi_{\text{ref}}(y_{1}|x)}-\beta\log
\frac{\pi_{\theta}(y_{2}|x)}{\pi_{\text{ref}}(y_{2}|x)}$. The superscript $p$ in $\mathcal{L}^{p}_{\mathrm{TODO}}$ denotes that this formulation is the objective of TODO when the responses in a pair exhibit a clear preference, rather than being tied. The detailed derivation process is provided in Appendix  A.4.

$$ $\displaystyle\mathcal{L}^{p}_{\mathrm{TODO}}(\pi_{\theta};\pi_{\text{ref}})$ $\displaystyle=-\mathbb{E}_{(x,y_{1},y_{2})\sim\mathcal{D}}[\ \log\sigma(\mu- \alpha)]$ (14) $\displaystyle=-\mathbb{E}_{(x,y_{1},y_{2})\sim\mathcal{D}}\bigg{[}\ \log\sigma \Big{(}\beta\log\frac{\pi_{\theta}(y_{1}|x)}{\pi_{\text{ref}}(y_{1}|x)}-\beta \log\frac{\pi_{\theta}(y_{2}|x)}{\pi_{\text{ref}}(y_{2}|x)}-\alpha\Big{)}\ \bigg{]}.$ $$

Equation  13 represents the possibility of treating pairwise responses $y_{1}$ and $y_{2}$ as a tied pair based on the TOBT model. Following Equation  13, the objective $\mathcal{L}^{t}_{\mathrm{{TODO}}}$ can be written as Equation  15. The superscript $t$ in $\mathcal{L}^{t}_{\mathrm{TODO}}$ signifies that this represents TODO’s objective when the pair is tied. The detailed derivation process is is provided in Appendix A.5.

$$ $\begin{split}\mathcal{L}^{t}_{\mathrm{TODO}}(\pi_{\theta};\pi_{\text{ref}})&=- \mathbb{E}_{(x,y_{1},y_{2})\sim\mathcal{D}}\bigg{[}\ \log\frac{\exp(2\alpha)-1 }{(1+\exp(\mu+\alpha))(1+\exp(-\mu+\alpha))}\ \bigg{]}\end{split}$ (15) $$

Given a preference dataset $(x^{i},y_{1}^{i},y_{2}^{i},\mathbb{I}^{i})\in\mathcal{D}$, the indicator $\mathbb{I}^{i}$ is determined by Equation 16. Specifically, $\mathbb{I}^{i}=0$ indicates a clear preference or quality difference between the two responses to the same prompt $x$, while $\mathbb{I}^{i}=1$ represents that two responses $y_{1}^{i}$ and $y_{2}^{i}$ are tied. Then, the final loss $\mathcal{L}_{\mathrm{TODO}}$ of TODO can be represented by Equation 17.

$$ $\mathbb{I}^{i}=\begin{cases}1,&y_{1}^{i}\equiv y_{2}^{i},\\ 0,&y_{1}^{i}\succ y_{2}^{i}.\end{cases}$ (16) $$

$$ $\mathcal{L}_{\mathrm{TODO}}=(1-\mathbb{I})\mathcal{L}^{p}_{\mathrm{TODO}}+ \mathbb{I}\mathcal{L}^{t}_{\mathrm{TODO}}$ (17) $$

### 4.3 The effect of different preference relations on TODO updates

To elucidate the dynamics of TODO parameter updates, we introduce a gradient-based analysis that distinguishes between scenarios where pairwise responses show a clear preference or are tied. The comprehensive derivation is detailed in Appendices A.6 and A.7.

For the gradient update (18) for pairwise data with a clear preference, the introduction of a positive small value $\alpha$ results in more substantial weight adjustments when the reward difference is misestimated compared to DPO. This refinement mitigates noise from narrow reward margins, allowing TODO to more effectively learn distinct preferences by concurrently enhancing the likelihood of the favored response $y_{1}$ and diminishing that of the unfavored response $y_{2}$.

$$ $\nabla_{\theta}\mathcal{L}^{p}_{\mathrm{TODO}}(\pi_{\theta};\pi_{\text{ref}})= -\mathbb{E}_{(x,y_{1},y_{2})\sim\mathcal{D}}\bigg{[}\underbrace{\beta\sigma(- \mu+\alpha)}_{\text{higher weight than DPO}}\bigg{[}\nabla_{\theta}\log(\pi(y_ {1}|x))-\nabla_{\theta}\log(\pi(y_{2}|x))\bigg{]}\bigg{]}.$ (18) $$

For the gradient update (19) for pairwise tie data, $G(\mu)=\frac{\exp(-\mu+\alpha)-\exp(\mu+\alpha)}{(1+\exp(-\mu+\alpha))(1+\exp(
\mu+\alpha))}$ is monotonically decreasing with respect to $\mu$, and $G(0)=0$. When $\mu=0$, two responses obtain the same rewards, and DPO continues to update policy models as per Equation 7, which shifts the distribution to reduce the likelihood of the “dispreferred” response $y_{2}$, potentially discarding valuable information. In contrast, TODO refrains from updating any parameters to maintain the consistent preference alignment of the tied responses.

When the estimated reward difference for tied responses is $\mu>0$, suggesting $y_{1}$ has a higher reward than $y_{2}$, TODO’s gradient update strategy will reduce the likelihood of $y_{1}$ and increase that of $y_{2}$. Conversely, if $\mu<0$, the update will elevate the likelihood of $y_{1}$ and lower that of $y_{2}$, ensuring the preference consistency between the tied responses is preserved.

$$ $\nabla_{\theta}\mathcal{L}^{t}_{\mathrm{TODO}}(\pi_{\theta};\pi_{\text{ref}})= -\mathbb{E}_{(x,y_{1},y_{2})\sim\mathcal{D}}\bigg{[}\underbrace{G(\mu)\nabla_{ \theta}\log(\pi(y_{1}|x))}_{\text{part 1}}+\underbrace{G(-\mu)\nabla_{\theta} \log(\pi(y_{2}|x))}_{\text{part 2}}\bigg{]}.$ (19) $$

## 5 Experimental settings

### 5.1 Models and datasets

Models. We select two different series of models, Mistral-7B (Jiang et al., ) and Llama 3-8B (AI@Meta, ), as our experimental backbone models. We select zephyr-sft-full(^2^22[https://huggingface.co/alignment-handbook/zephyr-7b-sft-full](https://huggingface.co/alignment-handbook/zephyr-7b-sft-full)) ()
as the supervised fine-tuning (SFT) version of Mistral model and llama3-8b-sft-ultrachat(^3^33[https://huggingface.co/kykim0/llama3-8b-ultrachat-sft](https://huggingface.co/kykim0/llama3-8b-ultrachat-sft))
as the SFT version model of Llama 3 model. Both zephyr-sft-full and llama3-8b-sft-ultrachat are fine-tuned on Ultrachat-200k (Ding et al., ) dataset.

Training datasets. For the datasets used in the preference alignment process, we construct 20k-size datasets with different tie data proportions from Ultrafeedback (Cui et al., ). Responses sharing the same quality score are classified as tied. The quality score for each response is a weighted score across multiple assessment metrics, taking into account helpfulness, truthfulness, verbalized calibration and honesty. Each sampled dataset exhibits a tie data ratio that varies within the set {0, 0.1, 0.2, 0.3}. Other details of the training sets are shown in Appendix A.8.

Evaluation benchmarks. We first compare the effectiveness of preference modeling ability between DPO and TODO. To this end, we curate an in-distribution test set containing 1500 non-tied samples and select the Reward Bench () as an out-of-distribution dataset. Subsequently, we select a suite of well-established benchmark datasets to evaluate the models comprehensively: 1) MT Bench (Zheng et al., ), which contains open-ended questions designed to assess the multi-turn conversational capabilities and the ability to follow instructions of LLMs. 2) A diverse set of benchmarks containing Piqa (Bisk et al., ), ARC-c, ARC-e (Clark et al., ), Hellaswag (Zellers et al., ), MMLU (Hendrycks et al., ) and Winogrande  (Sakaguchi et al., ), which collectively evaluate the language comprehension and reasoning faculties of LLMs.

### 5.2 Training settings

All comparative results are derived from training each model for 3 epochs on their respective training datasets. We set $\alpha=0.5$ in TODO, which follows the principle of balancing the loss value of pairwise non-tie data and tie data. Please refer to Appendix  A.3 for more details. Other hyperparameters are shown in Appendix  A.9. We ensure the consistency of training hyperparameters among experiments for a fair comparison.

### 5.3 Evaluation settings

Accuracy of preference modeling. In assessing the efficacy of models fine-tuned with DPO and TODO for preference modeling, we employ prediction accuracy as the primary evaluation metric. For each pair of data where the preference rank $y_{1}$ is favored over $y_{2}$, we calculate the predicted probabilities for each preference rank for pair data, adhering to Equations 12 and 13. A prediction is deemed accurate if the model assigns the highest probability to the scenario where $y_{1}$ is preferred over $y_{2}$ across all possible ranks.

GPT based scoring in MT Bench. For the evaluation in MT Bench, we use gpt-4-turbo-2024-04-09 (OpenAI, ) to score generated results.

Accuracy of other benchmarks. For the evaluation of Piqa, ARC-c, ARC-e, Hellaswag, MMLU and Winogrande, we use Opencompass (Contributors, ) to assess final results, details of prompt templates and evaluation metrics can be found in Appendix 7.

## 6 Results and analysis

### 6.1 TODO improves human preference modeling with tie data

In this section, we assess the preference modeling capabilities of models trained with DPO and TODO across both in-distribution and out-of-distribution datasets, incorporating varying proportions of tie data in the training regimen. The in-distribution assessment is based on the above mentioned test set, while the out-of-distribution assessment utilizes the Reward Bench.

Figure: Figure 2: Accuracy of Mistral and Llama 3 models aligned with DPO and TODO on non-tie preference test set and Reward Bench. The X-axis denotes the proportion of tie data mixed in train set.
Refer to caption: x2.png

Figures 2 illustrates the accuracy results of Mistral and Llama 3 models aligned with DPO and TODO on the test set and Reward Bench, using training sets with varying proportions of tie data. Reward Bench contains pairwise responses divided into Chat, Chat hard, Safety, Reasoning and Prior preference data categories. We compute the average accuracy score of all categories to represent the final performance on Reward Bench. The detailed scores in each category of two series models on Reward Bench are provided in Appendix A.12.

We observe that both Mistral and Llama 3 models aligned with TODO generally achieve better performance than those aligned with DPO on both in-distribution and out-of-distribution data. Overall, when directly modeling human preferences mixing tie data, DPO often leads to sub-optimal results. TODO addresses this issue with more nuanced preference modeling. Experimental results demonstrate the effectiveness of the combinatorial optimization objectives in TODO.

### 6.2 TODO enhances alignment in LLMs

In this section, we evaluate the effectiveness of DPO and TODO across different benchmarks to comprehensively demonstrate the alignment capabilities of LLMs.

As illustrated in Figures 3a and 3b, the scores on the MT Bench for models aligned with DPO and TODO reveal that TODO consistently outperforms DPO across all training sets for both the Mistral and Llama 3 models. Specifically, TODO achieves peak performance when using the binary preference dataset for the Mistral models and incorporating a 20% tie data ratio for the Llama 3 models, respectively.

Figure: (a) Results on Mistral-7B
Refer to caption: x3.png

Table 2 and Table 3 respectively show the results on Piqa, ARC-c, ARC-e, Hellaswag, MMLU and Winogrande benchmarks of the Mistral and Llama 3 models, where we highlight the best performance in underline, and mark the better performance between DPO and TODO with different ratio of tie data in bold. As demonstrated in these tables, TODO achieves better performance than DPO across all train sets on both two models, and obtain the best average performance when mixing 20% and 30% of tie data in Mistral and Llama 3 serials models, respectively.

Optimal proportion of tie data enhances alignment. Our experimental findings across various benchmarks underscore the critical role and efficacy of TODO in managing non-binary preference data. Notably, optimal performance on the MT Bench and six additional benchmarks is consistently achieved when a specific ratio of tie data is integrated with TODO. This observation indicates that the strategic incorporation of tie data, in conjunction with TODO, can markedly enhance the alignment capabilities of LLMs.

**Table 2: Results of Mistral-7B aligned with different methods trained with various ratios of tie data.**
| Method | Tie Data Ratio | Piqa | ARC-c | ARC-e | MMLU | Hellaswag | Winogrande | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| SFT | ✗ | 80.09 | 51.33 | 74.97 | 60.26 | 75.69 | 73.64 | 69.33 |
| DPO | 0.0 | 77.15 | 49.87 | 69.56 | 59.44 | 73.55 | 67.88 | 66.24 |
| TODO | 0.0 | 81.01 | 54.59 | 78.44 | 60.16 | 79.01 | 72.93 | 71.02 |
| DPO | 0.1 | 79.87 | 53.82 | 77.21 | 60.07 | 79.04 | 73.48 | 70.58 |
| TODO | 0.1 | 80.25 | 54.68 | 77.76 | 60.46 | 79.27 | 73.95 | 71.06 |
| DPO | 0.2 | 80.25 | 55.02 | 77.51 | 59.79 | 79.08 | 73.01 | 70.78 |
| TODO | 0.2 | 81.07 | 55.11 | 78.52 | 59.40 | 79.14 | 73.80 | 71.17 |
| DPO | 0.3 | 75.73 | 48.15 | 68.84 | 59.41 | 71.97 | 69.30 | 65.57 |
| TODO | 0.3 | 77.37 | 51.59 | 71.75 | 59.79 | 74.36 | 69.38 | 67.37 |

**Table 3: Results of Llama 3-8B aligned with different methods with various ratios of tie data.**
| Method | Tie Data Ratio | Piqa | ARC-c | ARC-e | MMLU | Hellaswag | Winogrande | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| SFT | ✗ | 79.65 | 55.02 | 78.65 | 63.89 | 76.45 | 72.69 | 71.06 |
| DPO | 0.0 | 79.27 | 54.59 | 76.58 | 63.29 | 79.05 | 71.74 | 70.75 |
| TODO | 0.0 | 79.43 | 55.45 | 76.79 | 63.45 | 79.27 | 71.82 | 71.03 |
| DPO | 0.1 | 79.65 | 54.68 | 75.48 | 63.46 | 79.08 | 71.74 | 70.68 |
| TODO | 0.1 | 79.54 | 55.19 | 76.28 | 63.65 | 79.05 | 71.43 | 70.86 |
| DPO | 0.2 | 79.16 | 54.59 | 75.60 | 63.49 | 78.72 | 70.96 | 70.42 |
| TODO | 0.2 | 79.38 | 56.22 | 77.08 | 63.07 | 79.04 | 71.11 | 70.98 |
| DPO | 0.3 | 79.05 | 55.54 | 76.11 | 63.34 | 78.17 | 70.88 | 70.52 |
| TODO | 0.3 | 79.33 | 56.22 | 77.76 | 63.61 | 78.83 | 71.03 | 71.13 |

TODO’s superiority in binary preference alignment. In addition to enhancing performance by incorporating tie data into the training regimen, TODO can also be effectively employed in scenarios involving purely binary preference alignment. The experimental outcomes, as depicted in Figure 3 for the MT Bench and Tables 2 and 3 for other benchmarks, consistently indicate that models fine-tuned with TODO outperform their counterparts optimized with DPO, even in the absence of any tie data.

### 6.3 Key factors of TODO in handling tie data and obtaining better performance

In this section, we analyse the limitations of DPO and the advantages of TODO in handling tie data. We highlight two pivotal factors that TODO can enhance alignment: nuanced preference modeling and enriched diversity of information learned.

TODO refines DPO by accurately handling tied data. We consider the scenario where responses of equivalent quality are incorrectly labeled as binary preference data, as illustrated in Table 1. As stated in Section 4.2, $\mu$ represents the implicit reward margin between two responses during the training. For the $i$-th pair of responses, $y_{1}^{i}$ and $y_{2}^{i}$, which are essentially tied, DPO assigns a small value to $\mu^{i}$. Despite the trivial value of $\mu^{i}$, DPO unilaterally steers the policy model update by increasing the likelihood of one tied response $y_{1}^{i}$, while concurrently decreasing the likelihood of the other $y_{2}^{i}$. This biased update is a consequence of the binary BT model, following the gradient computation:

$$ $\nabla_{\theta}\mathcal{L}_{\mathrm{DPO}}=-\mathbb{E}_{(x,y_{1},y_{2})\sim \mathcal{D}}\bigg{[}\underbrace{\beta\sigma(-\mu)}_{\text{always positive}} \bigg{[}\nabla_{\theta}\log(\pi(y_{1}|x))-\nabla_{\theta}\log(\pi(y_{2}|x)) \bigg{]}\bigg{]}.$ (20) $$

The gradient update direction as shown in Equation 20 is decided by the Sigmoid function and a positive $\beta$ value, which makes DPO always update policy model in one direction. Such a gradient computation, when applied to tied data, can result in erroneous updates and introduce noise into the binary preference modeling process, potentially culminating in a suboptimal policy. Both DPO and TODO assign a small reward margin for tied pair responses, as compared to utilizing data with clear preference distinction. However, TODO employs combinatorial optimization objectives for pairwise responses accounting for the uncertainty of preference difference, and effectively rectifying the issues DPO faces with tied data. This changes the update direction depending on the exact preference relation, ensuring preference learning consistency between two tied responses. Concurrently, the gradient computation for non-tied responses facilitates the learning of preferred or dispreferred relations. This distinction is crucial for effective preference modeling and can learn “prefer”, “tie” and “disprefer” ranks. The exact reward margin changes of two models aligned with DPO and TODO using different ratios of tie data are provided in Appendix A.11 to support our analysis.

Learning from diverse tied responses improves the performance.
Both DPO and TODO are adept at ranking pairwise data with clear preference distinctions, distinguishing between “preferred” and “dispreferred” responses. However, when responses exhibit no clear preference or quality disparity, DPO’s optimization strategy compels the policy model to skew towards one response at the expense of the other, resulting in unnecessary information loss for the unfavored response. TODO mitigates this by incorporating a tie rank and a novel optimization objective, enabling the policy model to evolve in accordance with the consistent preference trends of both responses. This mechanism allows both the content and format of the two responses to be learned simultaneously, enabling the model to capture more diverse information from the same amount of data.

## 7 Conclusion and future work

This study illuminates a limitation in LLM preference alignment: the inability of binary models like DPO to resolve ties in preference data. To overcome this, we integrate a tie ranking system into preference modeling, refine the BT model into the more robust TOBT model, and introduce the TODO algorithm. Our experimental results demonstrate that TODO consistently outperforms DPO across a range of evaluations. The success of TODO stems from its nuanced handling of ties and the robust TOBT model. This approach is not only limited to direct preference optimization but is also compatible with various online and offline preference optimization policies and can be employed for reward model training, which are left for future work.

## Appendix A Appendix

### A.1 Tie data in current preference datasets

We conduct a statistical analysis of existing tie data across common preference data, as presented in Table 4. Current pairwise preference data are scored by LLMs or labeled by humans to establish preference rankings. Preference data scored by LLMs, commonly rely on the quality score considering multiple aspects to differentiate between preferred and dispreferred responses. Ultrafeedback  (Cui et al., ), as a popular preference dataset, is scored based on GPT-4 feedback, and contains 383k pairwise responses. We find that 17.0% of the data pairs exhibit identical quality scores. The Ultrafeedback$_$binaried  (), a subset of this dataset, also includes 12.1% of tie data. This analysis underscores the prevalence of ties in preference datasets.

**Table 4: Statistics on common preference datasets.**
| Datasets | Tie data(%) |
| --- | --- |
| UltraFeedback | 17.0 |
| Ultrafeedback binarized | 12.1 |
| lmsys-chatbot$_$arena$_$conversations | 29.4 |

Unlike datasets featuring explicit quality scores, some preference datasets include only human-provided preference labels. Figure 4 illustrates two pairs pf responses from the Chatbot Arena(^4^44[https://lmsys.org/blog/2023-05-03-arena/](https://lmsys.org/blog/2023-05-03-arena/)) (Chiang et al., ), judged by different users. Specifically, user793 prefers R1 and disfavors R2 for prompt 1. Conversely, user1245 exhibits no clear preference between the two responses for prompt 2, categorizing them as tied. The lmsys-chatbot$_$arena$_$conversations dataset, which collects the pairwise data from Chatbot Arena, reveals that 29.4% of the total samples are tied. This statistic highlights the significant presence of tie data in human-labeled preference data.

Figure: Figure 4: Human evaluation of pairwise responses across various prompts in the Chatbot Arena.
Refer to caption: x5.png

### A.2 The TOBT model

By substituting $t=\frac{y}{2}$ into $\frac{1}{4}\int\operatorname{\text{sech}}^{2}(\frac{y}{2})dy$, we obtain that

$$ $\displaystyle F(t)$ $\displaystyle=\frac{1}{2}\int\operatorname{\text{sech}}^{2}(t)dt=\frac{1}{2} \tanh{(t)},$ (21) $$

where $\tanh(t)=\frac{\exp(t)-\exp(-t)}{\exp(t)+\exp(-t)}$.
Then $r_{12}=\frac{1}{4}\int_{-(\ln\lambda_{1}-\ln\lambda_{2})+\alpha}^{\infty}\text
{sech}^{2}(y/2)\,dy$ can be written into $r_{12}=\frac{1}{2}\int_{t_{1}}^{\infty}\operatorname{\text{sech}}^{2}\left(t
\right)dt$, where $t_{1}=\frac{-(\ln\lambda_{1}-\ln\lambda_{2})+\alpha}{2}$. We can obtain the representation of $r_{12}$ following Equation  22:

$$ $\displaystyle r_{12}$ $\displaystyle=\frac{\tanh(\infty)-\tanh{(t_{1})}}{2}$ (22) $\displaystyle=\frac{1-\tanh{(t_{1})}}{2}$ $\displaystyle=\frac{1-\frac{\exp(t_{1})-\exp(-t_{1})}{\exp(t_{1})+\exp(-t_{1}) }}{2}$ $\displaystyle=\frac{1-\frac{\exp(2t_{1})+\exp(-2t_{1})-2}{\exp(2t_{1})-\exp(-2 t_{1})}}{2}$ $\displaystyle=\frac{1-\exp(-2t_{1})}{\exp(2t_{1})-\exp(-2t_{1})}$ $\displaystyle=\frac{1-\exp(\ln\lambda_{1}-\ln\lambda_{2}-\alpha)}{\exp(\ln \lambda_{2}-\ln\lambda_{1}+\alpha)-\exp(\ln\lambda_{1}-\ln\lambda_{2}-\alpha)}.$ $$

By substituting $\alpha$ with $\ln\phi$ in the last line of Equation  22, the final presentation of $r_{12}$ can be represented by Equation  23:

$$ $\displaystyle r_{12}$ $\displaystyle=\frac{1-\exp(\ln\lambda_{1}-\ln\lambda_{2}-\ln\phi)}{\exp(\ln \lambda_{2}-\ln\lambda_{1}+\ln\phi)-\exp(\ln\lambda_{1}-\ln\lambda_{2}-\ln\phi)}$ (23) $\displaystyle=\frac{1-\frac{\lambda_{1}}{\phi\lambda_{2}}}{\frac{\phi\lambda_{ 2}}{\lambda_{1}}-\frac{\lambda_{1}}{\phi\lambda_{2}}}$ $\displaystyle=\frac{\lambda_{1}}{\lambda_{1}+\phi\lambda_{2}}.$ $$

For the $r_{(12)}=\frac{1}{4}\int_{-(\ln\lambda_{1}-\ln\lambda_{2})-\alpha}^{-(\ln
\lambda_{1}-\ln\lambda_{2})+\alpha}\operatorname{\text{sech}}^{2}\left(\frac{y
}{2}\right)dy$, we can rewrite it into $r_{(12)}=\frac{1}{2}\int_{t_{2}}^{t_{1}}\operatorname{\text{sech}}^{2}\left(t
\right)dt$, in which $t_{1}=\frac{-(\ln\lambda_{1}-\ln\lambda_{2})+\alpha}{2}$ and $t_{2}=\frac{-(\ln\lambda_{1}-\ln\lambda_{2})-\alpha}{2}$. Then we compute $r_{(12)}$ according to Equation 24:

$$ $\displaystyle r_{(12)}$ $\displaystyle=\frac{1}{2}(\tanh(t_{1})-\tanh(t_{2}))$ (24) $\displaystyle=\frac{(\exp(t_{1})-\exp(-t_{1})(\exp(t_{2})+\exp(-t_{2})))-(\exp (t_{2})-\exp(-t_{2}))(\exp(t_{1})+\exp(-t_{1}))}{2(\exp(t_{1})+\exp(-t_{1}))( \exp(t_{2})+\exp(-t_{2}))}$ $\displaystyle=\frac{\exp(t_{1}-t_{2})-\exp(t_{2}-t_{1})}{\exp(t_{1}+t_{2})+ \exp(t_{1}-t_{2})+\exp(t_{2}-t_{1})+\exp(-t_{1}-t_{2})}$ $\displaystyle=\frac{\exp(\alpha)-\exp(-\alpha)}{\exp(-(\ln\lambda_{1}-\ln \lambda_{2}))+\exp(\alpha)+\exp(\alpha)+\exp(\ln\lambda_{1}-\ln\lambda_{2})}.$ $$

By substituting $\alpha$ with $\ln\phi$ in the last line of Equation 24, we can obtain the final representation of $r_{(12)}$ as given in Equation 25:

$$ $\displaystyle r_{(12)}$ $\displaystyle=\frac{\phi-\frac{1}{\phi}}{\frac{\lambda_{2}}{\lambda_{1}}+\phi+ \frac{1}{\phi}+\frac{\lambda_{1}}{\lambda_{2}}}$ (25) $\displaystyle=\frac{\lambda_{1}\lambda_{2}(\phi^{2}-1)}{\phi\lambda_{2}^{2}+ \phi^{2}\lambda_{1}\lambda_{2}+\lambda_{1}\lambda_{2}+\phi\lambda_{1}^{2}}$ $\displaystyle=\frac{\lambda_{1}\lambda_{2}(\phi^{2}-1)}{(\lambda_{1}+\phi \lambda_{2})(\lambda_{2}+\phi\lambda_{1})}.$ $$

### A.3 Selection of α 𝛼 \alpha italic_α in TODO

In this section, we elaborate on the process of selecting the optimal value for $\alpha$ in TODO. Initially, we generate a series of random values to emulate the expression $\beta\log\frac{\pi_{\theta}(y_{1}|x)}{\pi_{\text{ref}}(y_{1}|x)}-\beta\log
\frac{\pi_{\theta}(y_{2}|x)}{\pi_{\text{ref}}(y_{2}|x)}$, which represents the reward difference between two responses in the initial stage of preference alignment. Subsequently, we calculate the corresponding pairwise not-tied and tied losses across a range of $\alpha$ values. Figures 5a and 5b present the visualization of the results for two distinct value spans, illustrating the impact of $\alpha$ on the losses.

Figure: (a) $\alpha$ varies from 1 to 10

A delicate balance between the non-tie loss and the tie loss is beneficial to the TODO performance. Our observations indicate that the non-tie and tie losses are interconnected through a power-law relation. As depicted in Figure 5, the preference loss sharply ascends with increasing values of $\alpha$, exceeding the concurrent growth in tie loss.

To achieve equilibrium between these two types of losses and to identify an appropriate value of $\alpha$ that maintains the preference loss near the levels observed in the original DPO (which implies that $\alpha$ should not be markedly distant from zero), we establish two thresholds: the initial tie loss in TODO is restricted to a maximum of 1.5, and the initial preference loss is restricted to a maximum of 1.0. These constraints are instrumental in guiding the selection of $\alpha$, ensuring that the optimization process effectively reconciles the competing objectives of preference and tie loss minimization.

Initially, we graph the variations in both preference and tie losses across a spectrum of $\alpha$ values, ranging from 1 to 10, as illustrated in Figure 5a. This visualization reveals that the preference loss using TODO is consistently higher than desired. Subsequently, we refine our approach by focusing on a narrower band of $\alpha$ values, specifically from 0.01 to 0.1. The detailed outcomes of this refined analysis are shown in Figure 5b.

Our findings indicate that within the range of $\alpha\geq 0.5$ and $\alpha\leq 1$, both type of losses align with the established thresholds. That is, the preference loss does not exceed 1.0, and the tie loss remains below 1.5. Based on these criteria, we opt for $\alpha=0.5$ as it falls within the acceptable span and meets the requirements for effective optimization objective balancing in TODO. This selection ensures that the model’s optimization targets are well-calibrated for performance enhancement.

### A.4 The objective function of TODO with pairwise distinguished preference data

It is straightforward to derive the TODO objective in pairwise data with distinct preference difference. Under the TOBT model as Equation  12, we can obtain the possibility of $y_{1}$ being preferred over $y_{2}$ following Equation  26 by substituting $\phi$ into $\exp(\alpha)$.

$$ $\displaystyle p^{*}(y_{1}\succ y_{2}|x)$ $\displaystyle=\frac{\exp(r^{*}{(x,y_{1})})}{\exp(r^{*}{(x,y_{1})})+\phi\exp(r^ {*}{(x,y_{2})})}$ (26) $\displaystyle=\frac{\exp(r^{*}{(x,y_{1})})}{\exp(r^{*}{(x,y_{1})})+\exp(\alpha )\exp(r^{*}{(x,y_{2})})}$ $\displaystyle=\frac{1}{1+\exp(\alpha)\exp(r^{*}(x,y_{2})-r^{*}(x,y_{1}))}$ $\displaystyle=\frac{1}{1+\exp(r^{*}(x,y_{2})-r^{*}(x,y_{1})+\alpha)}$ $\displaystyle=\sigma(r^{*}(x,y_{1})-r^{*}(x,y_{2})-\alpha)$ $$

Recall that the (unavailable) ground-truth reward through is given as follows:

$$ $r^{*}(x,y)=\beta\log\frac{\pi^{*}(y|x)}{\pi_{\text{ref}}(y|x)}+\beta\log Z(x).$ (27) $$

Substituting Equation  27 into Equation  26, we derive the per-instance preference possibility as shown in Equation  28.

$$ $p^{*}(y_{1}\succ y_{2}|x)=\sigma\Big{(}\beta\log\frac{\pi^{*}(y_{1}|x)}{\pi_{ \text{ref}}(y_{1}|x)}-\beta\log\frac{\pi^{*}(y_{2}|x)}{\pi_{\text{ref}}(y_{2}| x)}-\alpha\Big{)}$ (28) $$

### A.5 The objective function of TODO with pairwise tie data

For instances of pairwise tie data, by utilizing the TOBT model as defined in Equation  13, we can compute the possibility that $y_{1}$ and $y_{2}$ are tied, as shown in Equation  29:

$$ $\displaystyle p^{*}(y_{1}\equiv y_{2}|x)$ $\displaystyle=\frac{\exp(r^{*}(x,y_{1}))\exp(r^{*}(x,y_{2}))(\phi^{2}-1)}{( \exp(r^{*}(x,y_{1}))+\phi\exp(r^{*}(x,y_{2})))(\exp(r^{*}(x,y_{2}))+\phi\exp(r ^{*}(x,y_{1})))}$ (29) $\displaystyle=\frac{\exp(r^{*}(x,y_{1})+r^{*}(x,y_{2}))(\exp(2\alpha)-1)}{( \exp(r^{*}(x,y_{1}))+\exp(\alpha)\exp(r^{*}(x,y_{2})))(\exp(r^{*}(x,y_{2}))+ \exp(\alpha)\exp(r^{*}(x,y_{1})))}.$ $$

Which can be transferred into:

$$ $\displaystyle p^{*}(y_{1}\equiv y_{2}|x)$ $\displaystyle=\frac{\exp(2\alpha)-1}{1+\exp(r^{*}(x,y_{1})-r^{*}(x,y_{2})+ \alpha)+\exp(r^{*}(x,y_{2})-r^{*}(x,y_{1})+\alpha)+\exp(2\alpha)},$ (30) $$

by dividing $\exp(r^{*}(x,y_{1})+r^{*}(x,y_{2}))$ in both the numerator and denominator.
Because $\exp(2\alpha)$ can be expressed by $\exp(r^{*}(x,y_{1})-r^{*}(x,y_{2})+\alpha+r^{*}(x,y_{2})-r^{*}(x,y_{1})+\alpha)$, we can rewrite the denominator of the last line in Equation  30 into $(1+\exp(r^{*}(x,y_{1})-r^{*}(x,y_{2})+\alpha))(1+\exp(r^{*}(x,y_{2})-r^{*}(x,y
_{1})+\alpha))$. Then we can obtain Equation  31.

$$ $p^{*}(y_{1}\equiv y_{2}|x)=\frac{\exp(2\alpha)-1}{(1+\exp(r^{*}(x,y_{1})-r^{*} (x,y_{2})+\alpha))(1+\exp(r^{*}(x,y_{2})-r^{*}(x,y_{1})+\alpha))}$ (31) $$

Substituting Equation  27 into Equation  31, we obtain the per-instance tie possibility as shown in Equation  32, where $\mu=\beta\log\frac{\pi^{*}(y_{1}|x)}{\pi_{\text{ref}}(y_{1}|x)}-\beta\log\frac
{\pi^{*}(y_{2}|x)}{\pi_{\text{ref}}(y_{2}|x)})$.

$$ $p^{*}(y_{1}\equiv y_{2}|x)=\frac{\exp(2\alpha)-1}{(1+\exp(\mu+\alpha))(1+\exp( -\mu+\alpha))}.$ (32) $$

### A.6 The gradient of TODO with pairwise distinguished preference data

The gradient of TODO with pairwise distinguished preference data can be expressed by following equation:

$$ $\nabla_{\theta}\mathcal{L}^{p}_{\mathrm{TODO}}(\pi_{\theta};\pi_{\text{ref}})= -\nabla_{\theta}\mathbb{E}_{(x,y_{1},y_{2})\sim\mathcal{D}}[\ \log\sigma(\mu- \alpha)\ ],$ (33) $$

where $\mu=\beta\log\frac{\pi_{\theta}(y_{1}|x)}{\pi_{\text{ref}}(y_{1}|x)}-\beta\log
\frac{\pi_{\theta}(y_{2}|x)}{\pi_{\text{ref}}(y_{2}|x)}$.
Then Equation  33 can be written into following form:

$$ $\nabla_{\theta}\mathcal{L}^{p}_{\mathrm{TODO}}(\pi_{\theta};\pi_{\text{ref}})= -\mathbb{E}_{(x,y_{1},y_{2})\sim\mathcal{D}}[\frac{\sigma^{{}^{\prime}}(\mu- \alpha)}{\sigma(\mu-\alpha)}\nabla_{\theta}(\mu)].$ (34) $$

Using the properties of Sigmoid function $\sigma^{{}^{\prime}}(x)=\sigma(x)(1-\sigma(x))$ and $\sigma(-x)=1-\sigma(x)$, we obtain the final gradient,
$\nabla_{\theta}\mathcal{L}^{p}_{\mathrm{TODO}}(\pi_{\theta};\pi_{\text{ref}})=
-\mathbb{E}_{(x,y_{1},y_{2})\sim\mathcal{D}}[\beta\sigma(\beta\log\frac{\pi_{
\theta}(y_{2}|x)}{\pi_{\text{ref}}(y_{2}|x)}-\beta\log\frac{\pi_{\theta}(y_{1}
|x)}{\pi_{\text{ref}}(y_{1}|x)}+\alpha)[\nabla_{\theta}\log(\pi(y_{1}|x))-
\nabla_{\theta}\log(\pi(y_{2}|x))]]$.

### A.7 The gradient of TODO with pairwise tie data

The gradient of TODO with pairwise tie data can be expressed into following form:

$$ $\nabla_{\theta}\mathcal{L}^{t}_{\mathrm{TODO}}(\pi_{\theta};\pi_{\text{ref}})= -\nabla_{\theta}\mathbb{E}_{(x,y_{1},y_{2})\sim\mathcal{D}}[\ \log(\frac{\exp( 2\alpha)-1}{f(\mu)})],$ (35) $$

where $f(\mu)=(1+\exp(-\mu+\alpha))(1+\exp(\mu+\alpha))$, and $\mu=\beta\log\frac{\pi_{\theta}(y_{1}|x)}{\pi_{\text{ref}}(y_{1}|x)}-\beta\log
\frac{\pi_{\theta}(y_{2}|x)}{\pi_{\text{ref}}(y_{2}|x)}$. Then Equation 35 can be rewritten into:

$$ $\nabla_{\theta}\mathcal{L}^{t}_{\mathrm{TODO}}(\pi_{\theta};\pi_{\text{ref}})= -\mathbb{E}_{(x,y_{1},y_{2})\sim\mathcal{D}}[\frac{f^{{}^{\prime}}(\mu)}{f(\mu )}\nabla_{\theta}(\mu)].$ (36) $$

We can derive the final gradient of TODO following Equation  37, where $G(\mu)=\frac{\exp(-\mu+\alpha)-\exp(\mu+\alpha)}{(1+\exp(\mu+\alpha))(1+\exp(-
\mu+\alpha)}$.

$$ $\nabla_{\theta}\mathcal{L}^{t}_{\mathrm{TODO}}(\pi_{\theta};\pi_{\text{ref}})= -\mathbb{E}_{(x,y_{1},y_{2})\sim\mathcal{D}}[G(\mu)[\nabla_{\theta}\log(\pi(y_ {1}|x))-\nabla_{\theta}\log(\pi(y_{2}|x))]]$ (37) $$

For any value of $\mu$, we have $G(\mu)=-G(-\mu)$, indicating $G(\mu)$ is an odd function. Additionally, $G^{\prime}(\mu)$, the first derivative of $G(\mu)$, is always negative derived from Equation  38, indicating $G(\mu)$ is monotonically decreasing with respect to $\mu$.

$$ $G^{\prime}(\mu)=\frac{-(\exp(-\mu+\alpha)+\exp(\mu+\alpha))(1+\exp(2\alpha)+2 \exp(-\mu+\alpha))}{(1+\exp(\mu+\alpha))^{2}(1+\exp(-\mu+\alpha)^{2}}.$ (38) $$

### A.8 Construction of the training datasets

The original Ultrafeedback dataset contains about 64k prompts from diverse resources, including UltraChat(Ding et al., ), ShareGPT (Chiang et al., ), Evol Instruct (Xu et al., ), TruthfulQA (Lin et al., ), FalseQA (Hu et al., ), and FLAN (Longpre et al., ). Each prompt is used to query multiple LLMs and generate 4 different responses, resulting in a total of 383k samples. Each response in pairwise data has its quality score provided by GPT-4 feedback. We sample 20k samples from these 383k samples to construct different training sets. To ensure consistency and fairness in data distribution, each sampling follows the original distribution of Ultrafeedback dataset, as shown in Table 5. We then construct datasets with different proportions of tie data, with the tie data ratios varying within the set $\{0,0.1,0.2,0.3\}$.

**Table 5: Data source distribution in each sampled train set.**
| Data source | Evol instruct | False QA | FLAN | Sharegpt | TruthfulQA | Ultrachat |
| --- | --- | --- | --- | --- | --- | --- |
| Percent(%) | 15.63 | 3.66 | 32.73 | 31.19 | 1.27 | 15.52 |

### A.9 Training hyperparameters

We use full-parameters fine-tuning when comparing different methods on Mistral-7B and Llama 3-8B models. We use Adam optimizer and the weight decay is set into 0. We use cosine learning rate scheduler, and the other detailed settings of DPO and TODO is shown in Table 6.

**Table 6: Training hyperparameters settings of DPO and TODO.**
| Model | Learning rate | Batch size | $\beta$ |
| --- | --- | --- | --- |
| Mistral+SFT | 5e-7 | 64 | 0.01 |
| Llama 3+SFT | 1e-6 | 128 | 0.01 |

### A.10 Evaluation metrics

For the evaluation based on OpenCompass, we use the default prompt template, and the specific metrics and evaluation mode settings are shown in Table 7. In this table, the PPL mode is used for multiple-choice tasks, utilizing the perplexity of each choice as the evaluation metric. The LL mode is used for the Winogrande task, where log likelihood is employed to evaluate task performance.

**Table 7: Evaluation details of multi downstream tasks, PPL represents accuracy based perplexity and LL represents accuracy based on log likelihood estimation.**
| Task | Piqa | ARC-c | ARC-e | MMLU | Hellaswag | Winogrande |
| --- | --- | --- | --- | --- | --- | --- |
| Mode | PPL | PPL | PPL | PPL | PPL | LL |
| Metric | 0-shot | 0-shot | 0-shot | 5-shot | 0-shot | 0-shot |

### A.11 Reward margin changes during the training process

Figure 6 illustrates the reward margin changes of Mistral+SFT and Llama 3+SFT models aligned with DPO and TODO. We observe that the growth of the reward margin in DPO decelerates as the proportion of tie data within the training set increases for both the Mistral model as shown in Figure 6a, and the Llama 3 model as shown in Figure 6b. For models aligned with TODO, a similar deceleration in the growth of the reward margin is noted with an increasing proportion of tie data in the training dataset, as shown in Figure 6c for the Mistral model and in Figure 6d for the Llama 3 model. Furthermore, models aligned with TODO exhibit a more pronounced disparity in reward margins when utilizing varying ratios of tie data, indicating a higher sensitivity of TODO to preference relations of the pairwise training data than DPO.

Figure: (a) Mistral series models aligned with DPO
Refer to caption: x8.png

### A.12 Detailed scores on Reward Bench

In this section, we show the detailed scores of each category on Reward Bench. For the results, we highlight the best performance over all results in underline, and mark the performance better aligning with different ratio of tie data in bold. Table 8 and Table 9 show the performance of the Mistral+SFT and Llama 3+SFT models aligned with DPO and TODO on Reward Bench.

**Table 8: Mistral model results on Reward Bench trained with different ratios of tie data.**
| Method | Tie Data Ratio | Chat | ChatHard | Safety | Reasoning | Prior | Average |
| --- | --- | --- | --- | --- | --- | --- | --- |
| SFT | ✗ | 66.48 | 54.61 | 35.73 | 62.94 | 40.20 | 51.99 |
| DPO | 0.0 | 91.34 | 66.45 | 75.79 | 72.36 | 49.23 | 71.03 |
| TODO | 0.0 | 93.85 | 64.25 | 76.88 | 72.17 | 61.07 | 73.64 |
| DPO | 0.1 | 91.9 | 67.98 | 73.64 | 72.18 | 53.67 | 71.87 |
| TODO | 0.1 | 94.13 | 64.69 | 71.80 | 73.43 | 64.25 | 73.66 |
| DPO | 0.2 | 91.34 | 64.47 | 78.27 | 74.37 | 49.83 | 71.66 |
| TODO | 0.2 | 95.81 | 64.69 | 78.49 | 62.30 | 61.63 | 72.58 |
| DPO | 0.3 | 85.75 | 64.04 | 75.78 | 75.01 | 45.56 | 69.23 |
| TODO | 0.3 | 89.94 | 65.13 | 77.13 | 73.37 | 57.06 | 72.53 |

**Table 9: Llama 3 model results on Reward Bench trained with different ratios of tie data.**
| Method | Tie Data Ratio | Chat | ChatHard | Safety | Reasoning | Prior | Average |
| --- | --- | --- | --- | --- | --- | --- | --- |
| SFT | ✗ | 67.60 | 55.48 | 41.89 | 65.17 | 43.26 | 54.68 |
| DPO | 0.0 | 93.3 | 67.32 | 75.18 | 84.23 | 54.78 | 74.96 |
| TODO | 0.0 | 93.58 | 65.79 | 77.66 | 83.32 | 59.72 | 76.01 |
| DPO | 0.1 | 93.85 | 63.82 | 77.14 | 83.52 | 53.91 | 74.45 |
| TODO | 0.1 | 96.37 | 64.69 | 79.32 | 84.08 | 59.59 | 76.81 |
| DPO | 0.2 | 91.62 | 66.45 | 78.18 | 84.80 | 53.70 | 74.95 |
| TODO | 0.2 | 92.74 | 66.89 | 75.50 | 84.37 | 60.37 | 75.97 |
| DPO | 0.3 | 89.39 | 65.35 | 78.31 | 87.02 | 52.87 | 74.59 |
| TODO | 0.3 | 93.02 | 64.91 | 74.35 | 84.64 | 60.33 | 75.45 |

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. Can TODO be integrated with other alignment methods beyond DPO, such as reinforcement learning from human feedback (RLHF)? If so, how would it interface with existing reward models?
  
2. Are there any trade-offs in using the TOBT model compared to traditional binary preference models? For example, does incorporating tie data introduce additional computational complexity or affect convergence?
soundness: 2
strengths: 1. The paper introduces a novel extension to the Bradley-Terry model by incorporating ties through the Tie-rank Oriented Bradley-Terry (TOBT) model. This allows for more nuanced preference modeling in LLM alignment.
  
2. The theoretical foundations of the TOBT model and the TODO algorithm are well-developed. The authors provide detailed derivations and clarify how the tie parameter α is integrated into the model.
  
3. The paper is generally well-written and structured. The methodology, including the mathematical formulations and the proposed algorithm, is clearly presented. The inclusion of figures and tables aids in understanding the results.
summary: The authors present an interesting approach to enhancing the alignment of LLMs through the use of ternary preferences. A limitation of binary models is their inability to effectively address tie situations in preference data, where two responses may be equally preferred or indistinguishable in quality. To address this issue, the authors propose an extension of the BT model, termed the Tie-rank Oriented Bradley-Terry (TOBT) model, which accommodates ties in preference relations. Building on this framework, they introduce a novel alignment algorithm called Tie-rank Oriented Direct Preference Optimization (TODO), specifically designed to manage ternary preference data that includes "prefer," "tie," and "disprefer." They conduct experiments comparing their TODO algorithm with the DPO. The results demonstrate that TODO consistently outperforms DPO in modeling preferences across both in-distribution and out-of-distribution datasets, as well as on several benchmarks. The paper also provides an analysis of the key factors contributing to TODO's improved performance.
weaknesses: 1. The extension from the traditional Bradley-Terry (BT) model to the Tie-rank Oriented BT (TOBT) model is central to the proposed method. However, the paper provides limited theoretical justification for this extension. The choice of the parameter α in the TOBT model appears somewhat arbitrary, and while Appendix A.3 attempts to justify its value based on balancing loss values, there is no rigorous analysis or sensitivity study demonstrating how different values of α affect the model's performance. Additionally, the paper lacks a thorough exploration of the probabilistic interpretation and properties of the TOBT model compared to the original BT model. A deeper theoretical analysis would strengthen the foundation of the proposed method and clarify its advantages and limitations.

2. The experiments primarily focus on datasets derived from Ultrafeedback and a select few benchmarks. While Ultrafeedback is a valuable resource, it may not fully represent the diversity and complexity of real-world human preference data, especially in terms of the prevalence and nature of tie data. The paper does not evaluate the method on more varied datasets, such as those involving different languages, domains, or more complex tasks where tie situations might behave differently. This raises concerns about the generalizability of the proposed method to broader applications and different types of data.
  
3. The paper compares the proposed TODO method primarily against Direct Preference Optimization (DPO). However, there are other existing methods in preference alignment and RLHF that could serve as relevant baselines, such as PPO, RPO, IPO), and KTO. By not including these comparisons, it is difficult to assess how TODO stands relative to the other existing methods in preference alignment. Including a broader range of baselines would provide a more comprehensive evaluation and strengthen the empirical claims.
  
4. While the paper highlights the prevalence of tie data in preference datasets and proposes methods to utilize it, But ties may often result from annotator uncertainty, ambiguity in prompts or responses, or inconsistencies among annotators. By treating tie data as equally informative as clear preferences without accounting for potential annotation noise or bias, the method might be incorporating noisy signals into the model. A discussion on how to handle annotation noise, perhaps by incorporating methods to assess annotator reliability or confidence, would strengthen the approach. Additionally, exploring strategies to improve data quality or to model uncertainty explicitly could enhance the effectiveness of the proposed method.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: Q1 (related to the first, main point listed under "Weaknesses"). Speaking of previous work, how does your tie-aware BT model differ from or improve upon existing multi-class generalizations of the BT model that I provided under weaknesses plus the ones that you also cited? Are there any components, ideas or other aspects of prior approaches to BT generalisation that could be incorporated into or compared against your method?

Q2. It seems that the results are also dependent on the tie data ratio in the full training set of 20k instances. How can a practitioner determine the best (or task-optimal) choice of the tie data ratio? What if tie data is lacking for some specific use-cases?

Q3. How were hyperparameters from Appendix A.9 selected? Is there any substantial variation of the results based on the hyper-parameters?
soundness: 3
strengths: - The paper is well written, and it is very easy to understand its core motivations and follow how the entire derivation of the tie-aware BT model and the corresponding TODO model unfolds from the main design assumptions.
- The provided results offer reasonable evidence on the benefits of TODO over DPO. The chosen evaluation data and models are adequate.
- The paper raises awareness on modeling ties in preference optimizations, where such valuable data often gets discarded under inadequate models or models with too strong assumptions (such as the binary BT) - taking into account finer granularity of preference data might yield further improvements in this line of work, and this paper sheds some new light on this link.
summary: This work aims to directly model the fact that binary preference models such as the standard and well-known Bradley-Terry (BT) model cannot integrate ties and uncertainties when working with preference data. The main idea is thus the 'ternarization' of the original binary BT model that takes into account possible ties, where 'ties' are modeled via an additional parameter alpha. A large part of the paper is then dedicated to motivating and deriving the ternary (i.e., tie-aware) BT model, and then, based on that model on deriving and explaining TODO, a tie-aware extension of the original DPO algorithm for preference optimization. Experiments with two standard models (Mistral and LLaMA) and with a spectrum of standard datasets demonstrate the benefits of the TODO approach over its base binary DPO approach, even in setups without any ties. The authors then aim to provide some theoretical and empirically grounded interpretations on what makes TODO a more expressive and better-performing alternative to DPO.
weaknesses: - One of the core issues with the work lies in the fact that other researchers in 'less recent times' have already attempted and derived generalised versions of the BT model, which better align with multi-class (i.e., non-binary) problems. Even a cursory search online reveals some very relevant literature which is not cited nor discussed in this work. I would require the authors to provide a thorough overview of that relevant work and discuss why a new (tie-aware) derivation might be needed here, and whether previous solutions could be repurposed for the extension of the original, binary BT model. For instance, I refer to the following relevant work here:
-- https://dl.acm.org/doi/pdf/10.5555/1248547.1248551
-- https://proceedings.neurips.cc/paper_files/paper/2004/hash/825f9cd5f0390bc77c1fed3c94885c87-Abstract.html
-- https://link.springer.com/article/10.1007/s10994-011-5240-0
See also Q1 related to this comment.

- Another concern is the lack of generality of TODO as it is derived directly as an extension of one concrete model: DPO. Following that, only DPO is also used as the only relevant baseline, and I see at least two potential issues here:
(i) While the authors state at the very end of the paper that the approach is "(...) not only limited to direct preference optimization but is also compatible with various online and offline preference optimization policies and can be employed for reward model training...", already this work should have coupled the idea with another preference optimization method to show this decoupling. In essence, the authors should both theoretically and empirically discern between making 'ternary decisions' and deriving enhanced preference optimization algorithms that incorporate the possibility to do ternary decisions. 
(ii) DPO is used as the only baseline, while there is a plethora of other online and offline preference optimization algorithms out there, where some of them should have been included as baselines and reference points. For instance, the KTO approach disposes of the need to even collect preference data and model preference decisions as part of the process (unlike DPO or PPO). One core question is then: can we simply use KTO instead of 'ternarizing DPO'? What would be the benefits of TODO versus KTO? 
(iii) The authors very vaguely comment in lines 122-123 that previous preference alignment studies "...overlook the potential of leveraging tied preference data to refine alignment, whereas our approach is complementary to most of them." It is unclear to which studies the proposed approach is complementary, and as mentioned above - it does not offer sufficient empirical validations to demonstrate that the approach is better than KTO, IPO, etc. I would suggest including direct comparisons to these other methods beyond DPO.

- All the experiments are based on the UltraFeedback dataset, and the training set size is limited to 20k data instances. The training set size might be a very important factor in defining the final performance (and benefits of ternary vs binary vs unary modeling as with KTO). Therefore, the paper should investigate how the standard DPO and KTO behave in comparison to TODO also for larger datasets (e.g., using the full UltraFeedback). Ideally, the work should experiment with another preference alignment dataset as well beyond UF, I would strongly suggest running experiments with another dataset, and compare against methods such as DPO and KTO also on larger datasets.

I am willing to reconsider my initial assessment and position if these central weaknesses get addressed by the authors.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. What is the intuition for why it works better than DPO even without ties in the data?
2. What is the naturally occurring frequency of ties in the UltraFeedback data?
3. How does it compare empirically against related approaches, especially ODPO?
soundness: 3
strengths: 1. The idea is well motivated, as ties are relevant especially in human preference elicitation and can be expected to become even more prevalent the stronger generative models get. 
2. The experiments are strongly supporting the new objective, the results are convincing.
3. The proposed solution is based on a solid intuition and supported by detailed derivation.
summary: This paper proposes the inclusion of ties in the DPO framework to be able to model preferences in a more nuanced way. It derives a ternary model for preferences analogously to the binary Bradley-Terry model, and defines a novel preference learning objective (TODO) accordingly. Experiments are conducted by preference-tuning Llama3 and Mistral with preferences from Ultrafeedback with varying proportions of ties, and evaluating them in and out of domain, and on downstream tasks. The new TODO objective outperforms DPO in preference modeling and downstream tasks across the bench.
weaknesses: The only claim that is empirically not supported is that TODO “exhibits better robustness against potential noise in binary preference data.” There is no experiment studying noise in preferences (ties do not need to be caused by noise, they also present adequate equalness). One could easily set up an experiment with artificial noise added to preferences to test this hypothesis.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: I am also very curious that if the performance of LLM alignment will improve if you just simply filter out the tie preference pairs out of the training set.
soundness: 3
strengths: The strengths of this paper can be listed as follows:
1. The motivation of this paper is clear and reasonable. Starting with the improvement of the BT model, this paper introduces the tie instance. This research idea is consistent with theoretical improvements of BT model.
2. The paper writing is good. I think it is easy to follow the authors' research ideas.
3. Experimental settings are detailed and well organized.
summary: Existing LLM alignment algorithms rely on the binary BradleyTerry model, which is hard to capture ties in preference. To address this issue, this paper proposes Tie-rank Oriented Direct Preference Optimization (TODO), an alignment algorithm that  leverages TOBT’s ternary ranking system. 

TODO rewrites the ranking probability in BradleyTerry to add tie instances and derive the loss function of TODO. Experimental results verify the effectiveness of TODO.
weaknesses: The weaknesses of this paper can be listed as follows:
1. **Limited baselines.** This paper only provides the comparison of experimental results between TODO and DPO, which is relatively limited. I think it is necessary to supplement some other algorithms as baselines, e.g. SimPO (https://arxiv.org/abs/2405.14734).
2. **Limited improvements.** The improvement of TODO over DPO is limited as shown in table 2 and table 3.

Meta-review: The authors introduce an extension of the Bradley-Terry (BT) model, called the Tie-rank Oriented Bradley-Terry (TOBT) model. The TOBT model incorporates ties, allowing for a more nuanced representation of preferences. Building on this foundation, the Tie-rank Oriented Direct Preference Optimization (TODO) algorithm is proposed to improve alignment further.

All reviewers found the paper well-written, the results reasonable, and the motivation clear. For the camera-ready version, experiments added during the rebuttal phase could be moved from the appendix to the main body to convince readers better. Additionally, reporting the sensitivity of hyperparameters in the main text would enhance usability and provide clearer guidance.

Decision: Accept (Poster)