Reward Learning From Preference With Ties

## Abstract

Abstract Reward learning plays a pivotal role in Reinforcement Learning from Human Feedback (RLHF), ensuring the alignment of language models. The Bradley-Terry (BT) model stands as the prevalent choice for capturing human preferences from datasets containing pairs of chosen and rejected responses. In preference modeling, the focus is not on absolute values but rather on the reward difference between chosen and rejected responses, referred to as preference strength. Thus, precise evaluation of preference strength holds paramount importance in preference modeling. However, an easily overlooked factor significantly affecting preference strength measurement is that human attitudes towards two responses may not solely indicate a preference for one over the other and ties are also a common occurrence. To address this, we propose the adoption of the generalized Bradley-Terry model – the Bradley-Terry model with ties (BTT) – to accommodate tied preferences, thus leveraging additional information. We prove that even with the access to the true distributions of prompt and response, disregarding ties can lead to a notable bias in preference strength measurement. Comprehensive experiments further validate the advantages of incorporating ties in preference modeling. Notably, fine-tuning with BTT significantly outperforms fine-tuning with BT on synthetic preference datasets with ties, labeled by state-of-the-art open-source LLMs.

## 1 Introduction

Reinforcement learning from human feedback (RLHF) (Christiano et al., ; Ziegler et al., ; Ouyang et al., ) has played a pivotal role in aligning large language models (LLMs) (Kenton et al., ), enhancing specific capabilities of LLMs in various fields, such as summarization (Stiennon et al., ), coding (Gao et al., ), and medical assistance (Moor et al., ). A crucial component of the RLHF process is the reward model, which serves as the primary mechanism for integrating human preferences and feedback into the learning process (Wang et al., ). The reward model guides the optimization procedure of RLHF towards objectives aligned with human preferences (Kaufmann et al., ). Therefore, the accuracy of the reward model greatly affects or even determines the effectiveness of alignment with human preferences. Moreover, the direct preference optimization (DPO) method (Rafailov et al., ) utilizes LLMs to implicitly represent the reward model through mathematical transformations, bypassing the complex RL optimization phase and focusing solely on the reward modeling phase. As a simplified alternative to RLHF, DPO has demonstrated computational efficiency and competitive performance compared to RLHF.

To learn a reward model from human preferences, obtaining high-quality human preference data is crucial (Wang et al., ), typically achieved by having human labelers annotate previously collected data consisting of a prompt and a pair of responses (Ouyang et al., ; Bai et al., ). We note that conventional approaches (Rafailov et al., ; Ziegler et al., ; Stiennon et al., ) often assume that the latent human preference model follows the Bradley-Terry (BT) (Bradley and Terry, ) model, where the preference distribution can be expressed as:

$$ $p_{r}\left(y_{1}\succ y_{2}\mid x\right)=\frac{\exp\left(r\left(x,y_{1}\right) \right)}{\exp\left(r\left(x,y_{1}\right)\right)+\exp\left(r\left(x,y_{2}\right )\right)}=\sigma(-\Delta r)$ $$

where $r$ is the latent reward model, $\sigma$ is sigmoid function, and $\Delta r=r(x,y_{1})-r(x,y_{2})$ is preference strength. Consequently, human labelers are presented with only two options for each prompt $x$ and pair of responses $y_{1}$ and $y_{2}$: either $y_{1}$ is preferred or $y_{2}$ is preferred. However, this approach does not fully align with human attitudes towards preferences, as humans often perceive two responses as ties when their rewards have very little difference. To illustrate this point, we refer to the findings of Wang et al. () due to the scarcity of preference datasets including ties. In their study, $10$ different reward models are trained on Anthropic’s HH-RLHF (Bai et al., ) dataset assuming the BT preference model. The mean and standard deviation of these $10$ reward models can partially reflect the real human preferences of this dataset. From the results of Wang et al. (), the absolute value of the preference strength for a large amount of data is close to 0 0, indicating that the preference difference in these cases is minimal and the pair of responses can often be considered as ties. We provide examples of data with small absolute mean preference strength in Table 1 to further emphasize the importance of introducing ties in preferences.

**Table 1: Examples of pairs of responses with low absolute mean preference strength in the annotated Anthropic’s HH-RLHF dataset (Wang et al., ).**
| Prompt | Can you help me set up an outdoor running routine so that I can get in shape? |
| --- | --- |
| Chosen Response | What kind of running are you interested in? |
| Rejected Response | Sure, what kind of program are you looking for? |
| Preference Strength | Mean : 0.0027, Standard Deviation: 0.22 |
| Prompt | Can you tell me what the meaning and themes are of the book Oliver Twist |
| Chosen Response | Sure, here are some common themes, and there are a few interpretations that scholars have proposed: - the evil and temptations of London, - the books portray Oliver as a victim of the harsh society, - Oliver is also a victim of his situation, - the story is an attack on |
| Rejected Response | The meaning of the book is about the theme of being homeless and being poor. This is an important thing in the society we live in today. When we have children, we want them to be able to live a good life with lots of opportunities. However, we don’t have all the resources to provide that to them. So the book is about the struggles of poor people trying to live a life with a family, which is difficult and sad. |
| Preference Strength | Mean : 0.0, Standard Deviation: 0.36 |
| Prompt | I want to purchase a men’s suit, but I don’t know what I am looking for or how to get started. |
| Chosen Response | Sure! In what price range are you looking to spend? |
| Rejected Response | Great, let me walk you through the process. First we’ll start with the size, do you know your measurements? |
| Preference Strength | Mean : 0.00019, Standard Deviation: 0.23 |

In this paper, we propose to model human preferences with the Bradley-Terry model with ties (BTT) (Rao and Kupper, ). We demonstrate that if we blindly adopt BT as our preference model (with the true latent preference model being BTT) can lead to significant bias in measuring preference strength due to model mismatch. To illustrate the model mismatch problem occurring in conventional preference datasets lacking ties when simply assuming the latent preference model is BT, we first introduce a simulated preference generation procedure. Specifically, we generate two preference datasets—one with ties and one without ties. By analyzing the maximum likelihood estimates (MLE) based on BTT and BT preference models on these datasets respectively, we quantify the bias in measuring preference strength due to the model mismatch. Furthermore, we show that although the bias term is bounded, it can still have a substantial impact. Since most conventional preference datasets lack ties, we propose a novel method to address the preference model mismatch problem, which subtracts the bias term from the MLE loss function to recover the true preference strength measurement. This method can be viewed as a variant of adaptive margin (Touvron et al., ) when training the reward model and a variant of DPO with offset (ODPO) when training DPO (Amini et al., ). To further demonstrate the benefit of incorporating ties in preference modeling, we use state-of-the-art open-source LLMs to simulate human judgment and label ties in a conventional preference dataset without ties, and then evaluate the fine-tuned models on this synthetic preference dataset with ties. It is important to note that the main limitation of this paper is the inability to conduct experiments on real human-labeled preference datasets with ties, due to the scarcity of such datasets and the high cost of manual annotation and evaluation. Addressing this limitation could be considered for future work.

Main Contributions. Our contributions can be outlined as follows:

- •
We advocate for the inclusion of tie options when labeling preference data, aligning with human preference habits. To the best of our knowledge, we are the first to propose the use of BTT to model human preference.
- •
We derive the bias in measuring preference strength caused by model mismatch when assuming the latent preference model is BTT. To address this, we propose a novel bias-correction method to mitigate this bias in conventional preference datasets without ties, as validated by comprehensive experimental results.
- •
We generate a synthetic preference dataset with ties, labeled by state-of-the-art open-source LLMs, and evaluate fine-tuning with BTT and BT on this dataset. The results show that fine-tuning with BTT consistently outperforms fine-tuning with BT.

## 2 Related Work

The reward model plays a crucial role in RLHF, guiding LLMs towards objectives aligned with human preferences (Christiano et al., ; Kaufmann et al., ). Recent related work has addressed various aspects of reward modeling. Wang et al. () conducted a comprehensive study on reward models, proposing a method to measure the strength of preferences within the data and introducing contrastive learning to enhance the ability of reward models to distinguish between chosen and rejected responses. Zhu et al. () analyzed reward overfitting and overoptimization problems in RLHF, proposing to mitigate them using an iterative data smoothing method. Dai et al. () proposed training a cost model in addition to the reward model to decouple human preferences regarding helpfulness and harmlessness.

As a simplified alternative to RLHF, DPO (Rafailov et al., ) has achieved significant success and impact. The core concept of DPO involves implicitly representing the reward model using LLMs through a clever reparameterization. Recently, there has been extensive research focused on enhancing and broadening the scope of DPO. Amini et al. () propose DPO with an offset (ODPO), where the likelihood difference between the preferred and dispreferred response must exceed an offset value. Zhou et al. () extend DPO for multiple alignment objectives by training LMs as implicit collective reward models, combining all objectives with specific weightings. Chowdhury et al. () propose robust DPO methods to mitigate the bias introduced by noise in preference data on average.

The preference model serves as the foundation for reflecting human feedback, with the Bradley-Terry (BT) model (Bradley and Terry, ) being the most commonly used preference model in RLHF. Indeed, various generalized models based on the BT model have been proposed to address different scenarios, such as handling home advantage (Agresti, ), ties (Rao and Kupper, ), multiple comparisons (Plackett, ; Luce, ), and team comparisons (Huang et al., ). In particular, the Plackett-Luce (PL) model, a popular extension for handling multiple comparisons, has also found application in RLHF (Zhu et al., ; Song et al., ).

## 3 Preliminaries

RLHF typically comprises three phases: supervised fine-tuning (SFT), reward learning, and reinforcement learning. In the first phase, a pre-trained language model undergoes fine-tuning via supervised learning on high-quality data tailored for specific tasks such as dialogue and summarization. This fine-tuning process yields the model $\pi^{\mathrm{SFT}}$. The second phase involves reward learning on a preference dataset. To construct this dataset, prompts $x\sim\mathcal{X}$ are fed to $\pi^{\mathrm{SFT}}$, generating pairs of responses $\left(y_{1},y_{2}\right)\sim\pi^{\mathrm{SFT}}(y\mid x)$. These pairs are presented to human labelers, who express preferences. Conventional preference datasets do not allow ties and require one response to be preferred over the other, denoted as $y_{w}\succ y_{l}\mid x$, where $y_{w}$ and $y_{l}$ represent the preferred and dispreferred completions among $\left(y_{1},y_{2}\right)$, respectively. The most popular approach to modeling preference is the Bradley-Terry (BT) model, which assumes the human preference distribution $p^{*}$ as:

$$ $p^{*}\left(y_{1}\succ y_{2}\mid x\right)=\frac{\exp\left(r^{*}\left(x,y_{1} \right)\right)}{\exp\left(r^{*}\left(x,y_{1}\right)\right)+\exp\left(r^{*} \left(x,y_{2}\right)\right)}.$ (1) $$

where $r^{*}(y,x)$ is the latent reward model which is inaccessible. Assuming access to a static dataset of comparisons $\mathcal{D}=\left\{x^{(i)},y_{w}^{(i)},y_{l}^{(i)}\right\}_{i=1}^{N}$ sampled from $p^{*}$, we can parametrize a reward model $r_{\psi}(x,y)$ and estimate the parameters via maximum likelihood. Framing the problem as a binary classification we have the negative log-likelihood loss:

$$ $\mathcal{L}_{R}\left(r_{\psi},\mathcal{D}\right)=-\mathbb{E}_{\left(x,y_{w},y_ {l}\right)\sim\mathcal{D}}\left[\log\sigma\left(r_{\psi}\left(x,y_{w}\right)-r _{\psi}\left(x,y_{l}\right)\right)\right],$ $$

where $\sigma$ is the logistic function. And the third phase is to solve the following RL problem with the learned reward function:

$$ $\max_{\pi_{\theta}}\mathbb{E}_{x\sim\mathcal{D},y\sim\pi_{\theta}(y\mid x)} \left[r_{\psi}(x,y)\right]-\beta\mathbb{D}_{\mathrm{KL}}\left[\pi_{\theta}(y \mid x)\|\pi^{\mathrm{SFT}}(y\mid x)\right],$ (2) $$

where $\beta$ is a parameter controlling the deviation from the base reference policy $\pi^{\mathrm{SFT}}$.

DPO utilizes the fact that the optimization problem (2) has the closed form solution (Go et al., ; Korbak et al., ; Peng et al., ; Peters and Schaal, ):

$$ $\pi_{r}(y\mid x)=\frac{1}{Z(x)}\pi^{\mathrm{SFT}}(y\mid x)\exp\left(\frac{1}{ \beta}r(x,y)\right).$ $$

Then a clever reparameterization is applied to express the reward function in terms of its corresponding optimal policy $\pi_{r}$:

$$ $r(x,y)=\beta\log\frac{\pi_{r}(y\mid x)}{\pi^{\mathrm{SFT}}(y\mid x)}+\beta\log Z (x).$ $$

Applying this reparameterization to the ground-truth reward $r^{*}$ and corresponding optimal model $\pi^{*}$, then substituting this reparameterization into the BT model (1), analogous to the reward modeling approach, the loss function of DPO becomes:

$$ $\mathcal{L}_{\mathrm{DPO}}\left(\pi_{\theta};\pi_{\mathrm{ref}}\right)=- \mathbb{E}_{\left(x,y_{w},y_{l}\right)\sim\mathcal{D}}\left[\log\sigma\left( \beta\log\frac{\pi_{\theta}\left(y_{w}\mid x\right)}{\pi^{\mathrm{SFT}}\left(y _{w}\mid x\right)}-\beta\log\frac{\pi_{\theta}\left(y_{l}\mid x\right)}{\pi^{ \mathrm{SFT}}\left(y_{l}\mid x\right)}\right)\right].$ $$

Bradley-Terry model with ties (BTT) (Rao and Kupper, ) can be employed to model human preference with ties, i.e., the two response $\left(y_{1},y_{2}\right)\sim\pi^{\mathrm{SFT}}(y\mid x)$ are considered equal with respect to the prompt $x$:

$$ $\displaystyle p_{\theta}^{*}\left(y_{1}=y_{2}\mid x\right)=$ $\displaystyle\frac{(\theta^{2}-1)\exp\left(r^{*}\left(x,y_{1}\right)\right) \exp\left(r^{*}\left(x,y_{2}\right)\right)}{\left(\exp\left(r^{*}\left(x,y_{1} \right)\right)+\theta\exp\left(r^{*}\left(x,y_{2}\right)\right)\right)\left( \theta\exp\left(r^{*}\left(x,y_{1}\right)\right)+\exp\left(r^{*}\left(x,y_{2} \right)\right)\right)}$ (3) $\displaystyle p_{\theta}^{*}\left(y_{1}\succ y_{2}\mid x\right)=$ $\displaystyle\frac{\exp\left(r^{*}\left(x,y_{1}\right)\right)}{\exp\left(r^{*} \left(x,y_{1}\right)\right)+\theta\exp\left(r^{*}\left(x,y_{2}\right)\right)}$ $$

where $\theta\geq 1$ is the parameter controlling the tendency to ties, with a larger $\theta$ indicating a higher probability of ties occurring. It’s worth noting that when $\theta=1$, the BTT model is equivalent to the BT model.

## 4 Preference Modeling With Ties

For a given reward model $r$, RLHF focuses not on the absolute values $r(x,y_{1}),r(x,y_{2})$ but on the preference strength between the pair of responses (Wang et al., ):

$$ $\Delta r=r(x,y_{1})-r(x,y_{2})$ $$

In this section, we will explain that if the real preference model is BTT, but we do not provide human labelers with the option of a tie to generate the preference dataset, the learned reward model will exhibit significant deviation from the real reward model in measuring preference strength.

### 4.1 Preference Dataset Under BTT

Since previous preference datasets do not include ties, we will first explain the simulation process for obtaining a preference dataset without ties when assuming the preference model is BTT. Suppose we have $n$ samples, each consisting of a prompt and a pair of responses, denoted as $D=\{\left(x_{i},y_{i}^{1},y_{i}^{2}\right)\}_{i=1}^{n}$. With $D$ available, if we assume the true preference model is BTT, we can obtain preference datasets with and without ties using the following methods:

- •
Offer three options to human labelers: $y_{1}\succ y_{2}$, $y_{2}\succ y_{1}$, or $y_{1}=y_{2}$. Then, we can derive a preference dataset with ties $D^{BTT}$ from the original dataset $D$. We denote that $D^{BTT}=D^{BT}\cup D^{T}$, where $D^{BT}=\{\left(x_{i},y_{i}^{w},y_{i}^{l}\right)\},y_{i}^{w}\succ y_{i}^{l},i
\in\mathcal{J};D^{T}=\{\left(x_{i},y_{i}^{1},y_{i}^{2}\right)\},y_{i}^{1}=y_{i
}^{2},i\in\mathcal{K}$, and $\mathcal{J}\cup\mathcal{K}=\{n\}$.
- •
For the ties dataset $D^{T}$, ask human labelers to further specify which response is preferred, resulting in the dataset $D^{TN}=\{\left(x_{i},y_{i}^{w},y_{i}^{l}\right)\},y_{i}^{w}\succ y_{i}^{l},i
\in\mathcal{K}$. We denote $D^{BTTN}=D^{BT}\cup D^{TN}$.

###### Assumption 4.1 .

Human labelers randomly label responses in ties, assigning each response an equal probability of being preferred.

In summary, if we assume the preference model is the BTT model and provide the option for ties to human labelers, we obtain the preference dataset with ties $D^{BTT}$. By subsequently asking human labelers to specify preferred responses within ties, we derive the preference dataset without ties $D^{BTTN}$. Therefore, we can consider conventional preference datasets without ties as $D^{BTTN}$.

###### Assumption 4.1 .

### 4.2 Bias in Measuring Preference Strength

Assuming we have both $D^{BTT}$ and $D^{BTTN}$ derived from $D$, we can illustrate how to estimate the latent reward model using maximum likelihood estimation (MLE). Since we assume that the latent preference model is BTT and thus obtain the dataset with ties, the most accurate log-likelihood is:

$$ $LCE^{BTT}(r,D)=\sum_{(x,y_{w},y_{l})\in D^{BT}}\log p_{r}^{\theta}(y_{w}\succ y _{l}\mid x)+\sum_{(x,y_{1},y_{2})\in D^{T}}\log p_{r}^{\theta}(y_{1}=y_{2}\mid x)$ (4) $$

Conventional approaches to estimate the latent reward model typically utilize $D^{BTTN}$ to fit the BT model, with the log-likelihood given by:

$$ $LCE^{BT}(r,D)=\sum_{(x,y_{w},y_{l})\in D^{BTTN}}\log p_{r}(y_{w}\succ y_{l} \mid x)$ (5) $$

We can demonstrate that, even if we possess access to the true prompt and response distributions, there may exist a noteworthy discrepancy between the learned and the actual reward model in measuring preference strength, as illustrated by the following results.

First, we can establish the relationship between the true reward model $r^{*}$ and the learned reward model $\hat{r}$ by fully optimizing (5) in Theorem 4.2.

###### Theorem 4.2 .

$$ $\mathbb{E}\left[LCE^{BT}(r,D)\right]\leq\mathbb{E}\left[LCE^{BT}(\hat{r},D) \right],\forall r\neq\hat{r}$ $$

where $\hat{r}$ satisfies

$$ $p_{\hat{r}}(y_{1}\succ y_{2}\mid x)=q_{r^{*}}^{\theta}(y_{1}\succ y_{2}\mid x) ,\forall x\sim\mathcal{X},(y_{1},y_{2})\sim\pi^{\mathrm{SFT}}(y\mid x)$ (6) $$

and

$$ $q_{r}^{\theta}\left(y_{1}\succ y_{2}\mid x\right)=p_{r}^{\theta}\left(y_{1} \succ y_{2}\mid x\right)+\frac{1}{2}p_{r}^{\theta}\left(y_{1}=y_{2}\mid x\right)$ (7) $$

###### Proof.

By Assumption 4.1 we know that the true preference distribution without ties is $q_{r}^{\theta}$. Therefore, it is equivalent to verify that:

$$ $\displaystyle\mathbb{E}_{x\sim\mathcal{X},(y_{1},y_{2})\sim\pi^{\mathrm{SFT}}( y\mid x),(y_{w},y_{l})\sim q_{r^{*}}^{\theta}}\left[\log\frac{p_{r}(y_{w}\succ y _{l}\mid x)}{p_{\hat{r}}(y_{w}\succ y_{l}\mid x)}\right]\leq 0$ $$

by Jensen’s inequality we have:

$$ $\displaystyle\mathbb{E}\left[\log\frac{p_{r}(y_{w}\succ y_{l}\mid x)}{p_{\hat{ r}}(y_{w}\succ y_{l}\mid x)}\right]\leq\log\left(\mathbb{E}\left[\frac{p_{r}(y _{w}\succ y_{l}\mid x)}{p_{\hat{r}}(y_{w}\succ y_{l}\mid x)}\right]\right)$ $\displaystyle=\log\left(\mathbb{E}_{(x,y_{1},y_{2})}\left[q_{r^{*}}^{\theta}(y _{1}\succ y_{2}\mid x)\frac{p_{r}(y_{1}\succ y_{2}\mid x)}{p_{\hat{r}}(y_{1} \succ y_{2}\mid x)}+q_{r^{*}}^{\theta}(y_{2}\succ y_{1}\mid x)\frac{p_{r}(y_{2 }\succ y_{1}\mid x)}{p_{\hat{r}}(y_{2}\succ y_{1}\mid x)}\right]\right)$ $\displaystyle=\log\left(\mathbb{E}_{(x,y_{1},y_{2})}\left[p_{r}(y_{1}\succ y_{ 2}\mid x)+p_{r}(y_{2}\succ y_{1}\mid x)\right]\right)$ $\displaystyle=\log\left(\mathbb{E}_{(x,y_{1},y_{2})}\left[1\right]\right)$ $\displaystyle=0$ $$

∎

###### Theorem 4.3 .

Even if we have the access to the true prompt and response distributions, there can be a bias in measuring preference strength:

$$ $\Delta\hat{r}=\Delta r^{*}+\log\left(\frac{2\theta+\left(1+\theta^{2}\right) \exp(-\Delta r^{*})}{1+\theta^{2}+2\theta\exp(-\Delta r^{*})}\right),\forall(x ,y_{1},y_{2})$ (8) $$

where $\Delta r=r(x,y_{1})-r(x,y_{2})$.

Proof Sketch: From (6), we can know that:

$$ $\displaystyle p_{\hat{r}}(y_{1}\succ y_{2}\mid x)=$ $\displaystyle p_{r^{*}}^{\theta}(y_{1}\succ y_{2}\mid x)+\frac{1}{2}p_{r^{*}}^ {\theta}(y_{1}=y_{2}\mid x)$ $\displaystyle p_{\hat{r}}(y_{2}\succ y_{1}\mid x)=$ $\displaystyle p_{r^{*}}^{\theta}(y_{2}\succ y_{1}\mid x)+\frac{1}{2}p_{r^{*}}^ {\theta}(y_{2}=y_{1}\mid x)$ $$

By subtraction, we can get:

$$ $p_{\hat{r}}(y_{1}\succ y_{2}\mid x)-p_{\hat{r}}(y_{2}\succ y_{1}\mid x)=p_{r^{ *}}^{\theta}(y_{1}\succ y_{2}\mid x)-p_{r^{*}}^{\theta}(y_{2}\succ y_{1}\mid x)$ $$

Consequently, we can derive the relation between $\Delta\hat{r}$ and $\Delta r^{*}$. Detailed proof can be found in the appendix A.

To analyze the bias term $\log\left(\frac{2\theta+\left(1+\theta^{2}\right)\exp(-\Delta r^{*})}{1+\theta
^{2}+2\theta\exp(-\Delta r^{*})}\right)$, we can observe that its sign is opposite to $\Delta r^{*}$, indicating that the preference strength is attenuated due to latent preference model mismatch. Additionally, the bias term is a sigmoid-shaped function, bounded by $\log(\frac{1+\theta^{2}}{2\theta})$ in absolute value. However, despite this bound, the bias term can still be substantial. As mentioned earlier, Wang et al. () trained 10 different reward models on the Anthropic’s HH-RLHF dataset (Bai et al., ), and the mean preference strength of 83.6% of the data falls within the interval $[-0.6,2.94]$. In this range, the ratio between the bias term and $\Delta r^{*}$ can be considerable, as depicted in Figure 1.

Figure: Figure 1: Bias term has a significant impact
Refer to caption: extracted/5904291/bias.png

###### Theorem 4.2 .

###### Proof.

###### Theorem 4.3 .

### 4.3 Preference Strength Bias Correction Algorithm

Since conventional preference datasets typically lack ties, we propose a novel method to address the model mismatch issue on preference datasets without ties, assuming the latent preference model is the BTT model.
We acknowledge that the right side of (8) is a monotonic function with respect to $\Delta r^{*}$, implying a one-to-one mapping relationship between $\Delta\hat{r}$ and $\Delta r^{*}$. Thus, during the optimization procedure, when obtaining the value of $\Delta\hat{r}$, we can treat (8) as a nonlinear equation and solve for the value of $\Delta r^{*}$, subsequently subtracting the bias term from the current $\Delta\hat{r}$. The detailed description of this method can be found in Alg. 1. We note that this method can be viewed as a variant of DPO with an offset (ODPO) (Amini et al., ) when fine tuning with DPO.

Figure: Algorithm 1 Preference Strength Bias Correction

## 5 Experiments

In this section, we empirically demonstrate the benefits of incorporating ties in preference learning. First, we conduct a simulation experiment to show that, when the ground truth reward function is accessible and the preference dataset is labeled according to the BTT model, the reward model trained with the BT model exhibits a stronger preference strength bias compared to the one trained with the BTT model. Second, we apply Algorithm 1 to address the model mismatch problem on conventional preference datasets without ties. Finally, we use two state-of-the-art open-source LLMs Llama3-70b (abbreviated as Llama) (Meta, ) and Qwen2-72b-instruct (abbreviated as Qwen) (Yang et al., ) to label whether pairs in Anthropic’s HH-RLHF dataset (Bai et al., ) are tied, thereby generating a simulated preference dataset with ties. We then evaluate the fine-tuning using BT and BTT on this dataset. We choose DPO as our fine-tuning technique because it is an simplified and efficient alternative to RLHF and allows LLMs to be treated as implicit reward models. We follow Rafailov et al. (), fine-tuning on Anthropic’s HH-RLHF dataset (Bai et al., ) and consistently setting $\beta=0.1$ for DPO. Additional experimental details can be found in Appendix B.

### 5.1 Preference Bias With The Ground Truth Reward

In this section, we randomly generate a ground truth reward function $r^{*}(x,y),x\in\mathbb{N}^{+},y\in[0,1,2,3]^{n}$, along with a preference dataset labeled by the BTT model using $r^{*}$ (with tied pairs randomly assigned preferences). We then train two reward models, both parameterized by the same neural network, on this dataset using the loss functions 4 and 5, respectively. These trained reward models are denoted as $r^{BTT}$ and $r^{BT}$. Next, we evaluate the average preference bias of these two reward models relative to the ground truth reward under varying preference parameters $\theta$. The preference bias difference, $\Delta=|\Delta r^{BT}-\Delta r^{*}|-|\Delta r^{BTT}-\Delta r^{*}|$, is shown in Table 2. From the results, we observe that the preference bias of $r^{BTT}$ is consistently smaller than that of $r^{BT}$, indicating that the BTT model effectively reduces the preference bias with respect to the ground truth reward function, resulting in a more accurate reward model. We also find that as $\theta$ increases, the preference bias difference becomes larger, which aligns with the trends shown in Figure 1, as a larger $\theta$ in ground truth preference model indicates a higher probability of ties occurring.

**Table 2: The preference bias difference between $r^{BT}$ and $r^{BTT}$**
| $\theta=2$ | $\theta=5$ | $\theta=10$ |
| --- | --- | --- |
| 0.0206 | 0.0237 | 0.0353 |

### 5.2 DPO With a Bias-Correction Offset

We apply Alg. 1 to the conventional preference dataset without ties, Anthropic’s HH-RLHF, in order to mitigate the bias term using a DPO reward model. It is important to note that this approach can be viewed as a variant of the ODPO method (Amini et al., ), with the key difference being the bias-correction term. We train the small Pythia-160M model (Biderman et al., ) for one epoch and record the reward preference accuracy on the test set. It is also worth mentioning that we do not evaluate Pythia-160M’s inference capability, as the model is too small to generate meaningful responses. The experimental results are presented in Table 3. As shown, when $\theta=1$, the bias-correction term is consistently zero, which essentially reduces the method to DPO, serving as our baseline. From Table 3, we observe that all three ODPO methods, with $\theta\in\{2,5,10\}$, significantly outperform DPO, with ODPO at $\theta=5$ showing more than a $10\%$ improvement in accuracy.

**Table 3: Test Accuracy of DPO with a bias-Correction offset**
| $\theta=1$ | $\theta=2$ | $\theta=5$ | $\theta=10$ |
| --- | --- | --- | --- |
| 0.5333 | 0.5583 | 0.6042 | 0.5958 |

To further validate the effectiveness of Alg 1, we fine-tuned the larger Pythia-2.8B model (Biderman et al., ) on the HH-RLHF dataset using DPO and DPO with a bias-correction offset, and evaluated their responses using Llama and Qwen. Due to limited computing resources, we only conducted experiments for the optimal $\theta$, i.e., 5, as indicated in Table 3. The results, shown in Table 4, demonstrate that our method significantly outperforms DPO, confirming the effectiveness of the preference strength bias-correction offset. It is important to note that we provide evaluators with the option to label ties, and Llama and Qwen may occasionally refuse to evaluate certain offensive content. Therefore, we only include samples that are clearly evaluated as wins or losses when calculating the win rate.

**Table 4: Win rate of DPO with a bias-correction offset against DPO**
| Evaluator | Llama | Qwen |
| --- | --- | --- |
| Win rate | 0.5582 | 0.5370 |

### 5.3 Synthetic Preference Datasets with Ties

The most compelling experiment is to fine-tune two models using BT and BTT, respectively, on a real preference dataset with ties and then compare their win rates. However, due to the lack of human-labeled preference datasets with ties and the high cost of manual annotation and evaluation, we use an LLM to simulate human judgment and label ties in Anthropic’s HH-RLHF dataset. We then fine-tune Pythia-2.8B (Biderman et al., ) on this synthetic preference dataset with ties, applying the BT and BTT preference models, and evaluate the responses. When using the loss function 4, we refer to this approach as TDPO. To reduce bias, we utilize Llama and Qwen, alternately as labelers and evaluators. The two labeled preference datasets are summarized in Table 5.

**Table 5: Summary of preference datasets with ties**
| labeler | Llama | Qwen |
| --- | --- | --- |
| # of tied samples | 847 | 3553 |

Figure: Figure 2: TDPO win rate against DPO with varying ties sample ratio in preference dataset
Refer to caption: extracted/5904291/win_rate_vs_ties_ratio.png

We observe that Anthropic’s HH-RLHF dataset contains over $160k$ samples, with only a small portion labeled as ties. As a result, directly fine-tuning LLMs on the entire labeled dataset would lead to minimal impact from the tied samples. To emphasize the importance of these tied samples, we train DPO and TDPO on preference datasets with varying percentages of tied samples, with untied samples randomly selected. We still only conducted experiments for the optimal $\theta$, i.e., $5$, due to limited computing resources. The win rate results are presented in Figure 2. From the results, we observe that, regardless of the labeler and evaluator, the win rate of TDPO increases as the number of tied samples increases, and it significantly exceeds $50\%$ when only tied samples are present. This demonstrates that incorporating BTT with tied samples improves the quality of the trained reward model. Moreover, as shown in Figure 2, when Llama is the labeler and Qwen is the evaluator, the win rate of TDPO consistently exceeds $50\%$ when the tie ratio is greater than $0.2$. In contrast, when Qwen is the labeler and Llama is the evaluator, TDPO’s performance is slightly lower. This may be due to Qwen’s less strict criteria for ties, resulting in an overabundance of tied samples.

## 6 Discussion

In this paper, we introduced the concept of incorporating ties into preference modeling. Specifically, we applied the generalized Bradley-Terry model—the Bradley-Terry model with ties—to more accurately capture human preferences. Additionally, we analyzed the bias in measuring preference strength due to model mismatch and proposed a novel method to mitigate this bias. Extensive experiments demonstrate the benefits of considering ties in preference modeling. A limitation of this work is the absence of real human-annotated preference datasets with ties, as collecting such data is both expensive and time-consuming. Future work involving human-labeled preference datasets with ties could significantly improve the effectiveness of preference modeling.

## Acknowledgements

This research is partially supported by the National Natural Science Foundation of China (NSFC) [Grant NSFC-72225009, 72394360, 72394365].

## Appendix A Proof of Theorem 4.3

###### Proof.

From (6), we can know that:

$$ $\displaystyle p_{\hat{r}}(y_{1}\succ y_{2}\mid x)=$ $\displaystyle p_{r^{*}}^{\theta}(y_{1}\succ y_{2}\mid x)+\frac{1}{2}p_{r^{*}}^ {\theta}(y_{1}=y_{2}\mid x)$ $\displaystyle p_{\hat{r}}(y_{2}\succ y_{1}\mid x)=$ $\displaystyle p_{r^{*}}^{\theta}(y_{2}\succ y_{1}\mid x)+\frac{1}{2}p_{r^{*}}^ {\theta}(y_{2}=y_{1}\mid x)$ $$

By subtraction, we can get:

$$ $p_{\hat{r}}(y_{1}\succ y_{2}\mid x)-p_{\hat{r}}(y_{2}\succ y_{1}\mid x)=p_{r^{ *}}^{\theta}(y_{1}\succ y_{2}\mid x)-p_{r^{*}}^{\theta}(y_{2}\succ y_{1}\mid x)$ $$

Therefore,

$$ $\displaystyle\frac{\exp(\Delta\hat{r})}{1+\exp(\Delta\hat{r})}-\frac{1}{1+\exp (\Delta\hat{r})}$ $\displaystyle=\frac{\exp(\Delta r^{*})}{\theta+\exp(\Delta r^{*})}-\frac{1}{1+ \theta\exp(\Delta r^{*})}$ $\displaystyle\frac{\exp(\Delta\hat{r}-\Delta r^{*})}{1+\exp(\Delta\hat{r})}- \frac{1}{\left(1+\exp(\Delta\hat{r})\right)\exp(\Delta r^{*})}$ $\displaystyle=\frac{1}{\theta+\exp(\Delta r^{*})}-\frac{1}{\left(1+\theta\exp( \Delta r^{*})\right)\exp(\Delta r^{*})}$ $$

Then we can get:

$$ $\displaystyle\exp(\Delta\hat{r}-\Delta r^{*})$ $\displaystyle=\left(1+\exp(\Delta\hat{r})\right)\left[\frac{1}{\theta+\exp( \Delta r^{*})}-\frac{1}{\left(1+\theta\exp(\Delta r^{*})\right)\exp(\Delta r^{ *})}+\frac{1}{\left(1+\exp(\Delta\hat{r})\right)\exp(\Delta r^{*})}\right]$ $\displaystyle=\frac{1+\exp(\Delta\hat{r})}{\theta+\exp(\Delta r^{*})}+\frac{ \theta\exp(\Delta r^{*})-\exp(\Delta\hat{r})}{\left(1+\theta\exp(\Delta r^{*}) \right)\exp(\Delta r^{*})}$ $\displaystyle=\frac{1+\exp(\Delta\hat{r})}{\theta+\exp(\Delta r^{*})}+\frac{ \theta-\exp(\Delta\hat{r}-\Delta r^{*})}{1+\theta\exp(\Delta r^{*})}$ $$

Consequently,

$$ $\displaystyle\exp(\Delta\hat{r}-\Delta r^{*})\frac{2+\theta\exp(\Delta r^{*})} {1+\theta\exp(\Delta r^{*})}$ $\displaystyle=\frac{1+\exp(\Delta\hat{r})}{\theta+\exp(\Delta r^{*})}+\frac{ \theta}{1+\theta\exp(\Delta r^{*})}$ $$

Then,

$$ $\displaystyle\exp(\Delta\hat{r}-\Delta r^{*})$ $\displaystyle=\frac{1+\theta\exp(\Delta r^{*})}{2+\theta\exp(\Delta r^{*})} \cdot\frac{1+\exp(\Delta\hat{r})}{\theta+\exp(\Delta r^{*})}+\frac{\theta}{2+ \theta\exp(\Delta r^{*})}$ $\displaystyle=\frac{1+\theta\exp(\Delta r^{*})}{2+\theta\exp(\Delta r^{*})} \cdot\frac{\exp(-\Delta r^{*})+\exp(\Delta\hat{r}-\Delta r^{*})}{1+\theta\exp( -\Delta r^{*})}+\frac{\theta}{2+\theta\exp(\Delta r^{*})}$ $$

$$ $\displaystyle\exp(\Delta\hat{r}-\Delta r^{*})\left(1-\frac{1+\theta\exp(\Delta r ^{*})}{\left(2+\theta\exp(\Delta r^{*})\right)\left(1+\theta\exp(-\Delta r^{*} )\right)}\right)$ $\displaystyle=\exp(\Delta\hat{r}-\Delta r^{*})\frac{1+\theta^{2}+2\theta\exp(- \Delta r^{*})}{\left(2+\theta\exp(\Delta r^{*})\right)\left(1+\theta\exp(- \Delta r^{*})\right)}$ $\displaystyle=\frac{\left(1+\theta\exp(\Delta r^{*})\right)\exp(-\Delta r^{*}) }{\left(2+\theta\exp(\Delta r^{*})\right)\left(1+\theta\exp(-\Delta r^{*}) \right)}+\frac{\theta}{2+\theta\exp(\Delta r^{*})}$ $\displaystyle=\frac{\theta+\exp(-\Delta r^{*})}{\left(2+\theta\exp(\Delta r^{* })\right)\left(1+\theta\exp(-\Delta r^{*})\right)}+\frac{\theta}{2+\theta\exp( \Delta r^{*})}$ $$

Finally, we can get:

$$ $\displaystyle\exp(\Delta\hat{r}-\Delta r^{*})$ $\displaystyle=\frac{\theta+\exp(-\Delta r^{*})}{1+\theta^{2}+2\theta\exp(- \Delta r^{*})}+\frac{\theta\left(1+\theta\exp(-\Delta r^{*})\right)}{1+\theta^ {2}+2\theta\exp(-\Delta r^{*})}$ $\displaystyle=\frac{2\theta+\left(1+\theta^{2}\right)\exp(-\Delta r^{*})}{1+ \theta^{2}+2\theta\exp(-\Delta r^{*})}$ $$

Denote

$$ $f(x)=\frac{2\theta+\left(1+\theta^{2}\right)\exp(x)}{1+\theta^{2}+2\theta\exp( x)}$ $$

then we know that:

$$ $f^{\prime}(x)=\frac{\exp(x)\left(1-\theta^{2}\right)^{2}}{\left(1+\theta^{2}+2 \theta\exp(x)\right)^{2}}\geq 0$ $$

Therefore,

$$ $\frac{2\theta}{1+\theta^{2}}=\lim_{x\rightarrow-\infty}f(x)\leq f(x)\leq\lim_{ x\rightarrow\infty}f(x)=\frac{1+\theta^{2}}{2\theta}$ $$

Consequently, we have:

$$ $|\Delta\hat{r}-\Delta r^{*}|\leq\log(\frac{1+\theta^{2}}{2\theta})$ $$

∎

###### Proof.

## Appendix B Experiment Details

### B.1 Experimental Setup

For each single experiment, we choose the same $64$ batch size, RMSprop optimizer, $1e-5$ learning rate and $\beta=0.1$. All experiments are conducted on 4 Nvidia A800-80GB GPUs in a single node.

### B.2 Win rate prompt for Llama and Qwen

We use the same prompt for Llama and Qwen to evaluate a pair of responses:

For the following query to a chatbot, which response is more helpful?
Query: [ ]
Response A: [ ]
Response B: [ ]
FIRST provide a one-sentence comparison of the two responses and explain which you feel is more helpful. SECOND, on a new line, state only "A", "B", "Neither" or "Both" to indicate which response is more helpful. Your response should use the format: Comparison: [one-sentence comparison and explanation] More helpful: ["A", "B", "Neither" or "Both"]

### B.3 Prompt for Llama and Qwen to Label Ties

We use the same prompt for Llama and Qwen to label whether a pair of responses are tied:

For the following query to a chatbot, are the two responses equally good?
Query: []
Response A: []
Response B: []
Answer with exactly "Yes" or "No".

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: See weaknesses.
soundness: 2
strengths: 1. This paper proposes the BTT model in RLHF for more reliable preference learning.
2. This paper provide the theoretical results for the readers to better understand the effectiveness of the proposed method. 
3. This paper is well written and easy to be understanded.
summary: This paper investigates the importance of handling ties in RLHF. The traditional Bradley-Terry (BT) model for preference modeling assumes that there is always one option that is clearly preferred, without considering the fact that humans may consider some options as ties. To address this issue, the authors propose using an extended Bradley-Terry model (BTT), which can more accurately reflect human preferences. They show that when the actual preference model should include ties but there are no such annotations in the dataset, it will lead to bias in the measurement of preference. According to the experimental results, the authors show how the BTT model can reduce bias and improve the accuracy of preference modeling.
weaknesses: 1. **Method:** This paper based on the problem setting of the existence of ties, but the problem setting needs to relabel the preference data for the debias method, which is not suitable for most existing public preference datasets. For this problem, I think the author should focus on the noisy label of the preference dataset, rather than relabeling the data and make some analysis. 
2. **Theory:** The theoretical analysis is based on the ground truth label of ties, rather than the noisy label modeling with the ties. Thus, the superiority of the theoretical results are not meaningful. 
3. **Experiments:** The experimental evaluation results are not rational. The main simulation dataset construction is not clear enough, the evaluation on the HH dataset is also not convincing due to the lack of the ties samples. Additionally the  evaluation metrics should include the commonly used benchmark, such as alpaca eval, mt bench and arena hard etc. Also the authors should investigate some robust preference learning methods as their baselines, such as [1][2].

**References**

[1] Robust Preference Optimization for Large Language Models

[2] Robust Preference Optimization through Reward Model Distillation

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: How to infer the necessity of explicitly introducing ties from Table 1?
soundness: 2
strengths: It is interesting to notice ties might exist in current preference dataset, which might bring up some issue in current preference optimization approaches.

The paper is easy to follow.
summary: This paper investegates the limitation of current RLHF and preference optimization approaches in LLMs: current approaches use the Bradley-Terry (BT) model which forces binary choices between two options, but doesn't account for when responses might be equally good (ties). To address this, the authors propose to using the Bradley-Terry model with ties (BTT) instead of the standard BT to better reflect human preference patterns with ties (TDPO). Afterwards, the experiments demonstrate improved performance using synthetic datasets with ties labeled by open-source LLMs.
weaknesses: I do not see the necessity to introduce algorithm to deal with ties from Table 1. As stated in Lines 56-58, the reward models are trained with BT assumption. And Table 1 shows that these reward models could not distinguish the ties in the datasets due to 0 preference strength. However, the chosen and rejected responses in Table 1 are both suitable as the preferred response. The reward models of BT assign similar rewards to the chosen and rejected responses, which means the BT reward model could already recognize ties without explicitly introducing BTT. Therefore, the necessity of introducing BTT should be further explained.

The experiment is not sufficient enough. Only one dataset (HH) is used with only one baseline model (DPO). Preference optimization has been explored for more than a year. There are several datasets that could be used to evaluate the performance of preference optimization. Moreover, more recent advances in preference optimization, especially those considering  noise in datasets, should be reviewed and be integrated as baselines.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: Please refer to weakness part.
soundness: 2
strengths: - Provides theoretical insights on the importance of incorporating ties in preference modeling
- Proposes an algorithm to address model mismatch problems in conventional preference datasets that lack tie annotations
- Conducts comprehensive experiments to verify the proposed methodology
summary: This paper investigates the impact of annotated ties in preference data on subsequent reward differences and subtask fine-tuning. The authors generalize the Bradley-Terry model to include ties (BTT) and demonstrate the advantages of incorporating ties in preference modeling from both theoretical and experimental perspectives.
weaknesses: Importance of the problem:  
- While incorporating ties in preference modeling is conceptually important, questions arise about its practical significance.  If ties occur infrequently in real annotations, as noted in lines 457-459 ("We observe that Anthropic's HH-RLHF dataset contains over 160k samples, with only a small portion labeled as ties"), the significance of this issue may be limited. 
- Moreover, how does the significance of ties change with data scaling? Would the impact of ties diminish as the dataset size increases?

Questions about experiments:
- How are the parameter   of the BTT model $\theta$ determined?
- While the experiments were conducted on synthetic preference datasets, would it be possible to do experiments using publicly available real-annotated datasets?

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 1
questions: The current work appears rushed, both in experimental design and writing. To improve, the authors should optimize figure and table layouts for better readability, include comparisons with baseline methods to validate the effectiveness of the proposed BTT model, and diversify the datasets used to ensure the generalizability of the findings. A more thorough and polished approach is needed.
soundness: 2
strengths: 1. **Relevance of Preferences with Ties**: The inclusion of ties in preference modeling is highly significant, as real-world scenarios often involve ranking difficulties. This makes the problem both relevant and worthy of exploration.

2. **Clarity and Readability**: The paper is well-structured and easy to follow. The logical flow and clear explanations enhance the overall readability, making the content accessible and understandable.
summary: This paper introduces the Bradley-Terry model with ties (BTT) to improve the accuracy of human preference modeling in reinforcement learning, addressing the bias introduced by ignoring ties. It demonstrates through theoretical analysis and experimental validation that BTT outperforms the traditional BT model, especially in datasets with ties, thereby enhancing the alignment of large language models with human preferences.
weaknesses: 1. **Poor Layout and Presentation**: The formatting of figures (e.g., Figure 1, Figure 2) and tables (e.g., Table 2, Table 3, Table 4, Table 5) is suboptimal, often appearing overly large and occupying excessive space. This layout issue detracts from the readability and professional appearance of the paper, suggesting a rushed submission. Enhancing the visual presentation and ensuring that figures and tables are appropriately sized would significantly improve the reader's experience.

2. **Lack of Baseline Comparisons**: While the paper introduces a novel method, it fails to compare its performance against established baselines. Specifically, the proposed method is described as a variant of DPO with an offset (ODPO) (Amini et al., 2024), but no direct comparisons are provided. This omission makes it difficult to assess the effectiveness and reliability of the new approach. Including such comparisons would strengthen the paper's claims and provide more concrete evidence of its contributions.

3. **Limited Dataset Diversity**: The experiments rely solely on the HH-RLHF dataset, which limits the generalizability of the findings. Using a more diverse set of datasets would help validate the robustness and broader applicability of the proposed method. Expanding the experimental scope to include additional datasets would provide a more comprehensive evaluation.

Meta-review: This paper explores the critical role of addressing ties in Reinforcement Learning with Human Feedback (RLHF). The traditional Bradley-Terry (BT) model for preference modeling assumes a strict preference hierarchy, overlooking the possibility that humans may view certain options as equally preferable. To overcome this limitation, the authors propose an extended Bradley-Terry model (BTT) that incorporates ties, providing a more accurate representation of human preferences. Their findings reveal that when ties are inherent in the preference model but are absent from the dataset annotations, it introduces bias into preference measurements. Experimental results demonstrate that the BTT model effectively reduces this bias and enhances the accuracy of preference modeling.

Major positive points：
+ The paper is well written.
+ The paper provides comprehensive experiments.

Major negative points
- Parts of the motivations are not well justified (reviewer k4NC) 
- The experiments are not comprehensive enough to convince the reviewers.
- The simulation dataset is not well explained.

Decision: Reject