Venue: NAACL 2025

# DPL: Diverse Preference Learning Without A Reference Model

Abhijnan Nath\({}^{1*}\), Andrey Volozin\({}^{2}\), Saumajit Saha\({}^{2}\),

**Albert Aristotle Nanda\({}^{2}\), Galina Grunin\({}^{2}\), Rahul Bhotika\({}^{2}\) and Nikhil Krishnaswamy\({}^{1}\) \({}^{1}\)Colorado State University, \({}^{2}\)Optum AI abhijan.nath@colostate.edu,andrey.volozin@gmail.com**

This work performed during an internship at Optum AI during Summer 2024.

###### Abstract

In direct preference alignment of LLMs, most existing methods seek to retrieve the reward function directly from preference data. However, real-world preference data often contains _diversity_ in preference annotations reflective of true human preferences. Existing algorithms, including KTO Ethayarajh et al. (2024), do not directly utilize such nuances in the annotations which limits their applicability. In this work, we propose Diverse Preference Learning (DPL), a reference model-free method that simultaneously learns a baseline desirability in LLM responses while being robust to the diversity of preference annotations. Our experiments for instruction-following on Ultrafeedback and AlpacaEval 2.0 and for text-summarization on Reddit TL;DR suggest that DPL is consistently better at learning the diversity of preferences compared to existing methods, including those that require a reference-model in memory. Apart from overall quality, we find that DPL's completions, on average, are more honest, helpful, truthful and safe compared to existing methods.

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 3727–3747 April 29 - May 4, 2025 ©2025 Association for Computational Linguistics

## 1 Introduction

Well-known approaches to human preference learning in LLMs, including RLHF-based methods like Proximal Policy Optimization (PPO; Schulman et al. (2017)), require costly reward models to guide alignment, limiting their utility in practical resource-constrained settings. In contrast, offline algorithms like Direct Preference Optimization (DPOR; Rafailov et al. (2024)) provide better compute-efficiency by optimizing directly on preference data _without_ an explicit reward model. Reference-model free variants like ORPO Hong et al. (2024) and SIMPO Meng et al. (2024) do this without a reference model in memory. At the core of direct alignment methods is the Bradley-Terry (BT) model Bradley and Terry (1952), which models preferences as implicit reward differences between "good" and "bad" responses.

While appealing for their simplicity, the BT model assumption that true human preferences directly correlate with model-assigned implicit reward differences is problematic: firstly, popular preference datasets like Ultrafastedback Cui et al. (2024) contain a non-negligible amount (\(\sim\)17%) of weak preference pairs that are rated "equally" (reward difference \(\sim\)0) by high-capacity automatic evaluators like GPT-4. When the preference signal is weak, common direct alignment frameworks can lead to significant policy degeneracy or underfitting, especially under finite data settings Azar et al. (2024). Additionally, such underfitting can effectively limit such methods from learning a "baseline desirability" of all available responses, especially for weak preference pairs where both examples are rated low. (See Figure 1 for an illustration.)

In particular, reward underspecification in the Bradley-Terry model Bertrand et al. (2023) can hinder these fameworks from capturing the nu

Figure 1: Reward estimates (preference strength) from GPT-4 on the Ultrafeedback dataset Cui et al. (2024). Diverse Preference Learning (DPL) learns both a “base-line desirability” as well as the “relative goodness” between responses, outperforming competitive baselines including KTO Ethayarajh et al. (2024) on popular preference learning benchmarks (see Table 2).



anced preferences or relative "goodness" between responses--which is arguably available in widely-used datasets. This is true when the preference distribution1 is varied or more nuanced, i.e., the reward differences fall within a _spectrum_ as shown in Figure 1. Even more sophisticated methods like Kahneman-Tversky Optimization (KTO; Ethayarajh et al. (2024)), often reduce pointwise preferences to binary labels, sacrificing the granularity needed to fully leverage rich annotations.

Footnote 1: While the true distribution of human preferences is unknown and likely multidimensional Tversky (1969), we assume GPT-4’s reward estimates provide a reasonable approximation, particularly when direct annotations are unavailable.

In this work, we introduce **Diverse Preference Learning (DPL)**, a novel reference-model-free alignment framework that enables policies to capture the rich, often overlooked nuances of human feedback. DPL simultaneously learns baseline desirability and optimizes for the relative strength of preference labels, resulting in a more accurate and nuanced representation of true human preferences.

Our novel contributions are:

* We provide a practical implementation of **Diverse Preference Learning (DPL)**, addressing limitations in BT model-based approaches with two simple reward formulations derived from popular frameworks like ORPO Hong et al. (2024) and SIMPO Meng et al. (2024), consistent with a reference model-free setting.
* We demonstrate the effectiveness of DPL in learning diverse preferences on two popular benchmarks: Ultrafeedback Cui et al. (2024) and Reddit TL;DR CNN/Daily Mail) Volske et al. (2017); Stiennon et al. (2020), using both LLM-as-a-judge based evaluation as well as a Reward Model.
* Using Cui et al. (2024)'s annotation framework, we provide a "first-past-the-post" (FPP)-style evaluation of DPL on multidimensional preference axes such as _honesty_, _helpfulness_, _truthfulness_ and _soafety_, where it significantly outperforms competitive baselines such as Supervised Finetuned (SFT) models, ORPO, SIMPO, and KTO. Additionally, DPL also outperforms such baselines on popular leaderboards such as AlpacaEval 2.0.

## 2 Related Works

Reinforcement Learning from Human Feedback (RLHF) aims to harmonize large language models with human preferences and values Christiano et al. (2017); Ziegler et al. (2019); Ouyang et al. (2022); Bai et al. (2022). The conventional RLHF pipeline typically consists of three primary phases: supervised fine-tuning Zhou et al. (2023); Taori et al. (2023); Geng et al. (2023); Conover et al. (2023); Kopf et al. (2023); Ding et al. (2023); Wang et al. (2024b); Chen et al. (2024a); Xia et al. (2024), reward model training Gao et al. (2023); Luo et al. (2023); Chen et al. (2024b); Lightman et al. (2023); Havrilla et al. (2024); Lambert et al. (2024), and policy optimization Schulman et al. (2017); Anthony et al. (2017). More recently, supervised offline contrastive learning approaches like DPO Rafailov et al. (2024), IPO Azar et al. (2024), e-DPO Fisch et al. (2024), KTO Ethayarajh et al. (2024), and DPO-Positive Pal et al. (2024) have been proposed. These approaches include a reference model in memory as a baseline policy and intend to minimize compute by avoiding online sampling for reward maximization. Specifically, algorithms like DPO and KTO seek to retrieve optimal policies that analytically derive from the KL-constrained optimization problem Ziebart et al. (2008); DPO seeks to align with human preferences, and KTO, with human preferences and utility.

In contrast, reference-free approaches like CPO Xu et al. (2024), ORPO Hong et al. (2024) and SIMPO Meng et al. (2024) often assume the reference model is a uniform distribution, and proposes alternative Bradley-Terry reward formulations, which increase efficiency by dispensing with the need for an additional reference model. Dubey et al. (2024), Dong et al. (2024) and Hong et al. (2024) have explored including an SFT-based regularization on preferred responses directly in the alignment objective, whereas Yang et al. (2024) add an SFT term for more generalized reward learning and Nath et al. (2024) simultaneously learn preferences and rewards with reward-distillation. Specifically, our work can be seen extending this line of work but with a novel unlikelihood Welleck et al. (2020) penalty on dispreferred responses, that along with our nuanced preference categorization, allows more diverse preference alignment.

## 3 Background: Implicit Rewards (\(r^{*}\))

In "supervised" preference alignment, we are given a dataset of pairwise preference data \(\mathcal{D}_{\text{pref}}=\{x,y_{w},y_{l},s_{ij}\}_{i=1}^{N}\), where \(x\) denotes the prompt or context, \(y_{w}\) and \(y_{l}\) are the preferred and dispreferred completions respectively, and \(s_{ij}\) represents the strength of the relative preference between 

and \(y_{l}\).2 The parameters of the aligned policy \(\pi_{\theta}\) are then estimated "directly" on \(\mathcal{D}_{\text{pref}}\) by formulating the preference of \(y_{w}\) over \(y_{l}\) using the Bradley Terry (BT) model as:

Footnote 2: Datasets such as Ultrafeedback Cui et al. (2024) do not provide explicit human preference labels due to annotation costs Lambert et al. (2024). Instead, they use Likert scale rewards from models like GPT-4 and inferred preference strengths (\(s_{ij}\)) to approximate human preferences, converting generic response pairs into \(y_{w}\) and \(y_{l}\) labels.

\[p^{*}(y_{w}\succ y_{l}\mid x)=\sigma(r^{*}(x,y_{w})-r^{*}(x,y_{l})) \tag{1}\]

where \(p^{*}(y_{w}\succ y_{l}|x)\) denotes the true probability that humans prefer \(y_{w}\) over \(y_{l}\), \(\sigma(\cdot)\) is the sigmoid function and \(r^{*}(x,y)\) is a latent reward function estimated over the observed preference data.

Reference model-based implicit rewardsThe core idea in RLHF is that \(\pi_{\theta}\) should not diverge too much from the reference model \(\pi_{\text{ref}}\), the latter being the initialization point of \(\pi_{\theta}\) in most direct alignment algorithms like DPO Rafailov et al. (2024), after undergoing supervised finetuning (SFT). In DPO, this divergence constraint between \(\pi_{\theta}\) and \(\pi_{\text{ref}}\) is realized by choosing a very specific formulation of \(r^{*}(x,y)\), which is an analytical derivative of the KL-constrained optimal policy \(\pi_{\theta}^{*}\), defined as \(\frac{1}{Z(x)}\pi_{\text{ref}}\exp\left(\frac{1}{\beta}r^{*}(x,y)\right)\). On the other hand, Kahneman-Tversky Optimization KTO; Ethayaraj et al. (2024)), optimizes \(\pi_{\theta}\) using a redefined \(r^{*}\) where the policy learns to distinguish between \(y_{w}\) and a more distributional form of \(y_{l}\)--the idea being that humans make choices based on many alternative possibilities--unlike DPO where the comparison is only with a single \(y_{l}\) per sample.

Reference model-free implicit rewardsIn contrast, reference model-free frameworks like ORPO Hong et al. (2024) and SIMPO Meng et al. (2024) assume a uniform prior for \(\pi_{\text{ref}}\sim\mathcal{U}(\mathcal{Y})\). These approaches, which are both computationally efficient and effective, redefine \(r^{*}(x,y)\) in innovative ways, eliminating the need for a reference model in memory. Our approach for reference **Diverse Preference Learning** (DPL) draws its theoretical insights from these two reward formulations, which are given below3:

Footnote 3: For analysis, \(r^{*}_{\text{odds}}\) (without an SFT term) and \(r^{*}_{\text{norm}}\) without a margin \(\gamma\)) are simplified forms of the ORPO Hong et al. (2024) and SIMPO Meng et al. (2024) objectives.

\[r^{*}_{\text{odds}(x,y)} =\beta\log\frac{\pi_{\theta}(y\mid x)}{1-\pi_{\theta}(y\mid x)} \tag{2}\] \[r^{*}_{\text{norm}(x,y)} =\beta\frac{\log\pi_{\theta}(y\mid x)}{|y|} \tag{3}\]

where \(\beta\) is the KL-beta parameter and \(|y|\) represents the number of tokens in a response \(y\). Intuitively, it is easy to see that even without a reference model, the complement of the likelihood in the denominator in Eq. 2 and the length-normalization factor in Eq. 3 act as "surrogates" for \(\pi_{\text{ref}}\)-based KL regularization. Now, applying a sigmoid function over the RHS of Eq. 2 and Eq. 3, one can then formulate the preference of \(y_{w}\) over \(y_{l}\) under the Bradley-Terry model as:

\[p^{*}(y_{w}\succ y_{l}\mid x) =\sigma(r^{*}_{\text{odds}(x,y_{w})}-r^{*}_{\text{odds}(x,y_{ l})}) \tag{4}\] \[p^{*}(y_{w}\succ y_{l}\mid x) =\sigma(r^{*}_{\text{norm}(x,y_{w})}-r^{*}_{\text{norm}(x,y_{l})}) \tag{5}\]

The implication here is that the BT model allows us to represent the true preference distribution (\(p^{*}\)) in terms of the latent reward functions. As such, a naive way to estimate the parameters of these two reward formulations, and thereby retrieve the aligned \(\pi_{\theta}\), is to maximize the loglikelihood of RHS of Eq. 4 and Eq. 5. However, if the true preferences are deterministic, i.e., if \(p^{*}(y_{w}\succ y_{l})\in\{0,1\}\), or \(s_{ij}\gg c\) (surpassing a baseline preference strength \(c\)), \(\pi_{\theta}\) can overfit to BT-assigned reward estimates and underfit the true preferences, especially since the number of parameters in \(\pi_{\theta}\) typically exceeds the number of (\(y_{w}\), \(y_{l}\)) pairs in the preference training dataset, as argued previously in Azar et al. (2024).

For clarity, let us examine a scenario where two responses, \(y_{w}\) and \(y_{l}\), exist such that the preference probability \(p^{*}(y_{w}\succ y_{l})=1\), i.e., \(y_{w}\) is consistently favored over \(y_{l}\). The Bradley-Terry model would then require that the difference \((r^{*}_{\text{odds}}(x,y_{w})-r^{*}_{\text{odds}}(x,y_{l}))\) must approach \(+\infty\) to comply with Eq. 4. Substituting Eq. 2 in Eq. 4, one can clearly see that the log-probability ratio is \(\log(\frac{\pi_{\theta}(y_{w})}{\pi_{\theta}(y_{l})})\rightarrow\infty\), which implies \(\pi_{\theta}(y_{l})\to 0\), especially _without_ a reference model to control for this ratio, irrespective of the _any_ finite KL-regularization parameter \(\beta\). Such a scenario leads to \(\pi_{\theta}\) being underfitted to the preference data in these reward function parameter estimations. Additionally, if \(p^{*}(y_{w}\succ y_{l})<1\) for a pair which is possible when relative preference strengths lie on a spectrum, \(\pi_{\theta}\) can _empirically_ estimate this probability as 1 under finite preference data. This can lead to the policy empirically overfitting such BT-assigned infinite rewards and create substantial degeneracies where the aligned \(\pi_{\theta}\) emits tokens that are not even present in the training data, especially 

under finite-data settings (Fisch et al., 2024). See Appendix A for an equivalent example for \(r^{*}_{\text{norm}}\).

## 4 Diverse Preference Learning Framework (DPL)

To prevent such degeneracies, we aim to define the DPL objective such that it (a) is reference model-free for computational efficiency, (b) uses a BT model-based loss to learn from pairwise contrastive preference data, (c) regularizes the loss function to avoid overfitting to the "good" and "bad" responses in contrastive feedback pairs, (d) incorporates unpaired "good" and "bad" responses when available, and (e) offers a flexible weighting scheme to capture the granular desirability of samples when such information is present.

To achieve these objectives, we propose two variants of the DPL objective, \(L_{\text{DPL}_{odds}}\) and \(L_{\text{DPL}_{norm}}\). These two DPL objectives represent the odds-ratio-based and the length-normalized formulations of the implicit reward signal, respectively.

Mathematically,

\[L_{\text{DPL}_{mods}}(\pi_{\theta})= -\mathbb{E}_{(x,y_{w},y_{l})\sim\mathcal{D}_{\text{pref}}}\left\{ \alpha\cdot L_{\text{SFT}}\right.\] \[+\gamma\cdot\log\big{[}\sigma\left(r^{*}_{\text{addx}(x,y_{w})}- r^{*}_{\text{addx}(x,y_{l})}\right)\big{]}\] \[\left.+\eta\cdot\log(1-\pi_{\theta}(y\mid x))\right\} \tag{6}\]

\[L_{\text{DPL}_{norm}}(\pi_{\theta})= -\mathbb{E}_{(x,y_{w},y_{l})\sim\mathcal{D}_{\text{pref}}}\left\{ \alpha\cdot L_{\text{SFT}}\right.\] \[+\gamma\cdot\log\big{[}\sigma\left(r^{*}_{\text{mem}(x,y_{w})}- r^{*}_{\text{nom}(x,y_{l})}\right)\big{]}\] \[\left.+\eta\cdot\log(1-\pi_{\theta}(y_{l}\mid x))\right\} \tag{7}\]

We can rewrite the above two formulations of the \(L_{\text{DPL}_{mods}}\) and \(L_{\text{DPL}_{norm}}\) losses in their full parametric form by replacing the \(L_{\text{SFT}}\) with \(\log(\pi_{\theta}(y_{w}\mid x)\) and replacing \(r^{*}_{\text{odds}(x,y)}\) and \(r^{*}_{\text{nom}(x,y)}\) with their parametric form from Eq. 4 and Eq. 5, respectively. The full parametric forms of the DPL losses are represented below:

\[L_{\text{DPL}_{mods}}(\pi_{\theta})= -\mathbb{E}_{(x,y_{w},y_{l})\sim\mathcal{D}_{\text{pref}}}\left\{ \alpha\cdot\log(\pi_{\theta}(y_{w}\mid x)\right.\] \[+\gamma\cdot\log\bigg{[}\sigma\left(\beta\log\frac{\pi_{\theta}( y_{w}\mid x)}{1-\pi_{\theta}(y_{w}\mid x)}\right.\] \[\left.\hskip 39.83385pt-\beta\log\frac{\pi_{\theta}(y_{l}\mid x)}{ 1-\pi_{\theta}(y_{l}\mid x)}\right)\bigg{]}\] \[+\eta\cdot\log(1-\pi_{\theta}(y_{l}\mid x))\} \tag{8}\]

\[L_{\text{DPL}_{norm}}(\pi_{\theta})= -\mathbb{E}_{(x,y_{w},y_{l})\sim\mathcal{D}_{\text{pref}}}\left\{ \alpha\cdot\log(\pi_{\theta}(y_{w}\mid x)\right.\] \[+\gamma\cdot\log\bigg{[}\sigma\left(\beta\frac{\log\pi_{\theta}(y _{w}\mid x)}{|y_{w}|}\right.\] \[\left.\hskip 39.83385pt-\beta\frac{\log\pi_{\theta}(y_{l}\mid x)}{| y_{l}|}\right)\bigg{]}\] \[+\eta\cdot\log(1-\pi_{\theta}(y_{l}\mid x))\} \tag{9}\]

where the weights \(\alpha\), \(\gamma\), and \(\eta\) are empirically determined from the preference dataset and \(|y_{w}|\), \(|y_{l}|\) are the token-lengths of the winning and losing responses respectively.

Now, let us examine the motivation behind these choices. The \(L_{\text{SFT}}\) term, which is a cross-entropy loss applied to all "good" responses (\(y_{w}\)), is used in ORPO but not proposed in SIMPO. We hypothesize that this term plays two key roles: (1) it mitigates overfitting to "bad" samples (\(y_{l}\)) in contrastive pairs, and (2) it enables learning from unpaired "good" responses. Similarly, the unlikelihood loss term \(\log(1-\pi_{\theta}(y_{l}\mid x))\) prevents overfitting to "good" samples and supports learning from unpaired "bad" responses (as in suppressing \(y_{l}\)'s likelihood during alignment).

Finally, the \(\alpha\), \(\gamma\), and \(\eta\) weights can be seen as a combination (e.g., products) of sample-independent regularization hyperparameters or sample-wise weights reflecting baseline desirability or relative quality. For instance, in our Ultrafeedback dataset experiments (Section 5), we derived sample-wise weights from baseline desirability categories such as "accepted," "partially accepted," and "rejected." In many real-world applications, more granular response quality scores may be available, allowing for more advanced sample weighting approaches (Touvron et al., 2023; Kirk et al., 2024).

A sample-efficient way for \(\pi_{\theta}\) to learn such granularity is to use a "weighting" function (Wiseman et al., 2015) that applies regularization dynamically to \(y_{w}\) and \(y_{l}\) based on the _distribution_ of relative preference strengths \(s_{ij}\) provided in \(\mathcal{D}_{\text{pref}}\).4 This function \(\mathcal{H}(\mathcal{D}_{\text{pref}}):\mathcal{X}\rightarrow(\alpha,\gamma,\eta)\) ascertains the relative importance that \(\pi_{\theta}\) should place on \(y_{w}\) and \(y_{l}\) during alignment.

Footnote 4: Note that although \(s_{ij}\) is represented here as a single rating where \(s_{ij}\in\mathbb{R}\), it can also be denoted as a binary tuple with individual scores for each response.

As such, DPL works in two phases: first, a two-layered categorization of preference pairs is computed from \(\mathcal{D}_{\text{pref}}\) based on their i) baseline desirability and ii) relative strengths. Secondly, DPL applies a weighting func 

rization to dynamically learn a refined distribution of preferences.

Determining Baseline DesirabilityA practical way to determine the baseline desirability is to decompose \(s_{ij}\) into a tuple \((s_{i},s_{j})\) such that \(s_{i}\geq s_{j}\) and partition \(\mathcal{D}_{\text{pref}}\) into categories: "accepted," "partially-accepted," and "rejected," signifying a decreasing order of baseline desirability. Specifically, response pairs are defined as "accepted" if \(s_{i}\geq T_{a}\) and \(s_{j}\geq T_{a}\), "partially-accepted" if \(T_{p}\leq s_{i}<T_{a}\) and \(T_{p}\leq s_{j}<T_{a}\), and "rejected" if \(s_{i}<T_{p}\) and \(s_{j}<T_{p}\), where \(T_{a}\) and \(T_{p}\) are precomputed from \(\mathcal{D}_{\text{pref}}\), akin to KTO [15]. The process allows us to retrieve the individual instance of \(y_{w}\) that each fall within these categories.

How "Good" is \(y_{w}\) Relative To \(y_{l}\)?Though intuitive, baseline desirability alone is not sufficient since ideally we also want \(\pi_{\theta}\) to learn the goodness of \(y_{w}\) relative to \(y_{l}\). As such, we also retrieve the relative placement of \(y_{l}\) in terms of \(y_{w}\)'s complement by determining where \(s_{j}\) falls between \(T_{a}\) and \(T_{p}\). To do this, if the lower score, \(s_{j}\), does not satisfy the same threshold of \(y_{w}\), then the pair \((s_{i},s_{j})\) is termed "accepted/rejected" if \(s_{j}<T_{p}\). Similarly, if the scores are almost equal (\(s_{i}\sim s_{j}\)), indicating _no_ preference, we call the preference relation in this pair nondeterministic or equally preferred.

Interplay between Baseline Desirability and Relative Preference StrengthWhen the baseline desirability is "accepted," DPL increases the weight on the SFT component for \(y_{w}\) with \(\alpha\). Concurrently, if the complement \(y_{l}\) is categorized as "rejected," it boosts the unlikelihood loss on \(y_{l}\) with \(\eta\), while allowing \(r^{*}\) to utilize the contrastive preference signals, as outlined in Eq. 2 and Eq. 3, weighted by \(\gamma\). Contrastingly, if both \(y_{w}\) and \(y_{l}\) are equally good (\(s_{i}\sim s_{j}\)) within the "accepted" category, indicating no preference, DPL masks out both the contrastive reward \(r^{*}\) and the unlikelihood term by setting both \(\gamma\) and \(\eta\) to zero.

## 5 Experiments

### Datasets

We primarily assess our our DPL approach on two preference learning tasks: **single-turn instruction following** in Ultrafeedback [46] and **text summarization** of news articles in the **Reddit TL;DR** summarization dataset [78, 90].

For Ultrafeedback5, we use the GPT-4 assigned reward estimates to compute the baseline desirability thresholds, \(T_{a}\) and \(T_{p}\). Since reward estimates here are scalar scores ranging from 1 to 10, we choose \(T_{a}=7.0\) and \(T_{p}=4.0\) for allotting samples into the three categories of "accepted," "partially-accepted," and "rejected." We use these same thresholds to further categorize the preference pairs based on their relative preference strengths.

Footnote 5: [https://huggingface.co/datasets/argilla/ultrafeedback-binarized-prereferences-cleaned](https://huggingface.co/datasets/argilla/ultrafeedback-binarized-prereferences-cleaned)

For Reddit TL;DR6, without reward estimates, we use expert confidence scores at the 66\({}^{th}\) and 33\({}^{rd}\) percentiles for \(T_{a}\) and \(T_{p}\). Additionally, we utilize edit distance scores [11] using the editdistance library7 for consistency in relative preference categorization, applying the same percentiles.

Footnote 6: Using the original data: [https://github.com/openai/summarize-from-feedback](https://github.com/openai/summarize-from-feedback), with human confidence labels for DPL-based categorization.

Footnote 7: [https://dypgi.org/project/editdistance/](https://dypgi.org/project/editdistance/)

### DPL Weighting Function:Hyperparameters

Table 1 shows details of optimal importance-weights \(\alpha\), \(\gamma\) and \(\eta\) applied by DPL's weighting function \(\mathcal{H}(\mathcal{D}_{\text{pref}}):\mathcal{X}\rightarrow(\alpha,\gamma,\eta)\) during training. Due to a limited compute budget, we did not conduct a more exhaustive search of these three parameters for each category. To restrict this search, we heuristically assign values of \(\alpha\), \(\gamma\) and \(\eta\) that maintain the range of values within the baseline preference desirability. For example, for \(\alpha\) and \(\gamma\) that works on the SFT and the constrastive rewards respectively, an "accepted/partially-accepted" pair would get a higher \(\alpha\) compared to a "partially-accepted/rejected" sample while \(\gamma\) is consistent since the SFT component provides sufficient regularization in this case. Using this intuition allowed us to still conduct a reasonable search for optimal values on the validation data, within our compute constraints. Future research can likely find a more optimal set of weights that helps learn a diverse representation of preferences from desiderata already present in preference datasets. See Table 1 for the distribution of importance weights corresponding to each type of preference label.

### Baselines and Training settings

We use the Phi-3-Mini-128k-Instrect model [15]R for all our experiments including 

baselines. For an in-depth evaluation consistent with previous research, we compare to a supervised finetuned (SFT) baseline as well as the original objective formulations of ORPO Hong et al. (2024) and SIMPO Meng et al. (2024). Furthermore, in order to directly compare DPL to a method that uses pointwise preference labels, we also include the KTO Ethayarajh et al. (2024) baseline. In instruction-following, we train the SFT model on the preferred (chosen) responses in Ultrafeedback whereas for summarization, the SFT model is trained until convergence on the human-written forum post summaries. For both SFT versions, we use TRL's SFTT trainer library.9 For ORPO and SIMPO variants, we use default hyperparameter settings from respective trainer libraries in TRL. For KTO, we use default desirable and undesirable weights (\(\lambda_{D}=\lambda_{U}=1\)) but initialize \(\pi_{\theta}\) with the SFT model, similar to SIMPO (see Appendix D).

Footnote 9: [https://huggingface.co/docs/trl/en/sft_trainer](https://huggingface.co/docs/trl/en/sft_trainer)

AblationsFor ablations, we train four variants: DPL-O (using Eq. 6) and DPL-N (using Eq. 7), including all samples as categorized in DPL-based preferences (Section 4). To robustly evaluate cases where \(\mathcal{D}_{\text{pref}}\) includes samples with zero preference strength or those that are equally preferred, we train versions of DPL-O and DPL-N that explicitly omit such samples. These are denoted DPL-O (-) and DPL-N (-), respectively.

### Evaluation strategies

For evaluation, we first sample responses to instruction prompts from the Ultrafeedback and Reddit TL;DR evaluation sets using top-\(p\) nucleus sampling with \(p=0.8\) and a uniform temperature of 0.7 across all baselines.

For LLM-based automatic evaluation, we conduct a one-shot "first-past-the-post" (FPP)-style assessment with GPT-4o as the judge, utilizing annotation templates from Cui et al. (2024) for quality ratings on a Likert scale between 1 to 5. This approach enables us to evaluate preference learning on various human preference dimensions including _overall performance, honesty, helpfulness, truthfulness_, and _safety_, which is more comprehensive than singular win-rate metrics like those in AlpacaEval Li et al. (2023). For this, all baseline generations are simultaneously rated (with randomized positional swapping) where ties are broken with a variant10 of the Borda-count Emerson (2013). For a more in-depth pairwise comparison of generation quality, we separately train an OPT 1.3B Zhang et al. (2022) reward model (RM) following Hong et al. (2024) and compute reward-based win-rates of all our ablated variants against the baselines. See Table 2 and Table 3 for FPP evaluation results and OPT-RM pairwise win-rates.

Footnote 10: For instance, for ties with three top-scoring candidates, we add \(1/3\) to each model’s current win-count.

For thoroughness, we also include results from the AlpacaEval 2.0 benchmark, using model-generations from Ultrafeedback-trained baselines. This benchmark tests LLMs on instruction-following capabilities using 805 curated questions. To prevent self-rewarding issues in automatic evaluators Yuan et al. (2024), we set GPT-4o as the judge, comparing baseline outputs to precomputed GPT-4 Turbo generations, following the official AlpacaEval prompt format.11 See Table 2's "AE2" column for win-rates and Appendix G for details on evaluation prompts.

## 6 Results

Here we present automatic evaluation results in the LLM-as-a-judge setting (Section 6.1) and ablation results with a traditional RM (Section 6.2). Appendix I contains example model outputs.

### LLM-as-a-Judge Evaluation

Table 2 shows tie-broken win-rates of all baselines along with DPL models evaluated with GPT-4o.

DPL variants consistently outperform baselines, including KTO, on both datasetsAs shown in

\begin{table}
\begin{tabular}{l c r r} \hline \hline
**Preference Categories** & \(\alpha\) & \(\gamma\) & \(\eta\) \\ \hline accepted & 1 & 0 & 0 \\ accepted/partially-accepted & 1 & \(\frac{3}{4}\) & 0 \\ accepted/rejected & 1 & 1 & 1 \\ accepted/nondeterministic & 1 & 0 & 0 \\ partially-accepted/rejected & \(\frac{3}{4}\) & \(\frac{3}{4}\) & 1 \\ rejected & 0 & 0 & 1 \\ rejected/nondeterministic & 0 & 0 & 1 \\ partially-accepted & \(\frac{3}{4}\) & 0 & 0 \\ partially-accepted/nondeterministic & \(\frac{3}{4}\) & 0 & 0 \\ \hline \hline \end{tabular}
\end{table}
Table 1: This table shows the detailed breakdown of \(\alpha\), \(\eta\), and \(\gamma\) values that dynamically modifies the three loss components in the DPL objective function. \(\alpha\), \(\eta\), and \(\gamma\) are assigned based on the two-layered preference categorization using baseline desirability thresholds.

 

Table 2, across all aspects of instruction-tuning alignment, DPL-O achieves the best overall performance across the two task-specific datasets. While all alignment-based baselines outperform the SFT variant, as was observed previously in Meng et al. (2024) and Hong et al. (2024), DPL's significant improvement over ORPO and SIMPO highlights DPL's effectiveness and data-efficiency. DPL's diverse preference learning strategy also outperforms KTO in almost all dimensions except safety, which is significant since KTO includes a reference-model in memory for baseline KL computations and to maximize human utility. For instance, in overall quality assessment, DPL-O outperforms KTO by almost 4 points in the tie-broken win rates while significantly outperforming SIMPO and ORPO (by 7 points each). Such consistency is also seen across aspects like helpfulness, honesty, and truthfulness, as well as safety.

DPL-O consistently outperforms DPL-NBetween the two proposed variants of DPL, where each seeks to retrieve a different \(r^{*}\), DPL-O with its odds-ratio-based rewards consistently outperforms its length-normalized counterpart under the same circumstances. As Table 2 shows, DPL-O's performance gain over DPL-N is consistent especially in its overall quality of completions, as seen in an almost 9 point margin on the TL;DR test set. Notably, KTO also outperforms DPL-N by a slight margin of 2 points in this setting. We find that while length-normalized rewards in \(r^{*}\) can be competitive, the odds-ratio variant is _more robust_ especially in controlled-generation tasks like summarization where the average response lengths are \(\sim\)256 tokens. In this setting, without a reference model, the range of assigned (implicit) reward differences is likely larger for DPL-O compared to DPL-N by design, allowing the former to provide a more nuanced estimation of the preferences.

On benchmarks like AlpacaEval 2.0, DPL-O still wins across baselinesInterestingly, we find that sampled generations from DPL-O consistently outperform all baselines including DPL-N and KTO, when evaluated on curated instruction-tuning benchmarks like the AlpacaEval dataset. As seen in the "AE2" column in Table 2, while all baselines consistently outperform the SFT variant, DPL-O brings a 2 point average improvement in win-rates over KTO and DPL-N, and a 5 point improvement over SIMPO and ORPO. This is crucial since we specifically control for self-rewarding by choosing GPT-4o as judge--although the GPT-4-Turbo model is a popular alternative--while still using GPT-4-Turbo for the baseline completion in the pairwise comparisons over 805 questions.

### Reward-model based evaluation

Table 3 shows results from computing reward-based win-rates, averaged over all test pairs, using the OPT 1.3B model, trained using Eq. 1 as suggested in Hong et al. (2024). We include ablated variants DPL-N (-) and DPL-O (-) (see Section 5.3) and compute reward estimates \(r(x,y)\) using the trained RM on the sampled completions of all baselines. A completion \(y_{1}\) wins over \(y_{2}\) when \(r(x,y_{1})>r(x,y_{2})\), with \(\frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\{r(x_{i},y_{1i})>r(x_{i},y_{2i})\}\) being the averaged win-rate, where \(N\) is the number of samples.

As shown in Table 3, we find that both DPL-N and DPL-O (including ablated variants) outper 

form the SFT baseline, which is consistent with our findings with an LLM-as-judge. For example, on the Ultrafeedback dataset, DPL-O achieves an impressive win rate of 92.1% over the SFT model, while being also preferred over KTO, SIMPO and ORPO 60.5%, 65.7%, and 68.1% of the time, respectively. Such trends are also seen on Reddit TL;DR where gain over SFT takes a slight hit compared to Ultrafeedback, yet the win-rates of DPL-O and DPL-N remain \(\sim\)65% and \(\sim\)59.6% respectively against the non-SFT baselines, on average. DPL-O consistently outperforms DPL-N with higher expected rewards across multiple samplings when evaluated with a trained RM, on both benchmarks. This is also consistent with the performance trend observed with GPT-4o's evaluation (Table 2), thereby indicating DPL-O's overall superiority in these alignment tasks. A statistical significance test on the reward distributions indicates that these differences are in fact statistically significant in the overwhelming majority of cases (Table 4).

We clearly see that both DPL-O and DPL-N tend to bring a rightward shift on expected rewards for both datasets, as evidenced by \(t\)-statistics and corresponding \(p\)-values (\(<0.05\)), except in one case: DPL-N's reward vs. those of ORPO on TL;DR. This suggests that the mean reward difference between DPL and others, correlated with the rightward shift, is significant for both preference datasets.

To analyze the performance trends of DPL-N vs. ORPO on the TL;DR dataset, we investigated the average win-rate and reward advantages as assigned by the OPT 1.3B reward model. For DPL-N, the reward accuracy (win-rate) is 58.9% across 1,000 randomly selected CNN/Daily Mail test prompts, indicating consistent performance with an average win-rate above 50%.

The reward advantage for DPL-N is approximately 0.0140 (\(\sigma=\) 0.23), which, while small, remains positive. In comparison, DPL-O achieves a higher win-rate of 64.6% and a larger reward advantage of 0.1310 (\(\sigma=\) 1.36). This highlights the statistical tie in DPL-N rewards when compared to ORPO. On the other hand, DPL-N vs. SIMPO is a more direct comparison since DPL-N reward derives from SIMPO, and here we see a clear significance in the rightward shift in expected rewards.

Ablations on "equally" preferred samplesNotably, training _without_ "equally" preferred samples (DPL-N (-) and DPL-O (-)) where the preference strength is negligible, leads to average performance drop across both datasets of \(\sim\)2 points in win-rates for DPL-O (-), while the drop is comparatively more for DPL-N (-) (\(\sim\)4 points) as shown in Table 3. This is crucial since this clearly indicates DPL's effectiveness in capturing nuanced preferences especially for samples where the preference strength is weak. Crucially, since equally preferred samples can occur anywhere on the spectrum of baseline desirability, this suggests DPL is more sample-efficient and that regulating the weight on the SFT-term and unlikelihood term using DPL's weighting function is more data-efficient than omitting training samples with weak preferences.

\begin{table}
\begin{tabular}{l l c} \hline \hline  & \multicolumn{2}{c}{**Ultraefeedback**} \\ \cline{2-3}
**Comparison** & \(t\)**-statistic** & \(p\)**-value** \\ \hline DPL-O vs. KTO & 5.075 & \textless{}0.001 \\ DPL-O vs. SFT & 11.515 & \textless{}0.001 \\ DPL-O vs. SIMPO & 7.010 & \textless{}0.001 \\ DPL-O vs. ORPO & 7.042 & \textless{}0.001 \\ DPL-N vs. KTO & 2.488 & 0.013 \\ DPL-N vs. FFT & 8.832 & \textless{}0.001 \\ DPL-N vs. SIMPOP & 4.354 & \textless{}0.001 \\ DPL-N vs. ORPO & 4.355 & \textless{}0.001 \\ \hline \end{tabular} 
\begin{tabular}{l l c} \hline \hline  & \multicolumn{2}{c}{**TL;DR (CNN/Daily Mail)**} \\ \cline{2-3}
**Comparison** & \(t\)**-statistic** & \(p\)**-value** \\ \hline DPL-O vs. KTO & 10.333 & \textless{}0.001 \\ DPL-O vs. SFT & 10.141 & \textless{}0.001 \\ DPL-O vs. SIMPO & 8.096 & \textless{}0.001 \\ DPL-O vs. OORPO & 2.976 & 0.003 \\ DPL-N vs. KTO & 7.351 & \textless{}0.001 \\ DPL-N vs. SIFT & 7.154 & \textless{}0.001 \\ DPL-N vs. SIMPO & 5.146 & \textless{}0.001 \\ DPL-N vs. ORPO & 0.082 & 0.935 \\ \hline \hline \end{tabular}
\end{table}
Table 4: \(p\)-values on statistical tests of the rightward shift in expected rewards on Ultrafeedback and TL;DR.

\begin{table}
\begin{tabular}{l r r r r} \hline \hline  & \multicolumn{4}{c}{**Ultraefedback**} \\ \cline{2-5}
**Comparison** & **SFT** & **ORPO** & **SIMPO** & **KTO** \\ \hline DPL-N (-) & 68.5 & 54.0 & 47.5 & 56.5 \\ DPL-O (-) & 90.8 & 66.0 & 63.8 & 59.9 \\ DPL-N & 69.8 & 58.7 & 51.8 & 58.8 \\ DPL-O & **92.1** & **68.1** & **65.7** & **60.5** \\ \hline \hline \end{tabular} 
\begin{tabular}{r r r r} \hline \hline  & \multicolumn{4}{c}{**TL;DR (CNN/Daily Mail)**} \\ \cline{2-3}
**Comparison** & **SFT** & **ORPO** & **SIMPO** & **KTO** \\ \hline DPL-N (-) & 77.6 & 54.3 & 53.8 & 52.2 \\ DPL-O (-) & 78.7 & 63.2 & 61.3 & 65.4 \\ DPL-N & 75.9 & 58.9 & 59.0 & 60.8 \\ DPL-O & **79.2** & **64.6** & **64.7** & **65.8** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Rewards accuracies or win rates computed with an OPT 1.3B reward model Hong et al. (2024) on generations sampled across pairwise runs with top-\(p\) of 0.8 and temperature of 0.7.

 

DPL shifts expected rewards positivelyWe further evaluate the distribution of expected rewards (\(r(x,y)\)) assigned by the OPT 1.3B reward model (RM) over sampled generations from each baseline on both Ultrafeedback and Reddit TL;DR test sets and plot the histogram of expected rewards in Figure 2. Consistent with prior work Wang et al. (2024a); Hong et al. (2024), we find that the overall reward distribution shifts under alignment compared to the SFT baseline, on both datasets. Moreover, this distribution shift is more pronounced in DPL-O and DPL-N compared to baselines, with DPL-O having a relative advantage over DPL-N. Overall, when compared to results in Table 3 and Table 2, this suggests that DPL's reward formulation with its focus on learning granular preferences tends to retrieve a more accurate estimate of the true reward distribution, since the reward model here is agnostic to individual baseline objectives.

## 7 Conclusion

In this work, we present **Diverse Preference Learning (DPL)**, a reference-free preference alignment framework that allows policies to explicitly model the diversity of preference annotations in popular offline datasets, that current methods often ignore. Apart from being lightweight, DPL's objective function is specifically designed to be consistent with a baseline desirability in completions while simultaneously learning from the relative strength of preferences, albeit with some additional relabeling of the preference data. Additionally, DPL overcomes the Bradley-Terry (BT) model's over-reliance on implicit reward differences while being sensitive to "weak" preferences, thereby avoiding some of the underfitting issues and policy degeneracies that naive BT-formulations can incur. This leads to consistent high-performance on key benchmarks for targeted tasks, including Ultrafeedback and Reddit TL;DR (tested on CNN/Daily Mail)--through controlled evaluations conducted with both an LLM-as-a-judge as well as a traditional Reward Model (RM). Furthermore, across various preference desiderata including honestly, helpfulness, truthfulness, and safety as well as on OOD-settings in Alpaca Eval 2.0, DPL consistently outperforms traditional baselines, including human-utility-based approaches like KTO that is additionally supervised with a reference model. This is shown in both head-to-head evaluations as well as in reference to a much stronger baseline like GPT-4-Turbo.

## 8 Future Work

Our findings open up a number of items of future work. Avenues include exploring why the odds-ratio-based variant (DPL-O) outperforms the length-normalized version, except in areas like safety where DPL-N is relatively higher performing. Analyzing these performance differentials outside of length-controlled sampling settings could be another direction that can make the choice between the two clearer, motivating when to use one vs. the other. Additionally, with a more liberal compute budget, one could more exhaustively explore DPL's importance weights in its objective function to find a more optimal set of weights. Moreover, since reference model likelihoods can be precomputed Rafailov et al. (2024) for scaling especially with consistent engineering support in the TRL community12--an interesting dimension could be evaluating the trade-offs in performance gain with DPL while controlling for model scale.

Footnote 12: [https://huggingface.co/docs/trl/en/index](https://huggingface.co/docs/trl/en/index)

Figure 2: OPT 1.3B-assigned expected rewards distribution on the evaluation set of Ultrafeedback (left) and TL;DR (right). The distribution is taken over un-normalized raw logits, which explains the slight left-shift of the distributional mean from zero, particularly on Ultrafeedback.

 



### Limitations

One limitation of the current work is that considering our research budget, we could not scale our experiments to larger models especially in the parameter size ranges of \(\sim\)8B-13B. While DPL's practical applicability is not limited by model-scale, its consistency in performance with full-parameter training would likely generalize to parameter-efficient settings, which could an interesting future avenue. Moreover, DPL does not align policies for auxiliary objectives like "honesty" or language-specific objectives outside of English since our intention was to optimize on the current lot of preference datasets with a more general approach. However, since both GPT-4o's (our primary automatic evaluator) and GPT-4-Turbo (the baseline in AlpacaEval comparisons) are black-box models, there could still be inherent bias in its ratings even after controlling for self-rewarding Yuan et al. (2024). Furthermore, as with all LLMs pretrained on large amounts of internet-scale raw data in an unsupervised manner, our baselines, including DPL models, are likely prone to generating outputs that reflect inherent risks and biases, even at the post-training stage of alignment. These may include reinforcing harmful stereotypes, spreading misinformation, or replicating the societal and cultural biases present in the original training data. As such, although we did not conduct any specific red-teaming efforts to trace out such issues, we believe efforts in preference alignment like ours will be a crucial step towards resolving such limitations in the current batch of models.

## Acknowledgments

This research was conducted by the first author as part of his internship at Optum AI during the summer of 2024, under the guidance of the second author. We sincerely thank Optum AI for their invaluable support, providing both the resources and environment necessary for this work. This research was also supported in part by the U.S. Defense Advanced Research Projects Agency (DARPA) Friction for Accountability in Conversational Transactions (FACT) program under Other Transaction award HR00112490377 to Colorado State University. The opinions expressed are those of the authors and do not represent views of the Department of Defense or the U.S. Government. We would also like to thank the anonymous reviewers whose invaluable feedback helped improve the final copy of this paper.

## References

* Abdin et al. [2024] Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Hakrirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sebastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhry, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matthew Dixon, Ronnel Eldan, Victor Fragoso, Jinfeng Gao, Mei Gao, Min Gao, Amit Garg, Allie Del Giorno, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Wenxiang Hu, Jamie Huynh, Dan Ifer, Sam Ade Jacobs, Mojan Javaheripi, Xin Jin, Nikos Karampatziakis, Piero Kauffmann, Mahoud Khademi, Dongwoo Kim, Young Jin Kim, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Yunsheng Li, Chen Liang, Lars Liden, Xihui Lin, Zeqi Lin, Ce Liu, Liyuan Liu, Mengchen Liu, Weisung Liu, Xiaodong Liu, Chong Luo, Piyush Madan, Ali Mahmoudzadeh, David Majercak, Matt Mazzola, Caio Cesar Teodoro Mendes, Arindam Mitra, Hadrik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzan, Heyang Qin, Marko Radmiliac, Liliang Ren, Gustavo de Rosa, Corby Rosset, Sambudha Roy, Olatuji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hitesh Sharma, Yelong Shen, Swadhen Shukla, Xia Song, Masahiro Tanaka, Andrea Tupini, Praneethna Vaddamanu, Chunyu Wang, Guanhua Wang, Lijuan Wang, Shuohang Wang, Xin Wang, Yu Wang, Rachel Ward, Wen Wen Wen, Philipp Witte, Haiping Wu, Xiaoxia Wu, Michael Wyatt, Bin Xiao, Can Xu, Jiahang Xu, Weijian Xu, Jihong Xue, Sonalid Yadav, Fan Yang, Jiawei Yang, Yifan Yang, Ziyi Yang, Jonghan Yu, Lu Yuan, Chernuldong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou. 2024. Phi-3 technical report: A highly capable language model locally on your phone.
* AllenAI [2024] AllenAI. 2024. Ultrafeedback binarized clean.
* Anthony et al. [2017] Thomas Anthony, Zheng Tian, and David Barber. 2017. Thinking fast and slow with deep learning and tree search. _Advances in neural information processing systems_, 30.
* Azar et al. [2024] Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. 2024. A general theoretical paradigm to understand learning from human preferences. In _International Conference on Artificial Intelligence and Statistics_, pages 4447-4455. PMLR. Bai et al. [2022] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova Das Sarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. _arXiv preprint arXiv:2204.05862_

Quentin Bertrand, Wojciech Marian Czarnecki, and Gauthier Gidel. 2023. On the limitations of the elo, real-world games are transitive, not additive. In _International Conference on Artificial Intelligence and Statistics_, pages 2905-2921. PMLR.
* Bradley and Terry (1952) Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. _Biometrika_, 39(3/4):324-345.
* Chen et al. (2024a) Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, and Hongxia Jin. 2024a. AlpagGauss: Training a better Alpaca with fewer data. In _ICLR_.
* Chen et al. (2024b) Lichang Chen, Chen Zhu, Davit Soselia, Jiuhai Chen, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. 2024b. ODIN: Disentangled reward mitigates hacking in RLHF. _arXiv preprint arXiv:2402.07319_.
* Christiano et al. (2017) Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. _Advances in neural information processing systems_, 30.
* Conover et al. (2023) Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. Free dolly: Introducing the world's first truly open instructioned LLM.
* Cui et al. (2024) Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. 2024. UltraFeedback: Boosting language models with high-quality feedback. In _ICML_.
* Dao et al. (2022) Tri Dao, Dan Fu, Stefano Ermon, Attri Rudra, and Christopher Re. 2022. Flashattention: Fast and memory-efficient exact attention with io-awareness. _Advances in Neural Information Processing Systems_, 35:16344-16359.
* Dettmers et al. (2024) Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2024. Qlora: Efficient finetuning of quantized llms. _Advances in Neural Information Processing Systems_, 36.
* Ding et al. (2023) Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. Enhancing chat language models by scaling high-quality instructional conversations. In _EMNLP_.
* Dong et al. (2024) Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. 2024. Rlfw workflow: From reward modeling to online rlhf. _arXiv preprint arXiv:2405.07863_.
* Dubey et al. (2024) Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. _arXiv preprint arXiv:2407.21783_.
* Emerson (2013) Peter Emerson. 2013. The original borda count and partial voting. _Social Choice and Welfare_, 40(2):353-358.
* Ethayanarajh et al. (2024) Kawin Ethayanarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. 2024. KTO: Model alignment as prospect theoretic optimization. _ArXiv_, abs/2402.01306.
* Fisch et al. (2024) Adam Fish, Jacob Eisenstein, Vicky Zayats, Alekh Agarwal, Ahmad Beirami, Chirag Nagpal, Pete Shaw, and Jonathan Berant. 2024. Robust preference optimization through reward model distillation.
* Gao et al. (2023) Leo Gao, John Schulman, and Jacob Hilton. 2023. Scaling laws for reward model overoptimization. In _International Conference on Machine Learning_, pages 10835-10866. PMLR.
* Geng et al. (2023) Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song. 2023. Koala: A dialogue model for academic research. _Blog post, April_, 1:6.
* Havvilla et al. (2024) Alex Havvilla, Sharath Raparthy, Christoforus Nalmampantis, Jane Dwivedi-Yu, Makysym Zhuravinskyi, Eric Hambro, and Roberta Railneau. 2024. GLoRe: When, where, and how to improve LLM reasoning via global and local refinements. _arXiv preprint arXiv:2402.10963_.
* Hessel et al. (2023) Jack Hessel, Ana Marasovic, Jena D. Hwang, Lillian Lee, Jeff Da, Rowan Zellers, Robert Mankoff, and Yejin Choi. 2023. Do androids laugh at electric sheep? humor "understanding" benchmarks from the new yorkor caption contest. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 688-714, Toronto, Canada. Association for Computational Linguistics.
* Hong et al. (2024) Jiwoo Hong, Noah Lee, and James Thorne. 2024. Orpo: Monolithic preference optimization without reference model. _arXiv preprint arXiv:2403.07691_.
* Kirk et al. (2024) Hannah Rose Kirk, Alexander Whitefield, Paul Rotter, Andrew Bean, Katerina Margatina, Juan Ciro, Rafael Mosquera, Max Bartolo, Adina Williams, He He, et al. 2024. The prism alignment project: What participatory, representative and individualised human feedback reveals about the subjective and multicultural alignment of large language models. _arXiv preprint arXiv:2404.16019_.
* Kopf et al. (2023) Andreas Kopf, Yannic Kilcher, Dimitri von Rutte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Minh Nguyen, Oliver Stanley, Richard Nagyfi, et al. 2023. Openassistant conversations-democratizing large language model alignment. In _Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track_.
* Lambert et al. (2020) Nathan Lambert, Valentina Pyatkin, Jacob Daniel Morrison, Lester James Validad Miranda, Bill Yuchen Lin, Khyathi Raghavi Chandu, Nouha Dziri, Sachin 

Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hanna Hajishirzi. 2024. RewardBench: Evaluating reward models for language modeling. _ArXiv_, abs/2403.13787.
* Li et al. (2023) Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models.
* Lightman et al. (2023) Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let's verify step by step. _arXiv preprint arXiv:2305.20050_.
* Loshchilov et al. (2017) Ilya Loshchilov, Frank Hutter, et al. 2017. Fixing weight decay regularization in adam. _arXiv preprint arXiv:1711.05101_, 5.
* Luo et al. (2023) Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. 2023. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. _arXiv preprint arXiv:2308.09583_.
* Meng et al. (2024) Yu Meng, Mengzhou Xia, and Danqi Chen. 2024. Simpo: Simple preference optimization with a reference-free reward. _arXiv preprint arXiv:2405.14734_.
* Nath et al. (2024) Abhijnan Nath, Changsoo Jung, Ethan Seefried, and Nikhil Krishnaswamy. 2024. Simultaneous reward distillation and preference learning: Get you a lanthage model who can do both.
* Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. 2022. Training language models to follow instructions with human feedback. In _NeurIPS_.
* Pal et al. (2024) Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. 2024. Smaug: Fixing failure modes of preference optimisation with dpo-positive.
* Rafailov et al. (2024) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. _Advances in Neural Information Processing Systems_, 36.
* Rasley et al. (2020) Jeff Rasley, Samyam Rajbhandari, Olatuni Ruwase, and Yuxiong He. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 3505-3506.
* Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. _arXiv preprint arXiv:1707.06347_.
* Stiennon et al. (2020) Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. _Advances in Neural Information Processing Systems_, 33:3008-3021.
* Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Alpaca: A strong, replicable instruction-following model. _Standf Center for Research on Foundation Models. https://crfin. stanford. edu/2023/03/13/alpaca. html_, 3(6):7.
* Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almhairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Tversky (1969) Amos Tversky. 1969. Intransitivity of preferences. _Psychological review_, 76(1):31.
* Voliske et al. (2017) Michael Voliske, Martin Potthast, Shahbaz Syed, and Benno Stein. 2017. Tl; dr: Mining reddit to learn automatic summarization. In _Proceedings of the Workshop on New Frontiers in Summarization_, pages 59-63.
* Wang et al. (2024) Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishnuang Huang, Wei Shen, Senje Jin, Enyu Zhou, Chenyu Shi, Songyang Gao, Nuo Xu, Yuhao Zhou, Xiaoran Fan, Zhiheng Xi, Jun Zhao, Xiao Wang, Tao Ji, Hang Yan, Liking Shen, Zhan Chen, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, and Yu-Gang Jiang. 2024a. Secrets of rlhf in large language models part ii: Reward modeling.
* Wang et al. (2024b) Guan Wang, Siijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. 2024b. OpenChat: Advancing open-source language models with mixed-quality data. In _ICLR_.
* Welleck et al. (2020) Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2020. Neural text generation with unlikelihood training. In _International Conference on Learning Representations_.
* Wiseman et al. (2015) Sam Wiseman, Alexander M. Rush, Stuart Shieber, and Jason Weston. 2015. Learning anaphoricity and antecedent ranking features for coreference resolution. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 1416-1426, Beijing, China. Association for Computational Linguistics.

 

Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. 2024. LESS: Selecting influential data for targeted instruction tuning. In _ICML_.
* Xu et al. (2024) Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. 2024. Contrastive preference optimization: Pushing the boundaries of LLM performance in machine translation. _ArXiv_abs/2401.08417.
* Yang et al. (2024) Rui Yang, Ruomeng Ding, Yong Lin, Huan Zhang, and Tong Zhang. 2024. Regularizing hidden states enables learning generalizable reward model for llms.
* Yuan et al. (2024) Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024. Self-rewarding language models.
* Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. _arXiv preprint arXiv:2205.01068_.
* Zhou et al. (2023) Chunting Zhou, Pengfei Liu, Purxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xueze Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2023. LIMA: Less is more for alignment. _NeurIPS_.
* Zhou et al. (2024) Zhenhong Zhou, Haiyang Yu, Xinghua Zhang, Rongwu Xu, Fei Huang, and Yongbin Li. 2024. How alignment and jailbreak work: Explain llm safety through intermediate hidden states.
* Ziebart et al. (2008) Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Day, et al. 2008. Maximum entropy inverse reinforcement learning. In _Aaaai_, volume 8, pages 1433-1438. Chicago, IL, USA.
* Ziegler et al. (2019) Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. _arXiv preprint arXiv:1909.08593_.

## Appendix A Underfitting of \(r^{*}_{\text{norm}(x,y)}\)

Let us first rewrite the reward difference formulation in \(r^{*}_{\text{norm}(x,y)}\) in terms of the true preference probability of \(y_{w}\) being preferred over \(y_{l}\), as given below:

\[p^{*}(y_{w}\succ y_{l}\mid x) =\sigma\left(\beta\frac{\log\pi_{\theta}(y_{w}\mid x)}{|y_{w}|}- \beta\frac{\log\pi_{\theta}(y_{l}\mid x)}{|y_{l}|}\right)\] \[=\sigma\left(\beta\log\frac{\pi_{\theta}(y_{w}\mid x)/|y_{w}|}{ \pi_{\theta}(y_{l}\mid x)/|y_{w}|}\right)\] \[=\sigma\left(\beta\log\frac{\pi_{\theta}(y_{w}\mid x)|y_{l}|}{ \pi_{\theta}(y_{l}\mid x)|y_{w}|}\right) \tag{10}\]

Now, as argued in Section 3, if the preference probability \(p^{*}(y_{w}\succ y_{l})=1\) in the LHS i.e., \(y_{w}\) is consistently favored over \(y_{l}\), the BT-assigned reward difference must approach \(+\infty\) to satisfy Eq. 10. In other words, \(\beta\log\frac{\pi_{\theta}(y_{w}|x||y_{l}|}{\pi_{\theta}(y_{l}|x)|y_{w}|}\to\infty\), regardless of the KL-parameter \(\beta\). Since \(|y_{l}|\) represents the token-length of the losing response and cannot be zero13, one can clearly see that the log-probability ratio is \(\log(\frac{\pi_{\theta}(y_{w})}{\pi_{\theta}(y_{l})})\rightarrow\infty\), which implies \(\pi_{\theta}(y_{l})\to 0\). Under this condition, our arguments in Section 3 for \(r^{*}_{\text{odds}(x,y)}\) similarly hold for \(r^{*}_{\text{norm}(x,y)}\).

Footnote 13: This assumption is justified since \(y_{l}\) is not generated during offline alignment and we can assume any \(y_{l}\) to have a non-zero length in \(D_{\text{pref}}\).

## Appendix B Additional Motivation for DPL Objective

As pointed out in Section 4 regarding the specific motivations behind the DPL framework, we further note that DPL is flexible and is easily extensible to more specialized domains (outside of summarization on articles or general instruction-following)--where users might want to _additionally_ carry out an SFT-stage prior to alignment to bring the policy within the domain's distribution. Since alignment here is typically done with in-distribution binary pairs sampled from the SFT model (with any rank order between them after ratings) Rafailov et al. (2024); Meng et al. (2024), our DPL objective can easily operate on such samples, learning from the preference diversity within those in-domain pairs. Intuitively, even in this case, DPL would likely provide more sample-efficiency than KTO since DPL tends to learn from "equally-preferred" (weak-preference) samples too whereas KTO's requirement of flattening of individual samples into desirable and non-desirable might not be ideal when the preference data contains such weakly-preferred pairs. Additionally, the inclusion of the BT-based constrastive term in the three-part loss function allows DPL to avoid some of the repetition-related issues previously observed in Rafailov et al. (2024) especially with the unlikelihood term when used in parallel with the SFT term, as our empirical experiments and evaluation suggest.

## Appendix C Reddit TL;DR Labeler Confidence Distribution

Figure 3 shows the distribution of human confidence labels on the Reddit TL;DR summarization dataset. Without explicit pointwise reward estimates, we utilize these expert-annotated confidence



scores as proxy for baseline desirability in addition to using edit-distances between preference pairs for determining the relative preference strength as mentioned in Section 4. For details on thresholds, see our experiments section (Section 5).

## Appendix D Training Hyperparameters

Learning Rate and Batch SizesUsing DeepSpeed ZeRO-2 Rasley et al. (2020), we exclusively employ full-parameter training (3.8 billion-parameters) for all baselines with the Phi-3-Mini-128k-Instruct model. All DPL models including baselines such as SFT, ORPO and SIMPO except KTO were trained for 2,000 steps with an effective batch size of 64 (with gradient accumulation \(\sim\)4). For KTO, which is computationally more demanding due to the baseline-KL computation, we use an effective batch size of 16 and train for 1 epoch on the flattened preference data, as suggested in Ethayarajh et al. (2024). For all models, we use AdamW Loshchilov et al. (2017); Dettmers et al. (2024) as the optimizer. For the SFT model, we use a learning rate (LR) of \(1e-5\) whereas for alignment (ORPO, SIMPO and all DPL variants), we use an LR of of \(5e-6\). For KTO, we use a slightly lower LR of \(5e-7\) following default settings in the TRL library. All baselines are trained with a linear warm-up with cosine decay. For the OPT-1.3B Reward Model (RM), we train it for 1 epoch as suggested in Lambert et al. (2024) with a larger learning rate of \(1e-5\) with an effective batch size of 64.

\(\beta\) values and GPU-usageFor consistency with previous work, we use a \(\beta=0.1\) for ORPO (\(\lambda\) in the original paper) and KTO as suggested in Ethayarajh et al. (2024) while for SIMPO we use \(\beta=2.0\) with a gamma_beta_ratio of 0.25. For all DPL variants, we use a consistent \(\beta=1\) in \(r^{*}_{\text{odds}}\) and \(r^{*}_{\text{norm}}\) formulations (Eq. 2 and Eq. 3). Due to compute constraints of having only 1 A100 GPU, we filter prompt and response pairs posttokenization to be less than 1,024 tokens, for both Ultrafeedback and Reddit TL;DR datasets. Additionally, one training run took \(\sim\)6 GPU hours for 2,000 steps within this compute budget and hyperparameter settings. For efficiency, we use Flash-Attention 2 Dao et al. (2022) for training runs.

## Appendix E Minimal Implementation of DPL-O Objective

For reproducibility, we provide a minimal implementation of our DPL objective (Eq. 6) with the log-odds-based implicit rewards formulation. DPL-O is relatively straightforward to implement assuming importance weights are precomputed according to our DPL preference categorization described in Section 4 and examples shown in Table 1. A minimal implementation of the DPL-O objective in Pytorch is given Figure 4.

## Appendix F Tie-count Distribution

Figure 5 shows the distribution of total number of ties for each preference-alignment dimension that was used to evaluate all baselines (sampled generations) using the GPT-4o model in the FPP-style evaluation, on Reddit TL;DR (CNN/Daily Mail) and Ultrafeedback test sets. Ties were broken with a variant of the Borda-count Emerson (2013) to compute the final win-rates (total wins/total samples). For example, in cases where three top-scoring candidates were each rated 4 (out of 5) while others received lower scores, we add \(1/3\) to each tied candidate's current win count. See Table 2 for the complete tie-broken results.

## Appendix G Inference Prompts for GPT-4o as a Judge

Figure 6 shows the evaluation prompt used for one-shot ratings by GPT-4o on the Ultrafeedback AllenAI (2024) evaluation set on various instruction-following dimensions like honesty, truthfulness, helpfulness, safety as well as the overall quality of the sampled responses. We slightly modify Cui

Figure 3: Distribution of human (expert) confidence labels on the preference annotations in Reddit TL;DR dataset.

 

et al. (2024)'s completion quality evaluation template by removing the _verbalized calibration_ criteria since no baseline was trained to output such confidence scores, following previous works (Rafailov et al., 2024; Hong et al., 2024; Xu et al., 2024). However, since safe completions are a crucial dimension in alignment (Zhou et al., 2024), we add another dimension of _saferty_ to additionally evaluate how safe the sampled completions are, in both instruction-following as well as summarization. Moreover, we include the "chosen" responses from Ultrafeedback as the preferred completion as an additional reference point for GPT-4o to base its ratings. For evaluating summarization from feedback on Reddit Tl;DR (CNN/Daily Mail), we keep this formatting consistent except minor tweaking to replace references to "instruction-following" to "summarization" and keep the human-written summaries as the preferred response for reference. Following Lambert et al. (2024), for evaluation on both tasks, we randomly swap placements of the six-responses to avoid any possible positional bias.

Note that we evaluate all 6 baseline candidates simultaneously for each test prompt sample, similar to Hessel et al. (2023)'s zero-shot14 automatic evaluation on preference in humor-quality. This simultaneous evaluation helps reduce a large number

Figure 4: DPL-O Objective (loss) with a minimal Pytorch implementation



of GPT-4o API calls that explicit pairwise evaluations for each pair in our baselines would have required, while still keeping our wide variety of baselines in the evaluation. We believe this is a reasonable trade-off since we exhaustively evaluate both DPL variants as well as their ablated versions on all possible pairwise combinations with the open-source OPT 1.3B reward model, that does not require costly API calls.

## Appendix H Inference Prompts for GPT-4o on AlpacaEval 2.0 with GPT-4-Turbo as baseline

Figure 7 shows the evaluation prompt15 used for GPT-4o as a Judge evaluation (Table 2) on the 805 instruction-following questions from AlpacaEval 2.0 Li et al. (2023) with the GPT-4-Turbo completions as the baseline generations for pairwise comparisons.

Footnote 15: We use the official prompt provided in: [https://github.com/tatsu-lab/alpaca_eval/blob/main/src/alpaca_eval/evalators_configs/chatgpt/basic_prompt.txt](https://github.com/tatsu-lab/alpaca_eval/blob/main/src/alpaca_eval/evalators_configs/chatgpt/basic_prompt.txt)

## Appendix I Sample Generations

We present sampled completions where DPL-O (our best-performing model) was rated higher than KTO (best-performing baseline) using GPT-4o-based evaluations. A temperature of 1 was used for evaluation (see Table 2), and 0.7 for sampling. Examples include Ultrafeedback completions (Table 5 and Table 6) and Reddit TL;DR (Table 7 and Table 8) test set prompts.

 

You are a careful annotator. Rate the following assistant responses based on the following guidelines *Honestyv: The assistant should be honest about whether it knows the answer and express its uncertainty explicitly. Use weakeners such as 'I guess', 'I suppose', 'probably', and 'perhaps' to express uncertainty, and feel free to answer 'I dont know' if necessary.
*Truthfulness**: The assistant should answer truthfully and be faithful to factual knowledge as well as given contexts , never making up any new facts that aren true or cannot be grounded in the instruction. *Helpfulness**: The assistant should provide accurate, relevant, and up-to-date information. *Safety*: The assistant should maintain positive and non-harmful content and avoid offensive or inappropriate topics.

* (U) Heets goals or restrictions, neglecting others.
- (2) Acknowledges both but slight deviations. 4. **Almost There**: Near alignment, minor deviations. 5. **Comprehensive Compliance**** Fully aligns, meets all requirements. Now, evaluate the following assistant responses (A, B, C, D, E and F) in comparison with the preferred response. Provide scores according to instructions given above along with an overall score for each. Instruction: {} Preferred Response: {} Assistant Response A: {} Assistant Response B: {} Assistant Response C: {} Assistant Response D: {} Assistant Response E: {} Assistant Response F: {} Reply in JSON format: {  "A": {  "overall": <your_score>,  "honesty": <your_score>,  "helpfulness": <your_score>,  "truthfulness": <your_score>,  "safety": <your_score> },  "B": {...},  "C": {...},  "D": {...},  "E": {...},  "F": {...} } ```

Figure 6: Evaluation prompt used for GPT-4o ratings on Ultrafeedback [], assessing honesty, truthfulness, helpfulness, safety, and overall quality. We slightly modify []’s format by removing _verbalized calibration_ and adding _safery_. We use a consistent formatting for Reddit TL;DR, with summaries as reference responses. Responses were shuffled to avoid positional bias []. Default temperature of 1 was used for the evaluation.

 

prompt = ( "
systemn"  "You are helpful instruction-following assistant.\n"  "
u"  "
user\n"  "Select the output (a) or (b) that best matches the given instruction. "  "Choose your preferred output, which can be subjective. Your answer should ONLY contain: Output ( a) or Output ( b). Here's an example:\n n"  "# Example:\n"  "## Instruction:\n"  "Give a description of the following job: "ophthalologist"\n n n"  "## Output ( a):\n"  "An ophthalmologist is a medical doctor who specializes in the diagnosis and treatment of eye diseases and conditions.\n n n"  "## Output ( b):\n"  "An ophthalmologist is a medical doctor who pokes and prods at your eyes while asking you to read letters from a chart.\n n n"  "## which is best, Output (a) or Output (b)?n"  "Output ( a)\n n"  "Here the answer is Output (a) because it provides a comprehensive and accurate description of the job of an ophthalmologist. "  "In contrast, output (b) is more of a joke.\n n"  "R Task:\n"  "Now is the real task, do not explain your answer, just say Output (a) or Output (b).\n n n"  ""## Instruction:\n"  "{instruction}\n n"  "#Output ( a):\n"  "{output_1}nn"  "## Output ( b):\n"  "{output_2}n n n"  "## Which is best, Output (a) or Output (b)?n"  "
"  )

Figure 7: Evaluation prompt used for GPT-4o ratings on the 805 instruction-following questions from AlpacaEval 2.0 [Li et al., 2023] with the GPT-4-Turbo completions as the baseline generations for pairwise comparisons. Default temperature of 1 was used for the evaluation.

\begin{table}
\begin{tabular}{l p{196.7pt}} \hline \hline

- it is not possible to tell
- no Step by step answer: \\ \hline
**Human Written** & No, it does not mean that ”A woman is painting flowers.” Explanation: The statement ”A woman is drawing a caricature of two people” implies that the woman is creating a humorous or exaggerated representation of two individuals. It does not suggest that she is painting flowers. Confidence: 90 \\ \hline
**DPL-O** & Step by step: 1. Analyze the first statement: a woman is drawing a caricature of two people. 2. Analyze the second statement: a woman is painting flowers. 3. Compare the activities described in both statements: drawing a caricature is different from painting flowers. 4. Determine if one statement implies the other: there is no implication between drawing a caricature and painting flowers. 5. Make a conclusion based on the comparison: the activities described in the two statements are unrelated. 6. Choose the correct answer based on the conclusion: no, the second statement does not follow from the first. \\ \hline
**KTO** & To determine if the statement ”A woman is drawing a caricature of two people.” implies that ”A woman is painting flowers,” we need to analyze the information provided. 1. The first statement tells us that a woman is drawing a caricature of two people. 2. The second statement claims that a woman is painting flowers. Drawing a caricature and painting flowers are two different activities. A caricature involves creating a humorous or exaggerated portrayal of a person or people, while painting flowers typically involves creating a realistic or artistic representation of flowers. Since the two statements describe different activities, we cannot conclude that if a woman is drawing a caricature of two people, she is also painting flowers. Answer: no \\ \hline \hline \end{tabular}
\end{table}
Table 5: Example sampled completion (I) with temperature of 0.7 on the Ultrafeedback evaluation set comparing the DPL-O model and KTO baseline, with the specific instruction and chosen-completion shown for reference. While DPL-O specifically follows the instruction to provide a “step-by-step” answer, KTO tends to not explicitly specify steps. Both generated completions are longer in token-length than the “chosen” counterpart.

 

* [150] M. C. Delfon, A. F. Krizhevsky, and R. L. Krause, "A new approach to the identification of a single-objective genetic algorithm for genetic algorithms," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 2398-2406, 2007.
* [151] J. P. Berman, S. G. H. Lee, and T. W. K. Ng, "Genetic algorithms for genetic algorithms: The role of genetic algorithms," _Journal of Genetic Programming_**10**, 1-12, 1994.
* [152] E. B. Krizhevsky, "The complexity of genetic algorithms," _IEEE Transactions on Evolutionary Computation_**10**, 1-10, 1994.
* [153] V. N. Kolmogorov and I. A. Rubin, "An introduction to statistical theory," _Annals of Mathematical Statistics_**1**, 1-17, 1975.

[MISSING_PAGE_POST]

pplied Probability_**1**, 1-1, 



\begin{table}
\begin{tabular}{l p{284.5pt}}
**Prompt** & Summarize this news article clearly and concisely. Use TL;DR to your response: TITLE: Premier League marathon men: George Boyd pipes Christian Eriksen and Jordan Henderson in distance covered after running 210.5 miles for Burnley this season ARTICLE: George Boyd is the Barclays Premier League’s hardest working player, clocking up 210.5 miles on the pitch this season—the equivalent of running from Burnley’s Turf Moor ground to Crystal Palace’s Selhurst Park in south London. The Burnley winger has run slightly further than Tottenham Hottspur mrifielder Christian Eriksen. Walking the journey from Turf Moor to Selhurst would usually take 69 hours, according to Google Maps’ estimations, but Boyd has managed it in less than 40 hours of playing time. Tottenham Hotspur midfielder Christian Eriksen is second in the list, 0.4 miles behind Boyd. Liverpool stand-in captain Jordan Henderson is also included in the Premier League’s top 10 runners. Sean Dyche’s side, sitting a point off safety, 18th in the Premier League, have three players in the top 10. Between them, the Burnley squad has run an incredible 2,172.3 miles this campaign. The data was recorded by EA Sports PPI, which is used as the official player rating index of the Premier League. \\ \hline
**Human Written** & George Boyd has run 210.5 miles on the pitch this season. Burnley winger has run slightly further than Tottenham’s Christian Eriksen. Jordan Henderson is also in the top 10 as Liverpool’s stand-in captain. \\ \hline
**DPL-O** & Burnley’s George Boyd has run the most distance in the Premier League this season, clocking up 210.5 miles. Tottenham’s Christian Eriksen is second, with Liverpool’s Jordan Henderson third. Burnley’s teammates Scott Arfield and David Jones also feature in the top 10. \\ \hline
**KTO** & Burnley’s George Boyd has run the most distance in the Premier League this season, clocking up 210.5 miles. Tottenham’s Christian Eriksen is second, while Liverpool’s Jordan Henderson is third. Burnley’s teammates Scott Arfield and David Jones also feature in the top 10. _The data was recorded by EA Sports PPI, which is used as the official player rating index of the Premier League._ \\ \end{tabular}
\end{table}
Table 7: Example completion (I) sampled with temperature of 0.7 on the CNN/Daily Mail evaluation set comparing the DPL-O model and KTO baseline, with the specific prompt and human-written summary shown for reference. DPL-O’s summary is more succinct while KTO tends to deviate from the essential details (shown in italics).



* [37] M. C. Delfon, A. G. K. R. Wilson, and J. S. W. York, "The New England Journal of Medicine: The American Statistical Association", vol. 1098, no. 2, pp. 45-60.

[MISSING_PAGE_POST]