Reward-Augmented Data Enhances Direct Preference Alignment of LLMs

## Abstract

Abstract Preference alignment in Large Language Models (LLMs) has significantly improved their ability to adhere to human instructions and intentions. However, existing direct alignment algorithms primarily focus on relative preferences and often overlook the qualitative aspects of responses, despite having access to preference data that includes reward scores from judge models during AI feedback. Striving to maximize the implicit reward gap between the chosen and the slightly inferior rejected responses can cause overfitting and unnecessary unlearning of the high-quality rejected responses. The unawareness of the reward scores also drives the LLM to indiscriminately favor the low-quality chosen responses and fail to generalize to responses with the highest rewards, which are sparse in data. To overcome these shortcomings, our study introduces reward-conditioned LLM policies that discern and learn from the entire spectrum of response quality within the dataset, helping extrapolate to more optimal regions. We propose an effective yet simple data relabeling method that conditions the preference pairs on quality scores to construct a reward-augmented dataset. This dataset is easily integrated with existing direct alignment algorithms and is applicable to any preference dataset. The experimental results across instruction-following benchmarks including AlpacaEval 2.0, MT-Bench, and Arena-Hard-Auto demonstrate that our approach consistently boosts the performance of DPO by a considerable margin across diverse models such as Zephyr, Mistral, Qwen2, Llama3.1, Gemma2, and SPPO. Additionally, on six academic benchmarks including GSM8K, GPQA, MUSR, TruthfulQA, BBH, and ARC, our method improves their average accuracy. When applying our method to on-policy data, the resulting DPO model outperforms various baselines and achieves state-of-the-art results on AlpacaEval 2.0. Through comprehensive ablation studies, we demonstrate that our method not only maximizes the utility of preference data but also mitigates the issue of unlearning, demonstrating its broad effectiveness beyond mere dataset expansion. Our code is available at https://github.com/shenao-zhang/reward-augmented-preference .

## 1 Introduction

Reinforcement Learning from Human Feedback (RLHF) has recently seen remarkable success in aligning Large Language Models (LLMs) to follow instructions with human intentions. In this approach, AI-generated feedback serves as a stand-in for human preferences, assessing and ranking responses to prompts to construct a preference dataset. This dataset is then utilized in preference optimization algorithms to fine-tune LLMs. Among them, direct preference alignment (Rafailov et al., ; Azar et al., ; Zhao et al., ; Ethayarajh et al., ) that bypasses the need for an explicit reward model has garnered interest for their simplicity and cost efficiency. However, these algorithms mainly concern relative preferences and often overlook the quality of responses and their gaps, leading to limitations in their effectiveness.

Specifically, direct alignment algorithms such as DPO (Rafailov et al., ) focus on maximizing the implicit reward difference between accepted and rejected responses. This approach can lead to overfitting, as high-quality but rejected responses are unnecessarily unlearned (Adler et al., ). Even worse, since the dataset provides only a sample estimate of true preferences, the rejected responses can actually be more aligned with human preferences than the accepted ones in expectation. Similarly, due to the unawareness of the responses’ qualities, direct alignment will also result in the indiscriminate learning of the chosen responses, even when they are of low quality. As a result, the directly aligned LLMs often struggle to differentiate between responses of varying quality and fail to generalize effectively to more optimal or the highest-reward responses that are sparse in the preference data, which is another limitation.

To address these issues, we propose learning reward-conditioned policies as a straightforward fix to the above issues. By optimizing the LLM to generate responses conditioning on their qualities, the model is allowed to discern and leverage patterns within responses of varied quality. As a result, learning from both chosen and rejected responses alleviates the issue of unnecessarily unlearning high-quality rejected responses; distinguishing between varying-quality chosen responses alleviates the issue of indiscriminately accepting low-quality ones. By identifying common patterns in responses of similar quality and distinguishing them from those of differing quality, the LLM becomes more adept at generalizing to more optimal responses that are sparse in data.

With this motivation, we introduce an effective yet simple data relabeling method to construct reward-augmented datasets. We define a goal-conditioned reward using an indicator function that compares the goal reward with the actual quality score, such as the reward value given by the judge model during AI feedback. This allows us to relabel each preference pair, generating two new pairs conditioned on the reward goals of both the chosen and rejected responses. The resulting augmented dataset, which contains these newly conditioned pairs, can enhance the performance of existing direct alignment algorithms. Our method can be applied to any preference dataset and followed by off-the-shelf direct alignment algorithms to boost their performance.

In experiments, we first apply our method on UltraFeedback (Cui et al., ) and perform DPO (Rafailov et al., ) on this reward-augmented preference dataset by fine-tuning on various models, including Zephyr-7B-$\beta$ (Tunstall et al., ), Mistral-7B-Instruct-v0.3 (Jiang et al., ), Qwen2-7B-Instruct (Yang et al., ), Llama-3.1-8B-Instruct (Dubey et al., ), Gemma-2-9B-It (Team et al., ), and SPPO (Wu et al., ). The results show that our method consistently boosts the performance of these models as well as their DPO models by a large margin on instruction-following benchmarks such as AlpacaEval 2.0 (Dubois et al., ), MT-Bench (Zheng et al., ), and Arena-Hard-Auto (Li et al., ). Our method also improves the average accuracy on a variety of academic benchmarks (GSM8K, GPQA, MUSR, TruthfulQA, BBH, and ARC). Moreover, our findings also demonstrate an improved utility of the preference data: a subsequent round of DPO using the reward-augmented data can still significantly enhance the model fine-tuned with DPO; relabeling the binarized preference dataset with the DPO implicit reward leads to further performance gains. Additional ablation studies also suggest that our method addresses the problem of unlearning and is superior not just due to the increased dataset size. When applied to on-policy data, our method enhances the DPO model, enabling it to surpass various baselines and achieve state-of-the-art performance on AlpacaEval 2.0. Our code is available at [https://github.com/shenao-zhang/reward-augmented-preference](https://github.com/shenao-zhang/reward-augmented-preference).

## 2 Background

Consider a language model $\pi\in\Delta_{\mathcal{Y}}^{\mathcal{X}}$ that takes the prompt $x\in\mathcal{X}$ as input and outputs the response $y\in\mathcal{Y}$, where $\mathcal{X}$ and $\mathcal{Y}$ are spaces of prompts and responses, respectively. Given the prompt $x\in\mathcal{X}$, a discrete probability distribution $\pi(\cdot\mid x)\in\Delta_{\mathcal{Y}}$ is generated, where $\Delta_{\mathcal{Y}}$ is the set of discrete distributions over $\mathcal{Y}$. We define the true human preference distribution as

$$ $\displaystyle p^{*}(y_{1}\succ y_{2}\mid x)$ $\displaystyle:=\mathbb{E}_{h}\bigl{[}\mathmybb{1}(h\text{ prefers }y_{1}\text{ over }y_{2}\text{ given }x)\bigr{]},$ $$

where $h$ denotes the human rater and the expectation is over $h$ to account for the randomness of the human raters’ choices. After pretraining and Supervised Fine-Tuning (SFT), Reinforcement Learning from Human or AI Feedback (Ouyang et al., ; Bai et al., ) is typically employed to enhance the ability of the language model to follow instructions with human preferences.

#### RL from AI Feedback (RLAIF).

The RLAIF framework involves two major steps: preference dataset construction with AI feedback and preference optimization. As a surrogate for human preference, AI feedback, including LLM-as-Judge (Zheng et al., ; Cui et al., ) and Reward-Model-as-Judge (Adler et al., ; Dong et al., ), can be used to rank responses and generate preference pairs. Specifically, consider the judge model $r(x,y):\mathcal{X}\times\mathcal{Y}\rightarrow\mathbb{R}$ that outputs a scalar reward value representing the quality of $y$ under $x$.
For each prompt $x\in\mathcal{X}$, two responses, $y_{1}$ and $y_{2}$, are independently sampled—either from the same reference model (Xiong et al., ; Wu et al., ) or several different models (Zhu et al., ; Zhang et al., ). Then $r(x,y_{1})$ and $r(x,y_{2})$ are evaluated to determine the preferred response $y_{w}=\mathop{\mathrm{argmax}}_{y\in\{y_{1},y_{2}\}}r(x,y)$ and dispreferred response $y_{l}=\mathop{\mathrm{argmin}}_{y\in\{y_{1},y_{2}\}}r(x,y)$. By sampling responses and ranking them for a set of $N$ prompts, we get a preference dataset: $\mathcal{D}_{N}=\{x^{i},y_{w}^{i},y_{l}^{i},r(x^{i},y_{w}^{i}),r(x^{i},y_{l}^{
i})\}_{i=1}^{N}$.

#### Direct Alignment from Preference.

The objective for the LLM $\pi\in\Delta_{\mathcal{Y}}^{\mathcal{X}}$ is to maximize the KL-regularized expected reward. Recent works (Azar et al., ; Zhao et al., ; Tunstall et al., ; Ethayarajh et al., ) proposed to align the LLM directly with the preference data by deriving the preference loss as a function of the LLM by the change of variables. Among them, the Direct Preference Optimization (DPO) (Rafailov et al., ) loss has the following form:

$$ $\displaystyle\mathcal{L}_{\text{DPO}}(\pi;\mathcal{D}_{N})=-\mathbb{E}_{(x,y_{ w},y_{l})\sim\mathcal{D}}\biggl{[}\log\sigma\biggl{(}\beta\log\frac{\pi(y_{w} \mid x)}{\pi_{\text{ref}}(y_{w}\mid x)}-\beta\log\frac{\pi(y_{l}\mid x)}{\pi_{ \text{ref}}(y_{l}\mid x)}\biggr{)}\biggr{]},$ $$

where $\beta$ is a hyperparameter corresponding to the KL divergence regularization, $\sigma(\cdot)$ is the logistic function, and $\pi_{\text{ref}}$ is some reference LLM policy, such as the SFT model.

## 3 Reward-Conditioning Addresses Limitations of Direct Preference Alignment

### 3.1 Limitations of Direct Alignment from Preference

We will first demonstrate the limitations of vanilla direct alignment over the preference data.

#### High-Quality Rejected Responses are Unnecessarily Unlearned.

The dataset $\mathcal{D}_{N}$ often contains preference pairs where the rejected response $y_{l}$ is only marginally worse than the chosen one $y_{w}$. Direct alignment algorithms, however, primarily focus on relative preferences and are unaware of the responses’ quality values and gaps. Striving to maximize the reparameterized reward gap between the chosen and rejected responses will risk overfitting and the unnecessary unlearning of high-quality responses, potentially diminishing the model’s performance by discarding valuable alternatives. Furthermore, in such a finite data regime where only a sample estimate of the true preference is accessible, it can be very possible that $p^{*}(y_{l}\succ y_{w}\mid x)>0.5$, i.e., $y_{l}$ is in fact more preferred than $y_{w}$ in expectation. This issue becomes even more pronounced when the preference data generated with the imperfect judge model is noisy.

We illustrate this limitation with the example in Table 2. For $\mathcal{D}_{N=1}$ that contains a single preference pair(^1^11For simplicity, we write $(x,y_{w},y_{l},r(x,y_{w}),r(x,y_{l}))\in\mathcal{D}_{N}$ as $y_{w}>y_{l}$.) with reward $r(x,y_{1})=9$ and $r(x,y_{2})=8$, the optimal policy learned from $\mathcal{D}_{N=1}$ is $\pi^{*}(y_{1}\mid x)=1$. This causes the model to avoid generating $y_{2}$, a response of nearly equivalent quality.

#### Low-Quality Chosen Responses are Indiscriminately Learned.

For a similar reason, direct alignment algorithms also indiscriminately reinforce the chosen responses. As illustrated in Table 2, when $\mathcal{D}_{N=2}$ contains two preference pairs, where one of the chosen responses, $y_{2}$, is of low quality, $\pi^{*}$ still indiscriminately generates $y_{2}$ with an arbitrary probability $0\leq a\leq 1$, i.e., $\pi^{*}(y_{2}\mid x)=a$.

#### Reward Sparsity.

Preference data often contains responses that, despite being preferred in pairwise comparisons, exhibit substantial variation in quality. As a result, the optimal responses—those associated with the highest reward values—are sparse in the dataset. Since direct alignment algorithms do not account for these reward values, the trained model struggles to differentiate between responses of varying quality and fails to generalize effectively to the sparse optimal responses.

**Table 1: High-quality rejected responses such as $y_{2}$ can be unnecessarily unlearned: $\pi^{*}(\cdot\mid x)$ deterministically generates $y_{1}$. Reward-conditioned policies learn both responses and are easier to generalize to $r=10$ with the extracted features from $r=8$ and $r=9$.**
| response | $y_{1}$ | $y_{2}$ |
| --- | --- | --- |
| $r(x,y)$ | 9 | 8 |
| $\mathcal{D}_{N=1}$ | $\{y_{1}>y_{2}\}$ |  |
| $\pi^{*}(y\mid x)$ | 1 | 0 |
| $\pi^{*}(y\mid x;r=9)$ | 1 | 0 |
| $\pi^{*}(y\mid x;r=8)$ | 0 | 1 |

### 3.2 Reward-Conditioned Policies Resolve the Limitations

A straightforward way to address the limitations of direct alignment algorithms—specifically, their inability to account for the quality of responses—is to optimize a reward-conditioned policy. In this approach, the LLM policy is trained to generate responses corresponding to different reward values, enabling it to become aware of and adapt to these reward distinctions. By doing so, the LLM not only learns the patterns associated with the preferred responses but also retains the valuable information from the rejected ones, preventing the unlearning of high-quality rejected responses. For example, in Table 2, reward-conditioned policies learn to generate both $y_{1}$ and $y_{2}$, instead of unlearning $y_{2}$. This reward-based conditioning also enhances the model’s ability to differentiate between responses of varying quality, even if both are preferred over a rejected alternative, as illustrated in Table 2. Besides, by extracting common patterns across responses with different quality levels, the LLM becomes more generalizable and is capable of generating the highest-quality responses (e.g., $r=10$), which are often sparse in the training preference data.

## 4 Method

With the above motivation, we propose a data relabeling method that constructs a reward-augmented dataset by conditioning the preference pairs on the reward values given by the judge model. Specifically, we define the goal-conditioned reward $r_{g}(x,y)=\mathmybb{1}\bigl{(}g=r(x,y)\bigr{)}$ as a function of the quality reward $r$. The objective of the reward-conditioned policy $\pi(y\mid x;g)$ is thus to minimize the absolute difference between the goal reward $g$ and the response reward $r(x,y)$, which is equivalent to maximizing the goal-conditioned reward $r_{g}(x,y)$, i.e.,

$$ $\displaystyle\min_{\pi}\mathbb{E}_{g,x\sim\mathcal{D}_{N},y\sim\pi(\cdot\mid x ;g)}\bigl{[}|r(x,y)-g|\bigr{]}=\max_{\pi}\mathbb{E}_{g,x\sim\mathcal{D},y\sim \pi(\cdot\mid x;g)}\bigl{[}r_{g}(x,y)\bigr{]}.$ (4.1) $$

To optimize the RHS of Equation (4.1), we first observe that under the new goal-conditioned reward metric $r_{g}$, for each preference pair ${x^{i},y_{w}^{i},r_{w}^{i},y_{l}^{i},r_{l}^{i}}$ in $\mathcal{D}_{N}$, we have

$$ $\displaystyle r_{g=r_{w}^{i}}(x,y_{w}^{i})=1,r_{g=r_{w}^{i}}(x,y_{l}^{i})=0,r_ {g=r_{l}^{i}}(x,y_{l}^{i})=1,r_{g=r_{l}^{i}}(x,y_{w}^{i})=1.$ $$

Thus, each pair can be relabeled to create two new preference pairs based on two distinct goals: when $g=r_{w}^{i}$, $y_{w}^{i}\succ y_{l}^{i}$; when $g=r_{l}^{i}$, $y_{l}^{i}\succ y_{w}^{i}$. Then any direct alignment algorithm can be applied to this new goal-conditioned preference dataset. Compared to fine-tuning on the original dataset $\mathcal{D}_{N}$, the model learns to capture not only desirable behaviors but also undesirable ones from the reward-augmented dataset. This approach helps identify patterns across high- and low-quality responses, enabling the LLMs to discern and learn from the entire spectrum of response quality and extrapolate to more optimal responses at inference time, by conditioning on higher reward goals.

Figure: Figure 1: Construction of the reward-augmented preference dataset.
Refer to caption: x1.png

We illustrate our method in Figure 1. For each preference pair with index $i$ in $\mathcal{D}_{N}$, two goals are defined, corresponding to the reward values of the chosen response $y_{w}^{i}$ and the rejected response $y_{l}^{i}$. Specifically, under the first goal $g=y_{w}^{i}$, the relabeled rewards are $r_{g}(x,y_{w}^{i})=1$ and $r_{g}(x,y_{l}^{i})=0$. The original ranking of responses remains the same, except that the LLM is preference optimized conditioned on $g=y_{w}^{i}$. Similarly, under the second goal $g=y_{l}^{i}$, the relabeled rewards are $r_{g}(x,y_{l}^{i})=1$ and $r_{g}(x,y_{w}^{i})=0$. Thus, the chosen and rejected responses are reversed as $y_{l}^{i}$ and $y_{w}^{i}$, respectively. By generating preference pairs conditioned on the goal reward for both the chosen and rejected responses, we obtain a reward-augmented dataset of size $2N$. Finally, this new dataset can be used with any direct alignment algorithm, such as DPO.

In this work, we implement the reward-conditioned policy $\pi(y\mid x;g)$ as the LLM with a system prompt (or a prefix before the user prompt $x$ if system prompts are not supported by the LLM) such as “generate responses of score $g$”. At inference time, the LLM is conditioned on the highest possible reward value, e.g., $g=10$, to generate the responses.

## 5 Related Work

#### Preference Dataset Construction.

In order for the LLMs to follow instructions and better align with human intents, it is common practice to build a preference dataset containing a set of prompts and a pair of responses for each prompt, whose qualities are ranked by humans (Ouyang et al., ) or judge models (Bai et al., ). A popular pipeline (Cui et al., ; Tunstall et al., ; Wang et al., ; Ivison et al., ; Zhu et al., ) for constructing offline (i.e., fixed) datasets involves sampling off-policy responses from various LLMs for each prompt in the hope to increase the response diversity. The preference data can also be generated online (Guo et al., ) or iteratively (Bai et al., ; Xu et al., ; Gulcehre et al., ; Hoang Tran, ; Xiong et al., ; Dong et al., ; Calandriello et al., ; Rosset et al., ) by sampling and ranking on-policy responses from the training LLM. Recent works (Zhang et al., ; Cen et al., ; Xie et al., ) have also proposed systematically exploring the responses online and actively eliciting the preference. The proposed method in this paper is orthogonal to the construction ways of the preference data and can be applied to any dataset created either off-policy or on-policy.

#### Preference Optimization.

Preference optimization methods generally follow two approaches. The first involves learning a point-wise reward model, such as the Bradley-Terry model, and using RL algorithms like PPO (Schulman et al., ; Zheng et al., ; Xu et al., ) or REINFORCE (Williams, ; Li et al., ; Ahmadian et al., ), to maximize the KL-regularized expected reward. The second approach is direct alignment (Rafailov et al., ; Azar et al., ; Zhao et al., ; Ethayarajh et al., ; Liu et al., ), which gets rid of a separate reward model that is computationally costly to train. In this work, we mainly focus on the limitations of direct alignment algorithms, particularly their unawareness of the quality aspects of responses. For PPO-style alignment algorithms that fit and maximize an explicit reward, preference data is only used to learn the reward model, and policy training is performed in an online manner, where responses are sampled from the LLM and their reward values directly play a role during the RL optimization. This avoids drawbacks inherent to direct alignment methods, as detailed in Section 3.1.

#### Conditional LLM Fine-Tuning.

Conditioning LLMs during training has proven effective for aligning responses with specific human objectives. SteerLM (Dong et al., ; Wang et al., ) extends SFT by conditioning the LLM on the multi-dimensional annotated attributes in data, such as humor and toxicity, in order to steer model responses with user customizability. Directional Preference Alignment (DPA) (Wang et al., ) proposed a variant of rejection sampling fine-tuning (Yuan et al., ; Dong et al., ) that conditions on the direction of the multi-objective reward, i.e., a user-dependent linear combination of the reward attributes (helpfulness and verbosity in their experiments), that represents diverse preference objectives. These methods aim to train a single LLM that can flexibly adjust to various user preference profiles. On the contrary, our method targets the limitations of direct alignment algorithms by introducing reward-augmented relabeling. This also differs from Conditioned-RLFT (Wang et al., ), which leverages the data source information by learning a class-conditioned policy with RL-free supervised learning. Reward-aware Preference Optimization (RPO), introduced in Nemotron-4 (Adler et al., ), attempts to approximate the reward gap using the implicit reward and is motivated to resolve the unlearning issues of DPO, which our work also addresses. However, we show that more limitations beyond unlearning can be simply fixed with reward-conditioned LLMs and propose an easy-to-implement data relabeling method that integrates seamlessly with any direct alignment algorithm. We compare with all the aforementioned methods in our experiments.

## 6 Experiments

### 6.1 Reward-Augmented Data Boosts DPO Performance

We begin by conducting experiments to demonstrate that applying the proposed method to fixed offline preference datasets leads to consistent performance improvements in DPO.

Figure: (a) AlpacaEval 2.0 results. Left: Length-Controlled (LC) win rates. Right: Win rates.
Refer to caption: x2.png

#### Setup.

We adopt the UltraFeedback (Cui et al., ) preference dataset containing reward values scored by GPT-4 (LLM-as-Judge) that is ranged between $1$ and $10$ for each of the preference pairs. Our method constructs reward-augmented data by conditioning on these judge values. We fine-tune on various open-weight LLMs, including Mistral-7B-Instruct-v0.3(^2^22https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3), Qwen2-7B-Instruct(^3^33https://huggingface.co/Qwen/Qwen2-7B-Instruct), Llama-3.1-8B-Instruct(^4^44https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct), Gemma-2-9B-It(^5^55https://huggingface.co/google/gemma-2-9b-it), and SPPO (fine-tuned from Gemma2-9B-It)(^6^66https://huggingface.co/UCLA-AGI/Gemma-2-9B-It-SPPO-Iter3). We use the DPO implementation in the Huggingface Alignment Handbook (). The hyperparameters and prompts that we use are listed in Appendix A.1.

#### Results.

We first report the performance of the trained models on instruction-following benchmarks that use LLM as a judge, including AlpacaEval 2.0 (Dubois et al., ), MT-Bench (Zheng et al., ), and Arena-Hard-Auto (Li et al., ). The results are shown in Figure 2.

Across all instruction-following benchmarks, we observe that LLMs fine-tuned with DPO on the proposed reward-augmented data consistently outperform both their base models and those fine-tuned using DPO on the original UltraFeedback dataset by a considerable margin. Notably, direct alignment with the original preference data can sometimes degrade the performance of base models on specific benchmarks, such as Arena-Hard-Auto, which involves more complex reasoning tasks. In contrast, alignment using the reward-augmented data consistently yields superior results not only due to the improved style format gained from performing DPO on UltraFeedback.

Besides, we also evaluate the models on academic multi-choice QA benchmarks, including GSM8K (Cobbe et al., ), GPQA (Rein et al., ), MUSR (Sprague et al., ), TruthfulQA (Lin et al., ), BBH (Suzgun et al., ), and ARC Challenge (Clark et al., ). To better reflect the capabilities of LLMs, we adopt various settings for these benchmarks, including zero-shot, few-shot, and few-shot Chain-of-Thought (CoT). The results are shown in Table 3.

It can be observed that performing DPO on the reward-augmented preference data leads to better average academic scores for most families of models compared to models fine-tuned on the original UltraFeedback dataset and the base models. Besides, we didn’t observe severe alignment tax phenomenons (Askell et al., ; Noukhovitch et al., ; Li et al., ) after DPO, and our method is able to improve the base models on most of the benchmarks.

**Table 3: Performance comparison between the LLMs after DPO on UltraFeedback, on reward-augmented UltraFeedback, and their base models on academic multi-choice QA benchmarks in standard zero-shot, few-shot, and CoT settings. Here, n-s refers to n-shot, the bold texts represent the best results in each family of models.**
| Model | GSM8K<br>(8-s CoT) | GPQA<br>(0-s) | MUSR<br>(0-s) | TruthfulQA<br>(0-s) | BBH<br>(3-s) | ARC<br>(25-s) | Average |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Mistral-7B-Instruct-v0.3 | 52.39 | 30.62 | 47.35 | 59.71 | 46.64 | 58.53 | 49.21 |
| +DPO (UltraFeedback) | 53.22 | 28.94 | 47.35 | 64.74 | 47.46 | 60.32 | 50.34 |
| +DPO (Reward-Augmented) | 51.86 | 28.02 | 46.56 | 65.90 | 46.36 | 61.60 | 50.05 |
| Qwen2-7B-Instruct | 78.24 | 32.80 | 44.58 | 57.31 | 55.20 | 53.75 | 53.65 |
| +DPO (UltraFeedback) | 78.17 | 32.80 | 44.31 | 58.91 | 54.49 | 53.75 | 53.74 |
| +DPO (Reward-Augmented) | 81.05 | 32.97 | 45.77 | 57.99 | 54.94 | 54.52 | 54.54 |
| Llama-3.1-8B-Instruct | 76.72 | 33.89 | 39.95 | 54.00 | 50.74 | 55.38 | 51.78 |
| +DPO (UltraFeedback) | 78.47 | 33.72 | 43.39 | 56.61 | 51.31 | 57.51 | 53.50 |
| +DPO (Reward-Augmented) | 78.77 | 32.55 | 43.52 | 63.32 | 51.57 | 56.48 | 54.37 |
| Gemma-2-9B-It | 81.35 | 36.33 | 46.03 | 60.15 | 59.42 | 64.85 | 58.02 |
| +DPO (UltraFeedback) | 83.32 | 34.14 | 46.56 | 65.12 | 59.78 | 66.41 | 59.22 |
| +DPO (Reward-Augmented) | 83.62 | 35.74 | 48.15 | 65.27 | 59.82 | 65.87 | 59.75 |
| SPPO | 79.83 | 35.91 | 44.97 | 62.56 | 59.61 | 63.74 | 57.77 |
| +DPO (UltraFeedback) | 81.73 | 33.64 | 45.50 | 65.72 | 59.16 | 66.89 | 58.77 |
| +DPO (Reward-Augmented) | 80.67 | 36.16 | 48.68 | 67.39 | 58.88 | 65.53 | 59.55 |

### 6.2 Ablation Studies

#### Our Method Improves the Utility of Preference Data.

We provide two pieces of evidence that our method can get more juice out of the preference data compared to directly applying DPO.

**Table 4: SPPO can be improved with DPO by performing reward augmentation on the same data.**
|  | LC WR | WR | MT | Arena |
| --- | --- | --- | --- | --- |
| SPPO | 55.60 | 49.61 | 8.40 | 47.6 |
| +DPO (UF) | 52.75 | 40.58 | 8.41 | 40.4 |
| +DPO (RA) | 60.97 | 66.41 | 8.73 | 49.0 |

Firstly, we evaluate SPPO (Wu et al., ) fine-tuned with DPO on UltraFeedback (UF). The results are shown in Table 4. Since the SPPO model is already trained on UltraFeedback from Gemma-2-9B-It, an additional round of DPO training with the same data significantly degrades its performance. In contrast, performing DPO on Reward-Augmented (RA) UltraFeedback results in substantial performance gains for SPPO, indicating that our method enhances the utility of the preference data.

The second evidence is that after DPO, the implicit reward can be used to relabel and augment the same preference data. Specifically, after training Qwen2-7B-Instruct with DPO on

**Table 5: A second round of DPO on the reward-augmented data, i.e., DPO (IRA), relabeled with the implicit reward from the DPO model at the first round, i.e., DPO (UF), significantly improves it. Our method helps get more juice out of the binarized (i.e., without judge model rewards) preference data.**
|  | LC WR | WR | MT | Arena |
| --- | --- | --- | --- | --- |
| Qwen2-7B-It | 20.93 | 18.22 | 7.90 | 24.3 |
| +DPO (UF) | 21.46 | 19.35 | 8.33 | 21.9 |
| +DPO (RA) | 31.17 | 27.58 | 8.47 | 30.1 |
| +DPO (IRA) | 32.61 | 29.15 | 8.49 | 28.3 |

UltraFeedback, we leverage the resulting model $\pi_{\text{DPO}}$ to calculate the implicit reward for each prompt $x$ and response $y$, i.e., $\widehat{r}=\beta(\log\pi_{\text{DPO}}(y\mid x)-\log\pi_{\text{Qwen}}(y\mid x))$. Then we perform DPO on Qwen2-7B-Instruct using the Implicit-Reward-Augmented (IRA) UltraFeedback dataset. The results are shown in Table 5. We observe that augmenting the data with the implicit reward from the DPO (UF) model leads to superior performance even compared to augmenting the data with reward scores from the LLM judge, i.e., DPO (RA). This result highlights that DPO does not fully exploit the potential of the data. Moreover, this ablation demonstrates that our method is compatible with binarized preference datasets that only contain chosen and rejected response pairs, bypassing the need for reward scores from judge models.

#### Reward-Augmented Data is Superior Not Just Due to Its Increased Size.

In this part, we show that the success of our method is not merely due to the increased size of the training dataset. To illustrate this, we perform DPO on the dataset where reward augmentation is applied to the first half of the UltraFeedback data, which we denote as DPO (Half RA). By doing so, the reward-augmented data is of the same size as the original dataset, but with only half of the prompts and the corresponding responses being utilized. It can be observed from Table 6 that DPO (Half RA) outperforms fine-tuning on the whole UltraFeedback (UF) by a large margin and achieves comparable performance to applying reward augmentation across the entire UF dataset, which is denoted as DPO (RA).

Figure: Table 6: DPO trained on only half of the data with reward augmentation outperforms the baseline.
Refer to caption: x6.png

#### Reward-Augmented Data Mitigates the Unlearning Issue.

We first demonstrate that DPO suffers from the limitation of unnecessarily unlearning high-quality rejected responses, as discussed in Section 3.1. Specifically, on the test set of UltraFeedback, we calculate the log probability of each rejected response for the Qwen2-7B-Instruct model, its DPO (UF) model, and our method DPO (RA). In Figure 3, we plot the expected log probability for rejected responses with reward scores $\geq 5$. We find that DPO substantially decreases the probability of these high-quality rejected responses, confirming that the unlearning issue arises in practice. In contrast, our method alleviates this issue, although the probability is still slightly lower than the base model, which is proven to be the feature of DPO (Rafailov et al., ; Zhang et al., ; Xu et al., ).

#### Controllable Generation with Prompt.

In Table 7, we ablate how generations differ when

**Table 7: AlpacaEval 2.0 results when setting different goal rewards in the inference prompt.**
|  | $g=10$ | $g=8$ | $g=6$ | UF ($g=10$) |
| --- | --- | --- | --- | --- |
| LC WR | 31.17 | 28.66 | 25.56 | 24.44 |
| WR | 27.58 | 25.57 | 18.88 | 20.75 |

changing the goal rewards in the system prompt. We observe that the AlpacaEval 2.0 scores of the Qwen2-7B-It+DPO (RA) model change accordingly as $g$ varies. However, using the same $g=10$ prompt during inference for the Qwen2-7B-It+DPO (UF) model fails to give competitive results, indicating that our method is superior not only because of the additional system prompt.

#### Impact of the Accuracy of AI Feedback.

We consider the 19.8k prompts from a 1/3 subset of UltraFeedback following the setup from Snorkel (Hoang Tran, ). Five on-policy responses are first generated from Llama-3-8B-Instruct(^7^77https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct). An external reward model is followed to rank these responses. We choose the best and worst responses as the chosen and rejected ones. DPO is then performed on the resulting preference pairs and the reward-augmented pairs. To ablate how our method will be impacted by the accuracy of AI feedback, we experiment with two reward models as the ranker: PairRM (Jiang et al., ) and ArmoRM (Wang et al., ). PairRM is a small-sized (0.4B) pairwise reward model, while ArmoRM is a 8B model that is state-of-the-art on RewardBench (Lambert et al., ) and much stronger than PairRM. We implement a variant (denoted as RA+) of the proposed reward augmentation method that only conditions on the goal rewards of the chosen responses, not those of the rejected ones, leading to datasets that have the same size.

**Table 8: Ablation on the impact of AI feedback quality on the AlpacaEval 2.0 benchmark.**
|  | Llama-3- | PairRM (0.4B) | ArmoRM (8B) |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | 8B-Instruct | DPO (UF) | DPO (RA+) | DPO (RA) | DPO (UF) | DPO (RA+) |
| LC WR | 22.92 | 41.76 | 44.72 | 48.20 | 42.32 | 48.73 |
| WR | 23.15 | 45.79 | 44.70 | 53.17 | 42.79 | 45.36 |

The results in Table 8 demonstrate that training on augmented data conditioned on both chosen and rejected rewards is necessary for PairRM feedback, while relabeling with only the chosen rewards is sufficient to achieve strong performance for ArmoRM feedback. This aligns with our motivation outlined in Section 3.1: in noisy preference data, rejected responses may actually be of high quality, unlearning which can degrade performance. Similarly, low-quality chosen responses may also be reinforced. This issue does not arise with strong reward models that provide accurate preferences.

**Table 9: Comparison between our method, i.e., Llama-3-8B-Instruct+DPO (RA) and baselines fine-tuned on the same model and on-policy data ranked by PairRM.**
|  | SLiC-HF | ORPO | CPO | RRHF | KTO | IPO | RPO | R-DPO | SimPO | Ours |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LC WR | 26.9 | 28.5 | 28.9 | 31.3 | 33.1 | 35.6 | 40.8 | 41.1 | 44.7 | 48.2 |
| WR | 27.5 | 27.4 | 32.2 | 28.4 | 31.8 | 35.6 | 41.7 | 37.8 | 40.5 | 53.2 |

Moreover, in Table 9, we compare our method and various baselines under the same setting on the AlpacaEval 2.0 benchmark, including SLiC-HF (Zhao et al., ), ORPO (Hong et al., ), CPO (Xu et al., ), RRHF (Yuan et al., ), KTO (Ethayarajh et al., ), IPO (Azar et al., ), R-DPO (Park et al., ), and SimPO (Meng et al., ), where the results are from Meng et al. (), as well as the RPO (Adler et al., ) baseline that we implement. Our method outperforms the above algorithms by a considerable margin.

#### Conditioning on Multi-Attribute Rewards Enables SOTA Models.

In previous

**Table 10: Our method trained with DPO achieves SOTA when conditioning on $5$-dim rewards.**
|  | AlpacaEval 2.0 |  |  |
| --- | --- | --- | --- |
|  | LC Win Rate | Win Rate | Avg. Len. |
| Ours | 56.57 | 52.19 | 1840 |
| SimPO | 53.70 | 47.50 | 1777 |
| OpenChat | 17.48 | 11.36 | 1362 |

parts, our method is implemented by conditioning on the scalar reward values given by the judge models, either LLMs or reward models. We find that our approach is generalizable to settings of multi-dimensional rewards that correspond to different attributes, such as helpfulness and truthfulness. Specifically, we follow the setting from the last part to construct the preference dataset by applying the ArmoRM reward model to the on-policy responses generated by Llama-3-8B-Instruct. Since ArmoRM is a multi-objective model that not only gives a scalar reward value but also predicts human-interpretable fine-grained attributes, we first select $5$ attributes (namely complexity, instruction following, honesty, helpfulness, and intelligence depth) that have the highest average coefficients on the UltraFeedback data. Then we relabel the data by conditioning on the $5$-dim reward and follow the implementation of using ArmoRM described in the last part. The resulting model achieves state-of-the-art within the Llama-3-8B-Instruct model family, surpassing the strong baselines including SimPO (Meng et al., ) that is trained also on on-policy data ranked by ArmoRM, and OpenChat (Wang et al., ) fine-tuned with Conditioned-RLFT from the same Llama-3-8B-Instruct model.

#### Comparison with Conditional Fine-Tuning Baselines.

We further compared with additional conditional post-training baselines on the offline UltraFeedback dataset (i.e., without on-policy data), including DPA (Wang et al., ) that performs preference-conditioned rejection sampling fine-tuning, and SteerLM (Dong et al., ) that proposed attribute conditioned SFT. Since both baselines aim to optimize a user-controllable attribute-conditioned LLM that is optimal under diverse preference profiles with different coefficients of the reward’s attributes, in Figure 4, we plot the win rate curves of these methods under varying preference profiles, such as adjusting verbosity preferences as considered in Wang et al. (). It can be observed that fine-tuned from Zephyr-SFT(^8^88https://huggingface.co/alignment-handbook/zephyr-7b-sft-full), our method achieves the best AlpacaEval 2.0 win rate. In addition to the comparison with the implemented RPO (Adler et al., ) in Table 9, we also report the performance of RPO fine-tuned on additional models including Qwen2-7B-Instruct and Gemma2-9B-It. As shown in Table 11, the implemented RPO is outperformed by our method across these models.

Figure: Figure 4: Comparison with DPA and SteerLM on Zephyr-SFT.
Refer to caption: x7.png

## 7 Conclusion

In this paper, we first investigate the limitations of direct alignment algorithms, which arise from focusing solely on relative preferences while neglecting the qualities of the responses and their gaps. Specifically, since many rejected responses are only slightly worse than the chosen ones, striving to maximize the reparameterized reward gap will cause overfitting and unnecessarily unlearning the high-quality rejected response. Moreover, the directly aligned LLMs often struggle to differentiate between responses of varying quality, indiscriminately learning the low-quality chosen responses and failing to generalize effectively to more optimal responses that are sparse in the preference data. To resolve the above limitations, we introduce a straightforward solution—learning reward-conditioned policies. By optimizing the LLM to generate responses conditioned on their qualities, it can better differentiate between quality levels and learn from the entire spectrum. Motivated by this, we propose a data relabeling method that constructs reward-augmented datasets by conditioning on the quality of responses as the goal quality. In experiments, we fine-tune various LLMs by applying DPO on our reward-augmented data. The results demonstrate that our approach consistently delivers significant performance improvements across various instruction-following benchmarks and increases the average accuracy on academic benchmarks. Comprehensive ablation studies demonstrate that our method not only enhances the utility of preference data but also addresses issues beyond simple dataset expansion, such as mitigating the unlearning limitation. Our method also achieves state-of-the-art results on AlpacaEval 2.0 when applied to on-policy data.

## Appendix A Experiment Details

### A.1 Setup

We use the following prompt during training. Here, the reward values are the quality scores given by the judge models that exist in the preference dataset.
The prompt is set as the system prompt whenever the LLM supports, such as Qwen2-7B-Instruct and Llama-3.1-8B-Instruct, and it is prefixed before the original prompt when the LLM doesn’t support system prompting, such as Mistral-7B-Instruct-v0.3 and Gemma-2-9B-It.

Training prompt

You are an assistant that generates responses for the instruction while implicitly achieving the following target score (on a scale of 1-10, where 1 is lowest and 10 is highest):

Overall score: {reward_value}.

At inference time, we use almost the same prompt, except that the goal score is the highest one, i.e., the overall score is $10$.

Inference prompt

You are an assistant that generates responses for the instruction while implicitly achieving the following target score (on a scale of 1-10, where 1 is lowest and 10 is highest):

Overall score: 10.

In our experiments using UltraFeedback, we directly leverage the LLM-as-Judge scores provided by GPT-4 in the dataset, which range from $1$ to $10$. For our method that is applied to on-policy data ranked by external reward models, including PairRM and ArmoRM, we apply linear transformations to normalize the resulting reward scores, ensuring they are scaled within the same $1$ to $10$ range.

For hyperparameters, we utilize a KL regularization coefficient of $\beta=0.01$ in DPO, and we adopt the AdamW optimizer (Loshchilov, ). The batch size is set to $128$, with a learning rate of $5e-7$ and a warmup ratio of $0.1$. Furthermore, we observe that for models such as Qwen2-7B-Instruct and Gemma-2-9B-It on UltraFeedback, as well as Llama-3-8B-Instruct on on-policy data, both DPO and our proposed method yield improved performance when employing the conservative DPO (cDPO) technique (Mitchell, ). Consequently, for these models, we set the label smoothing hyperparameter from the Alignment Handbook (Tunstall et al., ) to $0.3$, while keeping it at 0 0 for the remaining models.

### A.2 Full Results

In Table 12, we present the full results on instruction-following benchmarks, which correspond to the performance illustrated in Figure 2 in the main text.

**Table 12: Results of the DPO models fine-tuned on UltraFeedback and on reward-augmented UltraFeedback. We evaluate on the instruction-following benchmarks including AlpacaEval 2.0, MT-Bench, and Arena-Hard-Auto.**
|  | AlpacaEval 2.0 | MT-Bench | Arena-Hard-Auto |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | LC WR | WR | Avg. Len. | Avg. | 1st | 2nd | Score | Avg. Len. |
| Mistral-7B-Instruct-v0.3 | 19.65 | 15.40 | 1503 | 7.67 | 8.00 | 7.34 | 17.0 | 494 |
| +DPO (UltraFeedback) | 18.76 | 16.93 | 1643 | 7.66 | 7.92 | 7.40 | 17.6 | 504 |
| +DPO (Reward-Augmented) | 25.99 | 28.36 | 2270 | 7.69 | 8.02 | 7.36 | 18.3 | 883 |
| Qwen2-7B-Instruct | 20.93 | 18.22 | 1788 | 7.90 | 8.23 | 7.56 | 24.3 | 617 |
| +DPO (UltraFeedback) | 21.46 | 19.35 | 1797 | 8.33 | 8.72 | 7.93 | 21.9 | 553 |
| +DPO (Reward-Augmented) | 31.17 | 27.58 | 1789 | 8.47 | 8.93 | 7.97 | 30.1 | 644 |
| Llama-3.1-8B-Instruct | 24.79 | 27.38 | 2081 | 8.44 | 8.99 | 7.90 | 26.9 | 831 |
| +DPO (UltraFeedback) | 28.67 | 30.21 | 2053 | 8.47 | 9.01 | 7.93 | 33.0 | 1070 |
| +DPO (Reward-Augmented) | 31.20 | 35.93 | 2006 | 8.47 | 8.91 | 8.03 | 34.4 | 824 |
| Gemma-2-9B-It | 49.20 | 37.58 | 1572 | 8.54 | 8.81 | 8.28 | 42.8 | 541 |
| +DPO (UltraFeedback) | 50.70 | 35.02 | 1464 | 8.54 | 8.70 | 8.37 | 35.8 | 456 |
| +DPO (Reward-Augmented) | 59.27 | 54.56 | 1872 | 8.59 | 8.93 | 8.25 | 43.9 | 611 |
| SPPO | 55.60 | 49.61 | 1822 | 8.40 | 8.53 | 8.26 | 47.6 | 578 |
| +DPO (UltraFeedback) | 52.75 | 40.58 | 1544 | 8.41 | 8.78 | 8.04 | 40.4 | 457 |
| +DPO (Reward-Augmented) | 60.97 | 66.41 | 2543 | 8.73 | 9.06 | 8.41 | 49.0 | 761 |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: - What are the benefits of keeping the rejected high-reward samples? Can the author provide evidence for consequential effects of the unlearning problem when LLM finetuned with the original DPO? The authors might need to start with a model that is not DPO fine-tuned and start to fine-tune using DPO and the proposed method.
- How does the proposed method compare with other recent advancements in preference alignment, such as those using model-based reinforcement learning or other reward-based approaches?
- Can the authors analyze which are rejected but high-reward samples in human feedback cases? Are they actually beneficial?
soundness: 3
strengths: - The paper presents a creative solution to the problem of overfitting and unlearning in direct preference alignment by introducing reward-conditioned policies. This approach is novel and extends the capabilities of existing alignment algorithms.
- The experimental results demonstrate significant performance improvements across a range of models and benchmarks. The comprehensive ablation studies further validate the effectiveness of the proposed method.
- The proposed method has broad implications for improving the alignment of LLMs with human preferences, which is a critical area of research for developing more reliable and user-friendly AI systems.
summary: The paper introduces a novel method to enhance the direct preference alignment of Large Language Models (LLMs) by incorporating reward-augmented data. The authors identify limitations in existing direct alignment algorithms, such as overfitting and unlearning of high-quality rejected responses, and propose a reward-conditioned policy to address these issues. The method involves a simple data relabeling technique that conditions preference pairs on quality scores, creating a reward-augmented dataset. This approach is shown to improve the performance of various models across multiple benchmarks, including instruction-following and academic benchmarks.
weaknesses: - While the method is innovative, it builds heavily on existing direct preference alignment algorithms.
- The experiments are extensive but primarily focus on instruction-following and academic benchmarks. Including additional real-world applications, diverse datasets could strengthen the generalizability claims.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: See above
soundness: 2
strengths: The empirical results in Section 6 are comprehensive across different capability and alignment evaluations as well as different models. The trends in the results are fairly consistent across settings, I think. The method itself is computationally efficient and simple to implement.
summary: This paper proposes augmenting reward modeling datasets with goal-conditioning. In this setting, one requests a response that has a particular reward score and assigns a binary label as to whether or not the datapoint achieves that score. Extensive experiments show that doing so improves the win rate and other capability-related evaluations of the model over relevant baselines.
weaknesses: 1. I am confused about Tables 1 and 2. How did you compute $\pi^*$? What are the assumptions used? If you assume a standard setting like the DPO paper, you would never get a policy that always generates one response over the other (because Bradley-Terry inherently assumes some noise). Also, just more realistically, LLMs are softmax parametrized, and the probability of a response can never be exactly 0. Overall, I am confused about Section 3.1, where, for example, "reward sparsity" is used to describe something different than what it traditionally means (sparsity over trajectory time, not over the dataset) and the logic is unclear. This makes the motivation for the algorithm unclear -- Section 3.2 is titled way too strongly for a method that has no clear theoretical backing. This weakness is the reason for giving a 6 instead of an 8. It is hard to justify accepting a paper with such vague and possibly incorrect text, even if the idea and method are performant.

2. I don't understand the method. All datapoints have only a binary indicator that one response was preferred over the other. So how are you determining the value of $r_w^i$ and $r_l^i$? Without some additional information, they would all be the same. 

3. I think the accuracy of the reward model matters a lot, and PairRM and ArmoRM are both pretty good reward models from my understanding. It might be better to do a more stark separation where eg one flips the ArmoRM score and sees what happens. I think this would be interesting, since LMs usually don't use the context in the way that we expect them to.

Minor: I would not use "unlearning" as a term to describe the phenomenon since that has a very different meaning in ML literature.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: What led to choosing the 1-10 scale for quality scores, and have you explored whether other scoring ranges might be more effective for reward conditioning?
soundness: 3
strengths: 1. The method is straightforward to implement, requires no architectural changes, and can be easily integrated with existing alignment algorithms.

2. The method is versatile since it works with both offline and online preference data and can handle multi-dimensional reward attributes.
summary: This paper addresses fundamental limitations in direct preference alignment algorithms that use binary preference data, where models tend to unnecessarily unlearn high-quality rejected responses while indiscriminately learning low-quality chosen ones. The authors propose a simple reward-augmented data relabeling method that conditions the training data on explicit quality scores, which allows models to learn patterns across the full quality spectrum. This approach is orthogonal to existing direct alignment algorithms and can be applied with any direct preference algorithm.
weaknesses: 1.  While the paper uses a 1-10 scale, there's no analysis of how different scoring scales (e.g., 1-5, 1-100) might affect performance or what the optimal scale might be.

2. The paper does not include human evaluation studies to validate the effectiveness of reward-augmented responses.

3. The paper would benefit from a discussion of limitations and potential future directions.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. How does the reliance on reward labels impact the overall data quality, and would the model benefit more from incorporating high-reward rejected responses compared to low-reward chosen ones?
2. How does the use of previously unseen instructions in DPO training affect the model's adaptability, and could additional examples or detailed explanations enhance the clarity of the experimental results?
3. Could the authors further justify the categorization of responses beyond binary labels, and would explain how positive and negative goal-conditioned learning enhances attention of data with varying quality.
soundness: 3
strengths: This paper presents a novel approach to preference alignment in LLMs by introducing reward-conditioned policies. The method enhances model performance by conditioning on reward values, enabling LLMs to better generalize to high-quality responses and mitigate the unlearning Issue.
summary: This paper tackles the limitations of direct alignment in LLMs by introducing reward-conditioned policies. The authors propose a data relabeling method that incorporates reward goals into preference datasets, allowing the model to learn from both high- and low-quality responses. By conditioning on the highest reward at inference, the approach enhances LLM performance across benchmarks, improving alignment and reducing overfitting.
weaknesses: 1. The method relies heavily on reward labels, whether in directly constructing the reward-augmented dataset or through a multi-attribute rewards dataset. The actual impact of these rewards on data quality remains unexplored. It is unclear if the model would benefit more from high-reward rejected answers than from low-reward chosen answers, which warrants further investigation.
2. Learning from preference usually involves guiding the model towards specific output styles, such as aligning to certain type of instructions or adapting to varied language tones. The authors used previously unseen instructions in DPO training that match existing data, potentially limiting the model's adaptability. The experimental results require additional examples and detailed explanations to clarify this impact. And in appendix $D_N^i$ training prompts should be added.
3. The authors’ setup inadequately considers response scores (rewards), despite splitting the augmented dataset into two categories, $r_w^i$ and $r_l^i$. The rationale categorization of additional quality dimensions should further considered beyond binary labels.
4. The proposed approach and insights lack a strong foundation. Restructuring the dataset (new pairs) based on reward intuitively seems more effective. In contrast, positive and negative goal-conditioned learning does not necessarily ensure that higher-quality responses receive more attention.
5. The formatting in line 200 contains typographical errors. $r_{g = r_l^i}(x, y_w^i) = 0$.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: Above
soundness: 3
strengths: 1. The proposed method is straightforward and easy to understand.
2. Experiments show that the proposed method outperforms DPO on multiple base models and various benchmarks.
3. The algorithm can somewhat mitigate the unlearning problem of DPO, which might be useful.
summary: This paper proposes a method for learning from a reward-augmented preference dataset. The method is straightforward. During training, the LLM is prompted with a special text like "Generate a response of reward 5". If such a prompt is given, only the response that has reward 5 is preferred during DPO training, the other response (even it has a higher reward) is considered as dispreferred one. This allows us not only to learn from preferred response data but also learn from the suboptimal response. This also allows the leverage of quantitative reward values. Exps is conducted on reward datasets like UltraFeedback, and show improvement over the DPO method.
weaknesses: 1. **Conditioning on low reward data is only empirically useful.** Despite improvements indicated by the experiments, I find the method to be highly empirical and lacks theoretical guarantees. During sampling, the LLM is always prompted to generate only high-reward responses, which makes learning low-reward prompts during training useful only because it proves "generalization".

2. **Lack of comparison with other reward-augmented methods**. One main claimed contribution of the paper is being able to learn from quantitative rewards, but it did not compare with any reward-augmented methods. Related prior works are 

Noise Contrastive Alignment of Language Models with Explicit Rewards   https://arxiv.org/abs/2402.05369

Ulma: Unified language model alignment with demonstration and point-wise human preference https://arxiv.org/abs/2312.02554

3. **Reward labels can be extremely noisy.**  In the experiments, all datasets use GPT-4 generated reward, which is much more deterministic than human reward. Preference learning exactly tries to avoid noises in human evaluations and only considers preference order. If the dataset reward is given by a human instead of GPT4. One response can be of multiple rewards according to different annotators. So it is unreasonable when you prompt the LLM to generate a response of reward 9, the response of reward 10 is "dispreferred".

4. **Reward values are continuous.** The method conditioned the LLM on a specific given reward value. However, since the reward signal is an inherently continuous value, it is not very suitable for use as the condition. "Generate a response of reward 3.75" does not make much sense, compared with "Generate a response that is truthful/contains no mathematic mistakes (some deterministic requirement)"

Meta-review: The paper introduces a novel data relabeling method for LLM preference alignment that conditions training on explicit reward scores rather than just binary preferences. This approach helps prevent "unlearning" of high-quality rejected responses and enables better generalization to optimal responses by learning patterns across the full quality spectrum. The empirical results show consistent improvements over standard DPO across multiple models and benchmarks, including both instruction-following tasks and academic evaluations.

Strengths emphasize on simplicity, as it requires only data relabeling without algorithm changes and comprehensive empirical validation across diverse models and benchmarks, and theoretical analysis providing convergence guarantees. The main weaknesses include heavy reliance on reward labels which may be noisy in practice, especially with human annotators vs GPT-4; lack of human evaluation studies to validate the real-world effectiveness; and the writing quality and clarity of explanations could be improved in several sections. Overall, there were a mix of reviews and the authors managed to address some of the concerns but given no strong champion for the work, I recommend the authors incorporate the reviewer suggestions on including human evaluation studies (QA benchmarks are very different from long-from generation). The paper may also benefit from more discussion of limitations and failure cases, and clearer positioning relative to methods using absolute preferences.

Decision: Reject