Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
Authors: Shenao Zhang
Zhihan Liu
Boyi Liu
Yufeng Zhang
Yingxiang Yang
Yongfei Liu
Liyu Chen
Tao Sun
Zhaoran Wang
Venue: ICML 2025 poster (2025)
Keywords: Alignment
RLHF
LLM
Primary area: deep_learning->large_language_models

Abstract: Preference alignment in Large Language Models (LLMs) has significantly improved their ability to adhere to human instructions and intentions. However, existing direct alignment algorithms primarily focus on relative preferences and often overlook the qualitative aspects of responses, despite having access to preference data that includes reward scores from judge models during AI feedback. Striving to maximize the implicit reward gap between the chosen and the slightly inferior rejected responses can cause overfitting and unnecessary unlearning of the high-quality rejected responses. The unawareness of the reward scores also drives the LLM to indiscriminately favor the low-quality chosen responses and fail to generalize to optimal responses that are sparse in data. To overcome these shortcomings, our study introduces reward-conditioned LLM policies that discern and learn from the entire spectrum of response quality within the dataset, helping extrapolate to more optimal regions. We propose an effective yet simple data relabeling method that conditions the preference pairs on quality scores to construct a reward-augmented dataset. The experiments across various benchmarks and diverse models demonstrate that our approach consistently boosts DPO by a considerable margin. Through comprehensive ablation studies, we demonstrate that our method not only maximizes the utility of preference data but also mitigates the issue of unlearning, demonstrating its broad effectiveness beyond mere data expansion. Our code is available at https://github.com/shenao-zhang/reward-augmented-preference.

Revard-Augmented Data Enhances Direct Preference Alignment of LLMs
Shenao Zhang Zhihan Liu Boyi Liu Yufeng Zhang
Yingxiang Yang? Yongfei Liu
Liyu Chen?
Tao Sun?
Zhaoran Wang'
Abstract
1. Introduction
Reinforcement Learning from Human Feedback (RLHF) has recently remarkable success in aligning Large Lan- guage Models (LLMs) to follow instructions with human intentions. In this approach, Al-generated feedback serves as stand-in for human preferences, assessing and rank- ing responses to prompts to construct a preference dataset. This dataset is then utilized preference optimization algo- rithms to fine-tune L.LMs. Among them, direct preference alignment (Rafailov et al., 2024b; Azar al., 2023; Zhao et al., 2023; Ethayarajh et al., 2024) that bypasses the need for an explicit reward model has garnered interest for its simplicity and cost efficiency. However, these algorithms mainly concern relative preferences and often overlook the quality of responses and their gaps, leading to limitations in their effectiveness.
Preference alignment in Large Language Models (LLMs) has significantly improved their ability to adhere to human instructions and intentions. However, existing direct alignment algorithms primarily focus on relative preferences and often overlook the qualitative aspects of responses, despite having access to preference data that includes reward scores from judge models during AI feedback. Striving to maximize the implicit reward gap between the chosen and the slightly inferior rejected responses can cause overfitting and unnecessary unlearning of the high-quaity rejected responses. The unawareness of the reward scores also drives the LLM to indiscrimi- nately favor the low-quality chosen responses and fail to generalize to optimal responses that are sparse in data. To overcome these shortcomings, our study introduces reward-conditioned LLM policies that discern and learn from the entire spectrum of response quality within the dataset, helping extrapolate to more optimal regions. We propose an effective yet simple data relabeling method that conditions the preference pairs on quality scores to construct eegard-augmented dataset. The experiments across various bench- marks and diverse models demonstrate that our approach consistently boosts DPO by considerable margin. Through comprehensive ablation studies, we demonstrate that our method not only maximizes the utility of preference data but also mitigates the issue of unlearning, demonstrating its broad effectiveness beyond mere data expansion. Our code is available https://gitee com/ shenao- zhang/ teward-auugmetted--eeeeeee
Specifically, direct alignment algorithms such as DPO (Rafailov et al., 2024b) focus maximizing the implicit reward difference between accepted and rejected responses. This approach can lead to overfitting, as high-quality but rejected responses are unnecessarily unlearned (Adler et al., 2024). Even worse, since the datasct provides only a sam- ple estimate of true preferences, the rejected responses can actually be more aligned with human preferences than the accepted ones in expectation. Similarly, due to the unaware- ness of the responses' qualities, direct alignment will also result in the indiscriminate learning of the chosen responses, even when they are of low quality. As result, the directly aligned LLMs often struggle to differentiate between re- sponses of varying quality and fail to generalize effectively to more optimal or the highest-reward responses that are sparse in the preference data, which is another limitation.
To address these issues, we propose learning reward- conditioned policies as straightforward fix. By optimizing the L.M to generate responses conditioning on their quali- ties, the model is allowed to discern and leverage patterns within responses of varied quality. As result, learning from both chosen and rejected responses alleviates the issue of unlearning high-quality rejected responses; distinguish- ing between varying-qualin chosen responses alleviates the issue of indiscriminately accepting low-quality ones. By identifying common patterns in responses of similar quality
NNrttwwestern University "ByteDance. Correspondence to: Shenao Zhang <shenao@uus halkemao@u.nothwwsseem.....
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
and distinguishing them from those of differing quality, the LLM becomes more adept generalizing more optimal responses that are sparse in data.
where h denotes the human rater and the expectation is over h to account for the randomness of the human raters' choices. After pretraining and Supervised Fine-Tuning (SFT), Reinforcement Learning from Human or AI Feed- back (Ouyang et al., 2022; Bai et al., 2022b) is typically employed to enhance the ability of the language model to follow instructions with human preferences.
With this motivation, we introduce an effective yet sim- ple data relabeling method to construct reward-augmented datasets, We define a goal-conditioned reward using an indi- cator function that compares the goal reward with the actual quality score, such as the reward value given by the judge model during AI feedback. This allows to relabel each preference pair, generating two new pairs conditioned on the reward goals of both the chosen and rejected responses. The resulting augmented dataset, which contains these newly conditioned pairs, can enhance the performance of existing direct alignment algorithms. Our method can be applied to any preference dataset and followed by off-the-shelf direct alignment algorithms to boost their performance.
RL from AT Feedback (RLAIF). The RLAIF framework involves two major steps: preference dataset construction with AI feedback and preference optimization. As surro- gate for human preference, Al feedback, including LLM- as-Judge (Zheng et al., 2024; Cui et al., 2023) and Reward- Model-as-Judge (Adler et al., 2024; Dong et al., 2024), can be used rank responses and generate preference pairs. Specifically, consider the judge model r(x: y) X x + R that outputs scalar reward value representing the quality of under x. For each prompt E X, two responses, y/1 and y2, are independently sampled-eitth either from the same reference model (Xiong et al., 2024; Wu et al., 2024) or sev- eral different models (Zhu et al., 2023a; Zhang et al., 2024). Then r(a,, n)) and r(a, y22 are evaluated to determine the preferred response you = argmax (e 7(z,, y) and dis- preferred response y argmin (y1.92) r(z, y). By sam- pling responses and ranking them for a set of N prompts, we get a preference dataset: the simplicity of our discussions, we assume that the reward function r is bounded in [0, [0,rmax] F'ma.)
In experiments, we first apply our method on UltraFeed back (Cui et al., 2023) and perform DPO (Rafailov et al., preference dataset by fine-tuning on various models, including Zephyr-7B-3 (Tun- stall et al., 2023b), 2023a), (wwe2--BBInnsruul (Yang al., 2024), Llama-3.1- 8B-Instruct (Dubey al.. 2024), Gemma-2-99 2-9B-I It (Team et al., 2024), and SPPO (Wu et al., 2024). The results show that our method consistently boosts the performance of these models as well as their DPO models by a large margin on instruction- instruction-folloiing benchmarks such as Al- pacaEval 2.0 (Dubois et al., 2024), MT-Bench (Zheng et al., 2024), and Arena-Hard Hard-Auto (Li et al., 2024b). Our method also improves the average accuracy on a variety of academic benchmarks (GSM8K, GPQA, MUSR, TruthfulQA, BBH, and ARC). Moreover, our findings also demonstrate an im- proved utility of the preference data: a subsequent round of DPO using the augard-augmented data can still significantly enhance the model fine-tuned with DPO; relabeling the bina- rized preference dataset with the DPO implicit reward leads to further performance gains. Additional ablation studies also suggest that our method addresses the problem of un- learning and is superior not just due to the increased dataset size. When applied to on-policy data, our method enhances the DPO model, enabling it to surpass various baselines and achieve state-of-the-ard performance on AlpacaEval 2.0.
2024b) on this seward-augments
- (Jiang et al.,
DN = {(x', y ) For
Direct Alignment from Preference. The objective for the LLM A is maximize the KL-regularized expected reward. Recent work (Azar et al., 2023; Zhao et al., 2023; Tunstall et al., 2023b; Ethayarajh et al., 2024) proposed to align the LLM directly with the paired data by deriving the preference loss as a function of the LLM by the change of variables. Among them, the Direct Preference Optimization (DPO) (Rafailov et al., 2024b) loss has the following form:
LDPO(IT DN) = (oo.w.)))-11 log
of Blog Teer (vu [x)
x (Bu x)
Blog T T(U x) Teet (3r xx)
where /3 is a hyperparameter corresponding to the KI. diver- gence regularization, at-) is the logistic function, and ne is some reference LLM policy, such as the SFT model.
2. Background
Consider an I.LM π E Aÿ that takes the prompt x E X as input and outputs the response y E V, where X' and y are spaces of prompts and responses, respectively. Given
3. RewdrrnConn Addresses Limitations of Direct Preference Alignment
x E X. a discrete probability distribution (- z) E Ay is generated, where Ay is the set of discrete distributions over y. We define the true human preference distribution as
3.1. Limitations of Direct Alignment from Preference
We will first demonstrate the limitations of vanilla direct alignment over the preference data.
p" p"(y1 > 32 x) = Ea [1(h prefers y1 over y2 given =)),
2

---

Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
High-Quality Rejected Responses are Unnecessarily Un- learned. The dataset y often contains preference pairs where the rejected response y/ is only marginally worse than the chosen one Vu- Direct alignment algorithms, however, primarily focus on relative preferences and are unaware of the responses' quality values and gaps. Striving to maxi- mize the reparameterized reward gap between the chosen and rejected responses will risk overfitting and unnecessary "unlearning", i.e., probability decrease, of high-quality re- sponses, potentially diminishing the model's performance by discarding valuable alternatives. Furthermore, in such a finite data regime where only a sample estimate of the true preference is accessible, it can be very possible that p" (yn= > the x) > 0.5, i.e., YI is in fact more preferred than you in expectation. This issue becomes even more pronounced when the preference data generated with the imperfect judge model is noisy.
responses of varying quality and fails to generalize effec- tively to the sparse optimal responses.
response
91
1/2
y/3
r(z, v) DN=2
9
0
{ui
>>
V/
1/2
>
Va
="(y ())
1 a
a
0
x"(v 2, g 9)
1
0
0
x (y 2. g 1)
0
0
x (y 2. = 0)
0
1
Table 2. Low-quality chosen responses such as ya can be learned:
x" indiscriminately generates yi and y2- Reward-conditioned policies distinguish the differences and learn the behaviors corre- sponding to different reward scores.
response
81
g/2
3.2.
Policies Learn from the Full Spectrum of Response Quality
r (xz, v) DN=1
9
8
{uı
>>
y2
(y [x)
1
0
A straightforward way to address the limitations of direct alignment aggoritiss-specfffcally their inability to ac- count for the quality of responses- to optimize a reward- conditioned policy. In this approach, the LLM policy is trained to generate responses corresponding different re- ward values, enabling it to become aware of and adapt to these reward distinctions. By doing so, the L.L. not only learns the patterns associated with the preferred responses but also retains valuable information from the rejected ones, preventing the unlcarning of high-quulity rejected responses. For example, in Table 1, doward-conditionss last two rows) learn to generate both yı and y2, instead of un- learning y2. This reward- conditioning also enhances the model's ability to differentiate between responses of varying quality, even if both are preferred over rejected alternative, as illustrated in Table 2. Besides, by extracting common patterns across responses with different quality levels, the LLM becomes more generalizable and is capable of generating the highest-quality responses with reward T'ax (e.g., 10), which are often sparse in the training data.
" (y/
I. g = 9)
1
T¹ (y g = 8)
0
Table 1. High-quality rejected responses such as y2 can he un- necessarily unlearned: =" (- z)) deterministically generates yı- Neward-conditionn policies learn both responses and are easier to generalize to g = 10 with features extracted from = 8 and 9,
We illustrate this limitation with the example in Table 1, where we define the maximum reward Tmax as 10. For DN== that contains a single preference pair' with reward r(x, 31) = 9 and r r(x, 3/2) = 8, the optimal policy learned from 1 is π (y) z) 1. This causes the model to avoid generating y/2 a response of nearly equivalent quality.
policies (the
Low-Quality Chosen Responses are Indiscriminately Learned. For a similar reason, direct alignment algo- rithms also indiscriminately reinforce the chosen responses. As illustrated in Table 2, when DN=2 contains two pref- erence pairs, where one of the chosen responses, /2. is of low quality, x² still indiscriminately generates y2 with arbitrary probability 0 < a < 1, i.e., T T122 | x) a.
4. Method
With the above motivation, we propose a data relabeling method that constructs reward-augmented dataset by con- ditioning the preference pairs on the reward values given by the judge model r. Specifically, we define the goal- conditioned reward function R(x, y.g) 9) -(g r(æ, v))² as a function of the reward function r. The objective of the reward-come condodntidon policy т x,g) is thus to minimize the square difference between the goal reward g and the response reward r(z, y). which is equivalent to maximizing the goal-conditioned reward R(x, y, g). i.e.
Reward Sparsity. Preference data often contains re- sponses that, despite being preferred in pairwise compar- isons, exhibit substantial variation in quality. As a result, the optimal responses- those associated with the highest reward value Tmax- are sparse in the dataset. Since direct alignment algorithms do not account for these reward val- ues, the trained model struggles to differentiate between
For simplicity, we write (z, yw y1) E D' as yo > Vi-
3

---

Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
Fort [I,
System prompt she "Generate responses of score riy"
Preference data
Reward-augmented data
Chosen response: y Rejected response:
Dj ( + +x X
Dy (ol yho m
DaN = DV U D
DPO
System prompt sf: GGeneaate Tesponses of score r/-"
D {sf tff+,,,l 1
Chosen response: y) Rejected response: X
Figure 1. Illustration of our method: construction of reward-augmented preference datasets followed by direct alignment algorithms.
min Eg gm Dx D...matt- [(g - rzz, 8))²
inference time, the LLM is conditioned on the optimal goal
x
g" mmux that is the highest possible reward value, e.g.,
max E -- D.y- R(x, v. (4.1)
g m'ax 10, to generate the responses.
To optimize the RHS of (4.1), we observe that under the new goal-conditioned reward metric r, we have
Morcover, we provide the following theoretical guarantees for our method (see A.4 for formal description).
R(x, Vwn 9 r > R(x, vi g r'y) - (rr n R(x, vi,g ri) = > R(x, Van = ri) = (ri rt)?
Theorem 4.1 (Informal version). Let J(T)
Exwdo.yret x.gr)) [[xx, y, g²)] be the performance mea- sure, where R denotes the ground-truth goal-conditioned reward function and g denotes the optimal goal. Under mild assumptions, the policy ß optimized from the reward-angmented DPO with an SFT regularizer satisfies that with probability at least 1 - 8,
Thus, each pair can be relabeled to create two new pref- erence pairs based on two distinct goals: when g = r V'n > yf: when g = rf. f/= > Vig Then any direct alignment algorithm can be applied to this new goal-conditioned prefer- ence dataset. Compared to tee-tuinng on the original dataset DN. the model learns to capture not only desirable behay- iors but also undesirable ones from the geward-augmentee dataset. This approach helps identify patterns across high- and low-quality responses, enabling the LLMs to discern and learn from the entire spectrum of response quality and extrapolate more optimal responses at inference time, by conditioning on higher reward goals.
J(f)) I V6 (66 (1 + exx(( (B)B))((((C)))) ((C (Ri (st)) + 11
4
+ +Egndd
[L (1,,, (
(4.2)
argmax ((T) and ' / /log (N: 1110)//88 with E (6 (1 N)) -N-- Here, N denotes the number of preference pairs in D, B denotes the upper bound of the reward models, and the partial coverage coefficient Cyp (R; nt", is defined in Assumption A.3.
where π
We illustrate our method in Figure 1. For each preference pair with index i in DN two goals are defined, correspond- ing to the reward values of the chosen response y'n and the rejected response yi- Specifically, under the first goal g = ri. the relabeled rewards are R(x, v g) = and R(a,visg) -(re (ré r/)2. The original ranking of re- sponses remains the same, except that the LLM is preference optimized conditioned on y r Similarly, under the sec- ond goal rř. the relabeled rewards are R(z, vi g) = 0 and R(x, V(ngg) g) = - (r rt)2. Thus, the chosen and rejected responses are reversed as yi and Vig By gener- ating preference pairs conditioned on the goal reward for both the chosen and rejected responses, we obtain reward- augmented dataset of size 2.V. Finally, this new dataset can be used with any direct alignment algorithm, such as DPO.
The detailed proof is provided in A.5. The above theorem shows that our method attains global convergence to the optimal policy and the suboptimality decays at the order of N 1/2 (N denotes the size of the reward-augmen augmenntdd prefer- ence dataset), which provides a theoretical justification for the strong empirical performance of the introduced reward- augmented DPO, Unlike prior works on goal-conditioned RL with supervised learning (Yang et al., 2022; Ghosh et al., 2019), which typically establish weaker results such as lo- cal performance improvements or the optimization of a lower bound on J (TT), our analysis guarantees global con- vergence to the optimal policy. This distinction underscores the significance of integrating DPO-like methods with goal- conditioned approaches.
In this work, we implement the reward-comm conditioon policy (y x. g) as the LLM with a system prompt (or a prefix before the user prompt x if system prompts are not supported by the LLM) such as "generate responses of score g". At
4

---

Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
5. Related Work
sampling fine-tuning (Yuan et al., 2023; Dong et al., 2023a) that conditions on the direction of the multi-objective re- ward, i..., user-dependent linear combination of the reward attributes (helpfulness and verbosity in their experiments), that represents diverse preference objectives. These methods aim to train a single LLM that can flexibly adjust to vari- ous user preference profiles. On the contrary, our method targets the limitations of direct alignment algorithms by in- troducing reward reward-augmented relabeling. This also differs from Conditioned-RLL RLFFT (Wang et al., 2023a), which su- pervised learns dass-conditioned policies with data source information, and (Cai et al., 2023; Chen et al., 2024) that focus on unified alignment with binarized or reward datasets. Reward- -wwre Preference Optimization (RPO), introduced in Nemotron-4 (Adler al., 2024), attempts to approximate the reward gap using the implicit reward and is motivated to resolve the unlearning issues of DPO, which our work also addresses. However, we show that more limitations beyond unlearning can be simply fixed with reward-conditioned LLMs and propose an easy-to- timpllmmmmeeeentt data relabeling method without algorithm changes. (Pang et al., 2024) ad- dressed DPO's s tendency to reduce the probability of the chosen response by incorporating an NLL loss. In contrast, our work focuses dency to overlook qualitative aspects of responses and proposes a data relabeling approach that requires no algo- rithm changes. It also differs from conditional sequence modeling based on SFT (Chen et al., 2021; Lloret et al., 2024). Due to the lack of textual feedback in UF, we empir- ically compare with the reward feedback variants of (Lloret et al., 2024), including SteerI.M and DPA. Similar to our work, (Kim et al., 2024) also investigates how preference optimization can overlook qualitative aspects of responses. However, their focus is on overfitting to preference data, and they propose incorporating quality margins into the optimization objective. In contrast, our approach does not involve algorithmic modifications, but rather directly targets the limitations caused by the unawareness of the reward scores. Our work also differs from (Park et al., 2024), which introduces constraint-based regularization term specifi- cally aimed at mitigating verbosity bias.
Preference Dataset Construction. In order for the LLMs to follow instructions and better align with human intents, it common practice to build preference dataset containing set of prompts and pair of responses for each prompt, whose qualities are ranked by humans (Ouyang et al., 2022) or judge models (Bai et al., 2022b). A popular pipeline (Cui et al., 2023; Tunstall et al., 2023b; Wang et al., 2024c; Ivison et al., 2023; Zhu et al., 2023a) for constructing offiin (i.e., fixed) datasets involves sampling off-policy responses from various LLMs for each prompt in the hope to increase the response diversity. The preference data can also be generated online (Guo et al., 2024) iteratively (Bai et al., 2022a; Xu et al., 2023; Gulcehre et al., 2023; Hoang Tran, 2024; Xiong et al., 2023; Dong et al., 2024; Calandriello et al., 2024; Rosset et al., 2024) by sampling and ranking on-policy responses from the training L.L.M Recent works (Zhang et al., 2024; Cen et al., 2024; Xie et al., 2024) have also proposed systematically exploring the responses online and actively eliciting the preference. The proposed method in this paper is orthogonal to the construction ways of the preference data and can be applied to any dataset created either off-policy or on-policy.
different limitation of DPO its ten-
Preference Optimization. Preference optimization meth- ods generally follow two approaches. The first involves learning a point-wise reward model, such as the Bradley- Terry model, and using RL algorithms like PPO (Schulman al., 2017: Zheng et al., 2023; Xu et al., 2024b) or REIN- FORCE (Williams, 1992; Li et al., 2023; Ahmadian al., 2024), to maximize the KL-regularized expected reward. The second approach is direct alignment (Rafailov et al., 2024b; Azar et al., 2023; Zhao et al., 2023; Ethayarajh et al., 2024; Liu et al., 2024b), which gets rid of a separate reward model that is computationally costly to train. In this work, we mainly focus on the limitations of direct alignment algo- rithms, particularly their unawareness of the quality aspects of responses. For PPO-style alignment algorithms that fit and maximize an explicit reward, preference data is only used to learn the reward model, and policy training is per- formed in an online manner, where responses are sampled from the LLM and their reward values directly play a role during the RL optimization. This avoids drawbacks inherent to direct alignment methods, detailed in Section 3.1.
Sample-efficient RL and RLHF Our theoretical analysis is also related to rich line of research on sample-efficient RL and RLHF (Jin et al., 2021; Zhan et al., 2023; Liu et al.,
2024a; Zhang et al., 2024; Liu et al., 2024b; Xiong et al., 2023; Cen et al., 2024; Wang et al., 2023c; 2022; Zhong et al., 2022), where they study the theoretical convergence of proposed algorithms to the optimal policy in RL and RLHF settings. Unlike these prior approaches, our work establishes global convergence guarantees for our proposed method in the goal-conditions RLHF setting without intro- ducing unrealistic assumptions such as linear reward class or complicated optimization within data-dependent level-sets.
Conditional LLM Fine-Tuning. Conditioning LLMs dur- ing training has proven effective for aligning responses with specific human objectives. SteerLM (Dong et al., 2023b; Wang et al., 2023b) extends SFT by conditioning the LLM on the multi-dimensions annotated attributes in data, such as humor and toxicity, in order to steer model responses with user customizability. Directional Preference Alignment (DPA) (Wang et al., 2024a) proposed a variant of rejection
5

---

Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
Modal
70
95
10
04000-70
0me03-10 TBP Land Lano-rase 145F
Gorma7400 40.0
3PR3
SPP3
(a) AlpacaEval 2.0 results. Left: LengthhControlled (I.C) win rates. Right: Win rates.
Nodel
Dase Model
UtroFodback
Rewond-- Awg.ondddegdedddddd URd Fetetes
Reward Augrented URo eechhrt
10
(b) MT-Bench average score.
(c) Arena-Hard-Auto score.
Figure 2. Instruction-followinng performance of the hase models, the models trained with DPO on UltraFeedhack, and the models trained with DPO on reward-augmented UltraFeedback on AlpacaEval 2.0, MT-Bench, and Arena-Hard Auto benchmarks. Our method demonstrates considerable improvements consistently across all benchmarks. The complete table is deferred to Appendix B.2.
6. Experiments
Across all cnstructiion-ollowwiii benchmarks, observe that LLMs fine-tuned with DPO the proposed reward- augmented data consistently outperform both their base models and those fine-tuned using DPO on the original UltraFeedback dataset by a considerable margin. Notably, direct alignment with the original preference data can some- times degrade the performance base models on specific benchmarks, such Arena-Hard-Auto, which involves more complex reasoning tasks. In contrast, alignment using the reward-augmented data consistently yields superior re- sults not only due to the improved style format gained from performing DPO on UltraFeedback.
6.1. Reward-Augmenteed Data Boosts DPO Performance
We begin by conducting experiments to demonstrate that applying our method on DPO to fixed offline preference datasets leads consistent performance improvements. Our setups are listed as follows.
Setup. We adopt the UttraFeedback (Cui et al., 2023) preference dataset for this experiment. Specifically, Ultra- Feedback contains reward values scored by GPT-4 (LLM- as-Judge), which are ranged between and 10 for each of the preference pairs. Our method constructs reward- augmented data by conditioning on these judge values. We choose to fine-tune various open-weight LLMs, including Mistral-7B-Instrueett Instruct-vO.3, Qwen2-7B Instrucct, Llama-3. I- SB-Instrect, Gemma-2-9B-B and SPPO (fine-tuned from Gemma2-9B-It The hyperparameters and prompts that we use are listed in Appendix B.
Besides, we also evaluate the models on six academic multi- choice question-anss answering benchmarks, including GSM8K (Cobbe et al., 2021), GPQA (Rein et al., 2023), MUSR (Sprague et al., 2023), TruthfulQA (Lin et al., 2021), BBH (Suzgun et al., 2022), and ARC' Challenge (Clark et al., 2018). To better reflect the capabilities of LLMs, we adopt various settings for these benchmarks, including zero-shot, few-shot, and few-shot Chain-of-Thoughh (CoT). The results are shown in Table 3.
Results. We first report the performance of the trained models on instruction-followin benchmarks that use strong LLMs such as GPT-4 as judges, including AlpacaEval 2.0 (Dubois et al., 2024), MT-Bench (Zheng et al., 2024), and Laama-Hard-Anne (Li et al., 2024b). The results are shown in Figure 2.
It be observed that performing DPO the reward- augmented preference data leads to better average academic scores for most families of models compared to models fine-tuned on the original UltraFeedbac' dataset and the
6

---

Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
GSM8K
GPQA
MUSR
TruthfulQA
BBH
ARC
Model
Average
(8-s CoT)
(0-s)
(0-s)
(0-s)
(3-s)
(25-s)
52.39
30.62
47.35
59.71
46.64
58.53
49.21
+DPO (Ultraeedbbcck)
53.22
28.94
47.35
64.74
47.46
60.32
50.34
+DPO -uggend-Augmented)
51.86
28.02
46.56
65.90
46.36
61.60
50.05
Qwen2-7B-Instruct
78.24
32.80
44.58
57.31
55.20
53.75
53.65
+DPO (Ulrraeeddacck
78.17
32.80
44.31
58.91
54.49
53.75
53.74
+DPO (Reward-Augmentedd)
81.05
32.97
45.77
57.99
54.94
54.52
54.54
Llama-3.1-8B-Inste
76.72
33.89
39.95
54.00
50.74
55.38
51.78
+DPO (UltraFeeddbckk)
78.47
33.72
43.39
56.61
51.31
57.51
53.50
+DPO (Reward-Augmented)
78.77
32.55
43.52
63.32
51.57
56.48
54.37
Gemma-2-9B-Ir
81.35
36.33
46.03
60.15
59.42
64.85
58.02
+DPO (Ultrafeedback)
83.32
34.14
46.56
65.12
59.78
66.41
59.22
+DPO )
83.62
35.74
48.15
65.27
59.82
65.87
59.75
SPPO
79.83
35,91
44.97
62.56
59.61
63.74
57.77
+DPO (UltraFeedback)
81.73
33.64
45.50
65.72
59.16
66.89
58.77
+DPO BReward-Augmentn
80.67
36.16
48.68
67.39
58.88
65.53
59.55
Table Performance comparison between the I.MSs after DPO UlraFeedback, reward-augmented UltraFeedhack, and their base models on academic multi-choice QA benchmarks standard zero-shot, few-shot, and CoT settings. Here, n-s refers to n-shot, the bold texts represent the best results in each family of models.
base models. Besides, we didn't 't observe severe alignment tax phenomenons (Askell et al., 2021; Noukhovitch et al., 2024; Li et al., 2024a) after DPO, and our method is able to improve the base models on most of the benchmarks.
perform DPO Qwen2-7B 1een2-7B-Instiin In--tustuct using the Implicit- Reward-Augmentee (IRA) UltraFeedback. The results are shown in Table 5. We observe that augmenting the data with the implicit reward from the DPO (UF) model leads to superior performance even compared to augmenting the data with reward scores from the L.M judge, i.e., DPO (RA). This result highlights that DPO does fully exploit the potential of the data. Moreover, this ablation demonstrates that our method is compatible with binarized preference datasets that only contain chosen and rejected response pairs, bypassing the need of judge models.
6.2. Ablation Studies
Our Method Improves the Utility of Preference Data.
We provide two pieces of evidence that our method can get more juice out of the preference data compared to di- rectly applying DPO. Firstly, we evaluate SPPO (Wu et al., 2024) fine-tuned with DPO on UltraFeedback (UF). The results are shown in Table 4. Since the SPPO model is al- ready trained UlrraFeedback from Gemma-2-9B-II additional round of DPO training with the same data signif- icantly degrades its performance. In contrast, performing DPO on Reward-Augmmented (RA) UltraFeedback results in substantial performance gains for SPPO, indicating that our method enhances the utility of the preference data.
LC WR
WR
MT
Arena
Qwen2-7B-It
20.93
18.22
7.90
24.3
+DPO (UF)
21.46
19.35
8.33
21.9
+DPO (RA)
31.17
27.58
8.47
30.1
+DPO (IRA)
32.61
29.15
8.49
28.3
data, ie.. DPO (IRA), relabeled with the implicit reward from the DPO model at the first round, i.c., DPO (UF), significantly improves it. Our method helps get more juice out of the binarized (i.e., without judge model rewards) preference data.
Table 5. A second round of DPO on the
LC WR
WR
MT
Arena
SPPO
55,60
49.61
8.40
47.6
+DPO (UF)
52.75
40.58
8.41
40.4
+DPO (RA)
60.97
66.41
8.73
49.0
Table SPPO can be improved with DPO by performing reward augmentation the same data.
Reward-Augmentee Data is Superior Not Just Due to Its Increased Size. In this part, we show that the success of our method is not merely due to the increased size of the training dataset. To illustrate this, we perform DPO on the dataset where reward augmentation is applied to the first half of the UltraFeedback data, which we denote as DPO (Half RA). By doing so, the reward- aararrd--gmeeeted data is the same size as the original dataset, but with only half of the prompts
The second evidence is that after DPO, the implicit reward can be used to relabel and augment the same preference data. Specifically, after training Qwen2-7B- -BB--trrruttt with DPO on UltraFeedback, we leverage the resulting model TDP to calculate the implicit reward for each prompt x and response y. i.e., Blogg ppro
x)) log Towen (1) x)). Then

---

Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
and the corresponding responses being utilized. It can be observed from Table 6 that DPO (Half RA) outperforms fine-tuning on the whole UltraFeedback (UF) by large margin and achieves comparable performance to applying reward augmentation across the entire UF dataset, which is denoted as DPO (RA).
find that our approach is generalizable to settings of multi- dimensional rewards that correspond to different attributes, such as helpfulness and truthfulness. Specifically, we fol- low the setting from last part to construct the preference dataset by applying the ArmoRM reward model on the on- policy responses generated by -aanaa---BB-nnsttuue Since ArmoRM a multi-objective model that not only gives a scalar reward value but also predicts cima-nnnterpretable fine- lineggrinedd attributes, first select 5 attributes (namely complexity, instruction following, honesty, helpfulness, and intelligence depth) that have the highest average coefficients on the UltraFeedback data. Then we relabel the data by conditioning on the 5-dim reward and follow the implemen- tation of using ArmoRM described in the last part. The resulting model achieves state-of-the-art within the Llama- 3-8B-Instruct model family, surpassing the strong baselines including SimPO (Meng et al., 2024) that is trained also on on- on-policy data ranked by ArmoRM, and OpenChat (Wang et al., 2023a) fine-tuned with Conditioned-RLPF from the same -aaaa-------nnttntt model.
LC WR
WR
MT
Arena
Qwen2-7B-It
20,93
18.22
7.90
24.3
+DPO (UF)
21.46
19.35
8.33
21.9
+DPO (RA)
31.17
27.58
8.47
30.1
+DPO (Half RA)
29.56
28.30
8.33
26.9
Gemma-2-9B-It
49.20
37.58
8.54
42.8
+DPO (UF)
50.70
35.02
8.54
35.8
+DPO (RA)
59.27
54.56
8.59
43.9
+DPO (Half RA)
53.12
43.74
8.66
41.3
Table DPO trained on only half of the data with reward augmen- tation outperforms the baseline.
Reward-Augmenter Data Mitigates the Unlearning Is- sue. We first demonstrate that DPO suffers from the lim- itation of unnecessarily unlearning high-qualiny rejected responses, as discussed in Section 3. Specifically, the test set of UltraFeeddack, we calculate the log probability of each rejected response for the Swee2--B-Innttree model, its DPO (UF) model, and our method DPO (RA). In Fig. 3, we plot the expected log probability for rejected responses
LC Win Rate
Win Rate
Avg. Len.
Ours SimPO
56.57
52.19
1840
53.70
47,50
1777
OpenChat 17.48
11.36
1362
Table 7. Our method trained with DPO achieves SOTA when con-
ditioning on 5-dim rewards.
Comparison with Conditional Fine-Tuning Baselines. We further compared with additional conditional post- training baselines on the offline UltraFeedback dataset (i.e., without on-policy data), including DPA (Wang et al., 2024a). SteerLM (Dong et al., 2023b), and (CAA (Chen et al., 2024). Since both baselines aim to optimize a user-controllable attribute-conditioned LLM that is optimal under diverse preference profiles with different coefficients of the reward's attributes, in Figure 4, we plot the win rates of these methods under varying preference profiles, such as adjusting verbosity preferences as considered in (Wang et al., 2024a). Fine-tuned from Zephyr-SFT, our method achieves the best AlpacaEval 2.0 win rate. We defer more ablations and baseline comparisons to Appendix B.3.
-100
Qnne22-BB-nstruct
+DPO (UF)
-200
+DPO (RA)
Log Probabiitt
330
400
-500
-600
-700
-800
Quality Score
Figure 3. Our method helps mitigate the unlearning issue of DPO.
whose reward scores 2 5. We find that DPO substantially decreases the log probability of these high-quality rejected responses, confirming that the unlearning issue arises in practice. In contrast, our method alleviates this issue, al- though the probability is still slightly lower than the base model, which is proven to be DPO's feature (Rafailov et al., 2024a; Zhang et al., 2024; Xu et al., 2024b).
14
DPA
FIONCA-R
HANNAAPP
12
SFT
NCA-R NCAP
DPO (UF)
Win Rates
10
(RA)
Conditioning on Multi-Attribute Rewards Enables SOTA Models. In previous parts, our method is imple- mented by conditioning on the scalar reward values given by the judge models, either LLMs or reward models. We
2000
Average Length
Figure 4. Comparisons with DPA, SteerLM, and (CnA
8

---

Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
Controllable Generation with Prompt. In Table 8, we ablate how generations differ when changing the goal re- wards in the system prompt. We observe that the AlpacaEval scores of the Biee2-7B-It+DPP (RA) model change ac- cordingly as g varies. However, using the same = 10 prompt during inference for (wen2-7B-It+-DP (UF) fails to give competitive results, indicating that our method is superior not only because of the additional system prompt.
hen, J., al. Nemotron-4 340b technical report. arXiv preprint vrXiv:2406./1704 2024.
Ahmadian, A., Cremer, C., Gallé, M., Fadace, M., Kreutzer, J., Üstün, A., and Hooker, S. Back to basics: Revisit- ing reinforce style optimization for learning from human feedback llms. arXiv preprint arXiv:2402. 14740, 2024.
Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., et al. A general language assistant as laboratory for alignment. arXiv preprini arXiv: 12..12..00011 2021.
g = 10
0 = 8
6
UF (g = 10)
LC WR
31.17
28.66
25.56
24.44
WR
27.58
25.57
18.88
20.75
Azar, M. G., Rowland, M., Piot, B., Guo, D., Calandriello, D., Valko, M., and Munos, R. A general theoretical paradigm to understand learning from human preferences. arXiv preprint arXiv:2310. 1206,, 2023.
Table 8. Performance when conditioned on different goal rewards in the inference prompt.
7. Conclusion
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training a helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint 2022a.
In this paper, we first investigate the limitations of direct alignment algorithms, which arise from focusing solely on relative preferences while neglecting the qualities of the responses and their gaps. Specifically, since many rejected responses are only slightly worse than the chosen ones, striv- ing to maximize the reparameterized reward gap will cause overfitting and unnecessarily unlearning the high-quality rejected response. Moreover, the directly aligned LLMs often struggle to differentiate between responses of varying quality, indiscriminately learning the low-quality chosen responses and failing to generalize effectively more op- timal responses that are sparse in the preference data. To resolve the above limitations, we introduce straightfor- ward solution leaunion reward-conditionee policies. By optimizing the LLM to generate responses conditioned on their qualities, it can better differentiate between quality levels and learn from the entire spectrum. Motivated by this, we propose a data relabeling method that constructs reward-augmented datasets by conditioning on the quality of responses as the goal quality. In experiments, we fine-tune various LLMs by applying DPO our reward-augmented data. The results demonstrate that our approach consistently delivers significant performance improvements across var- ious instruction- fowlwwiinnnooooiinggggooogg benchmarks and increases the average accuracy on academic benchmarks.
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKin- non, C., et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:22 2022b.
Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: I. the method of paired comparisons. Biometrika, 39(34))322 345, 1952.
Cai, T., Song, X., Jiang, J., Teng, F., Gu, J., and Zhang, G. Ulma: Unified language model alignment with demon- stration and point-wise human preference. arXiv preprint prXhv:2312.0255.5 2023.
Calandriello, D., Guo, D., Munos, R., Rowland, M., Tang, Y., Pires, B. A., Richemond, P. H., Lan, C. L., Valko, M., Liu, T., et al. Human alignment of large language models through online preference optimisation. arXiv preprint arXhv:2403.0886. 2024.
Cen, S., Mei, J., Goshvadi, K., Dai, H., Yang, T., Yang. S., Schuurmans, D., Chi, Y., and Dai, B. Valuu-incennivizer preference optimization: A unified approach to online and offline rlhf. arXiv preprint arXiv:2405. 19320, 2024.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Chen, H., He, G., Yuan, L., Cui, G., Su, H., and Zhu, J.
Noise contrastive alignment of language models with explicit rewards. arXiv preprint arXiv:2402.0555 2024.
Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. De- cision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 14:15504--1500 2021.
References
Adler, B., Agarwal, N., Aithal, A., Anh, D. H., Bhattacharya, P., Brundyn, A., Casper, J., Catanzaro, B., Clay, S., Co-

---

Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv preprint
Hoang Tran, Chris Glaze, B. H. Snorkel-misstrl--pairm dpo. 2024. URL https: https::/hugginn http:////gggggggg/eg co/ snorkelai/Snorken wasnskkee-Misttraa Mistral Swasstral-PairRae
2018.
Hong, J., Lee, N., and Thorne, J. Orpo: Monolithic pref- erence optimization without reference model. arXiv preprint arXiv:2403.07611, 2(4):5, 2024.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert. M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14,1668,, 2021.
Ivison, H., Wang, Y., Pyatkin, V., Lambert, N., Peters, M.. Dasigi, P., Jang, J., Wadden, D., Smith, N. A., Beltagy, I., et al. Camels in changing climate: Enhancing Im adaptation with tulu 2. arXiv preprint arXiv:2311.10702 2023.
Cui, G., Yuan, I.,, Ding, N., Yao, G., Zhu, W., Ni, Y., Xie, G. Liu, Z., and Sun, M. Ultrafeddbcck: Boosting lan- guage models with high-quality feedback. arXiv preprint arXiv:2310.01377 2023.
Jiang, A. Q. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, S., Casas, d. I., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., al. Mistral 7b, arXiv preprint arXiv:2310.06825, 2023a.
Dong, H., Xiong, W., Goyal, D., Zhang. Y., Chow, W., Pan, R., Diao, S., Zhang, J., Shum, K., and Zhang. T. Raft: Re- ward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.0011 2023a.
Dong, H., Xiong, W., Pang, B., Wang, H., Zhao, H., Zhou, Y., Jiag, N., Sahoo, D., Xiong, C., and Zhang, T. Rlhf workflow: From reward modeling to online rlhf. arXiv e-prints, pp. arXiv-2405, 2024.
Jiang, D., Ren, X., and Lin, B. Y. Lim-blender: Ensembling large language models with pairwise ranking and genera- tive fusion. arXiv preprint arXiv:2306.0256/ 2023b.
Jin, Y., Yang, Z., and Wang. Z. Is pessimism provably efficient for offline r!? In International Conference on Machine Learning, pp. 5084-5096. PMLR, 2021.
Dong, Y., Wang, Z., Sreedhar, M. N., Wu, X., and Kuchaiev, O. Steerim: Attribute conditioned sft as an (user-steerable) alternative to rlhf. arXiv preprint arXiv:2310.05344, 2023b.
Kim, K., Seo, A. J., Liu, H., Shin, J., and Lee, K. Margin matching preference optimization: Enhanced model alignment with granular feedback. arXiv preprint arXiv:2410.03145, 2024.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Ilama herd of models. arXiv preprint arXiv:2407.2178. 2024.
Lambert, N., Pyatkin, V., Morrison, J., Miranda, L.,, Lin, Y. Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y., et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprini S.Ki::2403.13799 2024.
Dubois, Y. Galambosi, B., Liang, P., and Hashimoto, T. B. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint urXiv:2404.04475 2024.
Li, S., Lin, R., and Pei, S. Multi-modal preference align- ment remedies regression of visual instruction tuning on language model. arXiv preprint arXiv:2402.10884, 2024a.
Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. Kto: Model alignment as prospect theoretic optimization. arXiv preprint urXiv:2402.01:33 2024.
Ghosh, D., Gupta, A., Reddy, A., Fu, J., Devin, C., Eysenbach, B., and Levine, S. Learning to reach goals via iterated supervised learning. arXiv preprint arXiv: 020088 2019.
Li. T., Chiang. W.-L., Frick, E., Dunlap, L., Wu, T., Zhu, B., Gonzalez, J. E., and Stoica, I. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406. 1939, 2024b.
Gulcehre, C., Paine, T. L., Srinivasan, S., Konyushkova, K., Weerts, L., Sharma, A., Siddhant, A., Ahern, A., Wang, M., Gu, C., et al. Reinforced self-training (rest) for language modeling. arXiv preprint 23x.v22388.08998, 2023.
Li, Z., Xu, T., Zhang, Y., Lin, Z., Yu, Y., Sun, R., and Luo, Z.-Q. Remax: simple, effective, and efficient rein- forcement learning method for aligning large language models. In Forty-first International Conference on Ma- chine Learning, 2023.
Guo, S., Zhang, B., Liu, T., Liu, T., Khalman, M., Llinares, F., Rame, A., Mesnard, T., Zhao, Y., Piot, B., et al. Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.04400 04792, 2024.
Lin, S., Hilton, J., and Evans, O. Truthfulqa: Measuring how models mimic human falschoods. arXiv preprint arXiv:2 00..0099997. 2021.
10

---

Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
Liu, Z., Lu, M., Xiong, W., Zhong, H., Hu, H., Zhang, S., Zheng, S., Yang. Z., and Wang. Z. Maximize to explore: One objective function fusing estimation, planning, and exploration. Advances in Neural Information Processing Systems, 36, 2024a.
Rosset, C., Cheng, C.-A., Mitra, A., Santacroce, M., Awadal-
lah, A., and Xie, 'T Direct nash optimization: Teaching language models to self-improve with general preferences.
arXiv preprint arXiv:2404.03715 2024.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and
Liu, Z., Lu, M., Zhang, S., Liu, B., Guo, H., Yang, Y., Blanchet, J., and Wang, Z. Provably mitigating overopti- mization in rlhf: Your sft loss is implicitly an adversarial regularizer. arXiv preprint 1nXi:22405.16433 2024b.
Klimov, O. Proximal policy optimization algorithms.
arXiv preprint 1r0iv:1707 06347, 2017.
Sprague, Z., Ye, X., Bostrom, K., Chaudhuri, S., and Durrett, G. Musr: Testing the limits of chain-of-then of--uhu with mul- tistep soft reasoning. arXiv preprint arXiv:23 10. 16049, 2023.
Lloret, S. A., Dhuliawala, S., Murugesan, K., and Sachan, M. Towards aligning language models with textual feedback. arXiv preprint arxiv:2407. 2024.
Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y.. Chung, H. W., Chowdhery, A., Le, Q. V., Chi, E. H., Zhou, D., et al. Challenging big- tasks and whether chain-of of-thought can solve them. arXiv preprint MrXiv:2210009211 2022.
Loshchilov, I. Decoupled weight decay regularization. arXiv preprint arXiv: 2017.
Meng, Y., Xia, M., and Chen, D. Simpo: Simple preference optimization with reference-free reward. arXiv preprinl 143744,405.14734, 2024.
Team, G., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju. S., Hussenot, L., Mesnard, T., Shahri- ari, B., Ramé, A., et al. Gemma 2: Improving open language models a practical size. arXiv preprint arXiv:2408.00//66 2024.
Mitchell, note on dpo with noisy preferences rela- tionship to ipo, 2023.
Noukhovitch, M., Lavoie, S., Strub, F., and Courville, A. C. Language model alignment with elastic reset. Advances in Neural Information Processing Systems, 36, 2024.
Tunstall, L., Beeching, E., Lambert, N., Rajani, N. Huang, S., Rasul, K., Rush, A. M., and Wolf, The alignment handbook. https://githin com/ huggingface/ iigiiigtttnnnbbonee 2023a.
Ouyang. L., Wu, J., Jiang, X., Almeida, D.,. Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 15:27730-227744 2022.
Tunstall, L., Beeching, E., Lambert, N., Rajani, N., Ra- sul, K., Belkada, Y., Huang, S., von Werra, L., Fourrier, C., Habib, N., et al. Zephyr: Direct distillation of Im alignment. arXiv preprint NVV::2310.16944 2023b.
Pang, R. Y., Yuan, W., He, H., Cho, K., Sukhbaatar, S., and Weston, J. Iterative reasoning preference optimization. Advances in Neural Information Processing Systems, 37: 116617-116637, 2024.
Wang, G., Cheng, S., Zhan, X., Li, X., Song, S., and Liu, Y. Openchat: Advancing open-source language models with mixed-quality data. arXiv preprint arXiv:2309. /2335, 2023a
Park, R., Rafailov, R., Ermon, S., and Finn, C. Disentan- gling length from quality in direct preference optimiza- tion. arXiv preprint arXiv::403.J 19159, 2024.
Wang, H., Lin, Y., Xiong, W., Yang, R., Diao, S., Qiu, S., Zhao, H., and Zhang, T. Arithmetic control of Ilms for diverse user preferences: Directional preference alignment with oulti-objective rewards. arXiv preprint arXiv: 1851, 2024a.
Rafailov, R., Hejna, J., Park, R., and Finn, C. From r to q¹: Your language model is secretly a q fufunction. arXiv preprint 4rXivv2404.1 12358, 2024a.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. Er- mon, S., and Finn, C. Direct preference optimization: Your language model secretly a reward model. Ad- vances in Neural Information Processing Systems, 36, 2024b.
Wang, H., Xiong, W., Xie, T., Zhao H., and Zhang, T. Inter- pretable preferences via multi-objective reward modeling and mixture-of-experts arXiv preprint arXiv:2406.124 12845, 2024b.
Wang, Y., Ivison, H., Dasigi, P., Hessel, J., Khot, T., Chandu, K., Wadden, D., MacMillan, K., Smith, N. A., Beltagy, I., et al. How far can camels go? exploring the state of instruction tuning on open resources. Advances in Neural Information Processing Systems, 36, 2024c.
Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang. R. Y., Dirani, J., Michael, J., and Bowman, S. R. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311 12022, 2023.

---

Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
Wang, Z., Wang, J., Zhou, Q. Li, B., and Li, H. Sample-efficient reinforcement learning via conserva- tive model-based actor-critic. Proceedings of the AAAI Conference on Artificial Intelligence, 36(8): 8612-8620, Jun. 2022. doi: 0..160%/aai.v3666 20839. URL https://ojss..o. aaai. org/ index. php/AAAI hwt.///tattttteeeeeeeeeeeeeeeeee 20839.
Xu, J., Lee, A., Sukhbaatar, S., and Weston, J. Some things more cringe than others: Preference opti- mization with the pairwise cringe loss. arXiv preprint arXiv:2312.1663 2023.
Xu, S., Fu, W., Gao, J., Ye, W., Liu, W., Mei, Z., Wang. G., Yu, C., and Wu, Y. Is dpo superior ppo for Ilm alignment? a comprehensive study, arXiv preprint arXiv:2404 10719, 2024b.
Wang, Z., Dong, Y., Zeng, J., Adams, V., Sreedhar, M. N., Egert, D., Delalleau, O., Scowcroft, J. P., Kant, N., Swope, A., et al. Helpsteer: Multi-attribute helpfulness dataset for steerlm. arXiv preprint 2Xiv:2311009522 2023b.
Yang, A., Yang. B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F. al. Qwen2 technical report. arXiv preprint arXiv:2407.10677 2024.
Yang, R., Lu, Y., Li, W., Sun, H., Fang, M., Du, Y., Li, X., Han, L.,, and Zhang, C. Rethinking goal-conditioned supervised learning and its connection offline rl. arXiv preprint arXiv:2202.044:
Wang, Z., Pan, T., Zhou, Q. and Wang, J. Efficient ex- ploration in resource-restriited reinforcement learning. Proceedings of the AAAI Conference on Artificial Intel- ligence, 37 10289,,11077,,10 Jun. 2023c. doi: 10.1609/ saai.v37i8.262211 URL https: //jss/////j////////// aaai. org/ index. AAAAT httpp///aa//rttcceeeeeeeeeeeeeee
2022.
Ye, C., Xiong. W., Zhang, Y., Jiang, N., and Zhang, T. A the- oretical analysis of nash learning from human feedback under general kl-regularized preference. arXiv preprint 2401v:2402.07314, 2024.
Williams, R. J. Simple statistical gradient-following algo- rithms for connectionist reinforcement learning. Machine learning, 8:229-256, 1992.
Yuan, H., Yuan, Z., Tan, C., Wang, W., Huang, S., and Huang. F. Rrhf: Rank responses to align language models with human feedback. Advances in Neural Information Processing Systems, 36, 2024.
Wu, Y., Sun, Z., Yuan, H., Ji, K., Yang, Y., and Gu, Q- Self-play preference optimization for language model alignment. arXiv preprint arXiv:2405.00675. 2024.
Yuan, Z., Yuan, H., Li, C.,, Dong, G., Lu, K., Tan, C., Zhou, C., and Zhou, J. Scaling relationship on learning math- ematical reasoning with large language models. arXiv preprint arXiv:2308.01825, 2023.
Xie, T., Cheng, C.-A., Jiang, N., Mineiro, P., and Agarwal, A. Bellman-consistent pessimism for offline reinforce- ment learning. Advances in neural information process- ing systems, 6466683-6694, 2021.
Zhan, W., Uehara, M., Kallus, N., Lee, J. D., and Sun, W. Provable offline preference-based reinforcement learning. In The Twelfth International Conference on Learning Representations, 2023.
Xie, T., Foster, J., Krishnamurthy, A., Rosset, C., Awadal- lah, A., and Rakhlin, A. Exploratory preference optimiza- tion: Harnessing implicit qa appapoixxiotoo for sample- efficient rlhf. arXiv preprint arXivv2405.21044 2024.
Zhang, S., Yu, D., Sharma, H., Yang, Z., Wang, S., Hassan, H., and Wang. Z. Self-cxploring language models: Active preference elicitation for online alignment. arXiv preprint arXiv:2405. 19332, 2024.
Xiong, W., Dong, H., Ye, C., Zhong, H., Jiang, N., and Zhang, T. Gibbs sampling from human feedback: A prov- able kl-constrained framework for rlhf. arXiv preprint arXiv:2312.11456 2023.
Zhao, Y., Joshi, R., Liu, T., Khalman, M., Saleh, M., and Liu, J. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305. 10425, 2023.
Xiong, W., Dong, H., Ye, C., Wang, Z., Zhong, H., Ji, H., Jiang, N., and Zhang, T. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. In Forty-firs International Con- ference on Machine Learning, 2024.
Zheng, I.., Chiang, W.-L.., Sheng, Y., Zhuang, S., Wu, 7.,, Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging judgge with mt-bench and chatbot arena. Ad- vances Neural Information Processing Systems, 36, 2024.
Xu, H., Sharaf, A., Chen, Y., Tan, W., Shen, L.,, Van Durme, B., Murray, K., and Kim, Y. J. Contrastive preference optimization: Pushing the boundaries of 1lm performance in machine translation. arXiv preprint arXiv:2401.0844 2024a.
Zheng, R., Dou, S., Gao, S., Hua, Y., Shen, W., Wang, B., Liu, Y., Jin, S., Liu, Q., Zhou, Y., et al. Secrets of rlhf in large language models part i: Ppo. arXiv preprint NKV::2307.04964 2023.
12

---

Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
Zhong, H., Xiong, W. Tan, J., Wang, L., Zhang, T., Wang, Z.,, and Yang. Z. Pessimistic minimax value it- eration: Provably efficient equilibrium learning from of- fline datasets. In International Conference on Machine Learning, pp. 27117-27142. PMLR, 2022.
Zhou, D.-X. The covering number in learning theory. Jour- nal of Complexity, 18(3):39-767, 2002.
Thu, B., Frick, E., Wu, T., Zhu, H., and Jiao, J. Starling-7b: Improving IIm helpfulness and harmlessness with rlaif, November 2023a.
Zhu, B., Jiao, J., and Jordan, M. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons. arXiv preprint arXiv:2301.122 2023b,
13

---

Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
A. Theory
In this section, we present the theoretical analysis for our proposed method.
A.1. Concepts
We provide some useful concepts for the simplicity of later discussions.
Hellinger distance Hellinger (pllg) between two probability density functions p and q defined on A' is defined as
DHellinger (pll4) = 2 xEx
(Vp(x)- v dæ.
Total variation (TV) distance Drv (pllg) between two probability density functions and defined on X' is defined as
2 pp()) - g(x)(dr.
Drv(plla) = 2 ZEX
Brabback-Leibbler (KL) divergence (L(pll4) between two probability density functions and q defined X' is defined
as
p(x) ((x)
(L(()l1)
log
p(x)dx.
J
We denote (FF I| (loo) as the e-covering number (Zhou, 2002) for function class F under the infinity norm || Ilao. Widely used theoretical analysis (Liu et al., 2024b; Zhan et al., 2023), the c-covering number characterizes the complexity of the function class F.
A.2. Theoretical Formulation
Goal-conditioned preference model. Consider a language model π E A that takes the prompt x E X' as input and outputs the response y E V, where X and y are spaces of prompts and responses, respectively. Given the prompt discrete probability distribution T-- x)) Ay generated, where Ay the of discrete distributions over V. We define the goal-conditioned reward function class as R C {R(x, v. X x y G - R], where is the goal space. The goal-conditioned Bradley-Terry model (Bradley & Terry, 1952) for annotations is described as
X,
exp(R( V1.
PR(VI > yol2, z. V1. Vo. exp( yr, g)) exp ((. y0. = a (R(x, 31. g) - R(x, yo 2llx,80990 g)),
(A.I)
where a=2) = 1/(1 + exp(-2)) is the sigmoid function. For notational simplicity, we also denote that the reward is parameterized by f E O. We denote the corresponding negative likellkl function for r for a reward-augmented preference dataset D ((x', Vin vi { as
C(R, D) = -E in D [log o [[x yws g) - R(x, vt. 9))]
(A.2)
where vig is preferred to yt by the annotation given the prompt x² and goal o for any i [N]. For notational simplicity, we denote the DPO loss by
( (yy x. (er x.g) g)
क ( x. g) Feet Trer(yt
xr, g). )
CDPO(T, D)= -E log (Blog
B log
(A.3)
Performance metric. For notational simplicity in the theoretical analysis, we denote by R the ground-truth goal- conditioned reward function. The alignment target is to maximize the expected true reward R conditioned the optimal goal g" E G. Thus, define the value function of any policy
J (T)) gr
[1t (=,
(A.4)
14

---

Reward-Augmentedd Data Enhances Direct Preference Alignment of LLMs
Algorithm 1 Theoretical Version of the Reward-Augmennted DPO
1: Input: Preference dataset D {(x², yiu vf) parameters B, 17 > 0, reference policy Tref, SFT policy asn for the regularizer, and goal labeling distribution pç-
2: Initialize the reward-augmented preference dataset D Ø.
3: for 1,..., No do
4: Sample goal In. from pe(- | x², vin) and update the reward-augmented preference dataset as D DU
{(x', Viu vis g.
5: Sample goal gi from pg(- x², vf) and update the reward-augment preference dataset as D DUi((=' vi gi).
6: end for
7: Solve policy Vi by optimizing the following objective
B Be n3 (og(=o(uolz;g")) Loro(To, DD}
min AE0
(A.9)
8: Output: Policy क xg-
Here we allow the prompt distribution do(-) to be different from that of the offline dataset distribution (-((-), but assumed be known. the meanwhile, we consider the policies that share the same support as the reference policy Tyee (Xiong al., 2023), that is, take policy class II as
II π T: x A(A) Supp(T(((a,g) g)) S Supp( ((z,g) x
(A.5)
The performance gap of a learned policy w.r.t. any given optimal policy x² is measured as
Gap x ( = [[+) [(+), given any optimal policy x* E II,
(A.6)
One popular choice to define the optimal policy is to maximize the KL.regularized reward, reward, i.e.,
n argmax =
Rd (xx
( 2..9") [2.99) m| 22, (..g"))
(A.7)
for fixed Bo 0.
Theoretical version of the reward-augmente DPO. We formulate the theoretical version of the reward-augmented DPO in Algorithm 1, where we add an SFT regularizer on the empirical objective to handle the issue of distribution shift and analyze the bound on the suboptimality (Liu et al., 2024b; Cen et al., 2024). One simple choice to define SFT policy TsFt is to utilize the chosen labels in the original preference dataset D, that is,
Tift
arginxx ( (=.gg.)) [log 7 (yw |x, xx,"))] *Ell
(A.8)
In practice, the goal relabeling distribution g P- x. is set to be deterministic selection of the annotated reward of the chosen response, i.c., g (z, y) for any i E [N] and given reward model FRM- We also remark that the size of the reward-augment preference dataset D is N = 2N, where No denotes the size of the original preference dataset D.
A.3. Assumptions for Theoretical Analysis
Similar to the theoretical analyses offline RLHF (Liu et al., 2024b; Cen et al., 2024), provide the following assumptions. Assumption A.1 (True reward model). We assume that the true goal-conditioned reward model Rº E: R for, and for any R E R and (z,, v. X x A G. it holds that R(x, y, g) [ B/2, B B/2 for positive constant B > 0.
Assumption A.1 is standard in sample complexity analysis (Zhu et al., 2023b; Zhan et al., 2023; Ye et al., 2024) in RLHE. Assumption A.2 (Regularity). We assume that the reward model class R, prompt space X, and goal space & are convex and compact.
15

---

Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
Assumption A.2 plays a role in establishing the equivalence between maximin and minimax optimizations, This assumption naturally satisfied when considering linear reward function (Zhu et al., 2023b; Xiong al., 2023; Cen et al., 2024) of the form (2, y, g) pzzz y. v. 0, where &p represents known feature map. More broadly, the assumption is also met by the class of Lipschitz continuous reward models.
Asumptton A.3 (Partial coverage coefficient). Given the optimal policy x' E IT, the coverage coefficient of the population distribution AD of the reward-augnn augmented preference dataset D w.r.t. reward model class R, optimal policy n², and the SFT policy sii denoted by Cup (R; 'T", Tatt). is defined as
E En-dy du on1 was (120,9 (111119))000000000"00)))0))))))) ).00m (11,))))))))))))))))))))) (R* (x, V1, v1. R (x, (u..y"))))) (R(x, V1, V1,9") R(x, yo. g".)"))]]
sup RER
(A.10)
B (o.. [[[[z, Bw. 9) - R(z, u. 9)) - (R(x, Ye 9) - R(x, M1 ((2.)19))))
We assume that CpD (R; nt na) < +oo for the given optimal policy . E TT.
Assumption A.3 characterizes how well the dataset D covers the optimal policy x¹ and the SFT policy 7st given the optimal goal g", instead of covering all the policies in the policy class. That the reason why we call this assumption "partial coverage". Different variants of partial coverage assumptions are posed in previous literature (Liu al., 2024b; Cen et al., 2024; Zhan et al., 2023; Xic et al., 2021) that study offline RLHF and RL to characterize the distribution shift between the optimal policy and the offline dataset distribution. We remark that the quantity Cp., (R; n², Tan) is upper bounded by the density ratio |do(-) @ x² (-I.g") @ Tan (-1.,99 3//p"//)//////////////001) "":pp")""'"""l)) >l
A.4. Theoretical Results
Under assumptions introduced before, we are ready to give the theoretical result for Algorithm 1 in the following theorem. Theorem A.4 (Suboptimality of Algorithm 1). Taking the policy class 1l as (A.5), supposing that Assumptions A. A.2, and A.3 hold, and assuming that the reward model class R has finite e-covering number under |l (R, || (loc) < +00 with = (6 (06(11 + BB) N)- Setting
n (1 + eB)( -2 24 log (- 3 3=1/VN
in Algorithm 1. Then the output policy मे of Algorithm satisfies that with probability at least 1 5,
I
v6 (1+ exp(B)) ((Cuo (R; TT', (R:T') 1)c
Gap" x" (+)
N
+ Ezwdo (IL(x'-|ax (-1x, (l
(A.11)
where == Vlog (N. 1/0) with E = (6 (1 + ee) N) Here, N denotes the number of preference pairs in D, R denotes the upper bound of the reward models, and the partial coverage coefficient Cus (R; m,n) is defined in Assumption A.3.
The detailed proof provided in Appendix A.5. Theorem A.4 shows that our proposed reward-augmented DPO (Algorithm 1) can attain global convergence to the optimal policy and the suboptimality decays at the order of 11/ (N denotes the size of the reward-augmented preference dataset). Theorem A.4 provides theoretical justification for the strong empirical performance of the awwrrd-ugmmened DPO introduced in this paper. Unlike prior works on cond-condi reinforcement learning with supervised learning (Yang et al., 2022; Ghosh et al., 2019), which typically establish weaker results such as local performance improvements or the optimization of a lower bound on ((T), our analysis guarantees global convergence to the optimal policy. This distinction underscores the significance of integrating DPO-like methods with goal-conditioned approaches.
A.5. Proof of Theorem A.4
Bridge Algorithm to the maximin optimization. Motivated by Liu et al. (2024b), we transform the optimization objective in Algorithm to minimax optimization objective, and then to a maximum optimization objective, where the maximum optimization objective can be analyzed with tools in RL analysis.
16

---

Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
Define the objective function Φ(5, R) as
Φ R) 17 Bx (1(12)))) R1. v1. R(æ, vo, vo 12.9")
- DKD. ( (-12, MTTaar + CR, D).
(A.12)
First, we prove that the derived policy स from Algorithm 1 satisfies
E argmax 6 R, x) where E argmin max ((T, R)- RER
(A.13)
TEll
well
By the definition of the optimization objective ((T, R) in (A.12), we have
min max ल(,, R) min
max nell
R(x, R(a.v1. v1.9") - - BB (")ee
RER well
RER
1] Eg ponoddosoomm(((l..9)
R(x ((x,yo.9')) (R ((R,D)
(A.14)
Then, we apply the following lemma to solve the inner maximization problem in (A.14).
Lemma A.5 (Oracle optimal KL-regularized policy). Given any reward model E R, the optimal policy æR to the maximization problem
max FET
B R(z,v.9") - "
(A.15)
given by
<n(11=.g)) Zx(z,g) TT (-(r,) exp (B-1R(aa) R(z,,gg)) :9))
(A.16)
Z = - exp exp(3-1R(z R-1RR(,,u R(x,yg) dr (v) g),
and correspondingly the optimal value of (A.15) is given by (A.15) = Rz |3 log (2 ))}
Proof of Lemma A.5. See the proof in Lemma 4.2 of Liu et al. (2024b).
By Lemma A.5 and (A.14), we have
min max (( R) = min BB (og(ZR(z,g")) - wo RER Fell
R(x, you 9*) + C(R,
(A.17)
IER
From Lemma A.5, we know that given any reward model R, we can reparameterize via its corresponding optimal goal-conditioned KL-regularized policy лR (Rafailov al., 2024b), that is.
TR(-10.9) g) (-
R(x, g) = 3 log
+ B log((R((zgg))
(A.18)
Plugging (A.18) into (A.19), show that the optimization problem in Algorithm relates the minimax optimization problem ((TT, Rt):
T T x, k..g") } + (TPO(TR..))
min max ΦTT, R) = min RER Nell
n
do (2.07)
log
Tee xx, x
RER
min
1/3 Bewdo.vo (x.9))
log (T x, xr,"")) + (TR D)}
(A.19)
RER
where the first equality uses the definition of DPO loss LDoo in (A.3). Since we know that क e argmax of (f, γ) and solves the minimization problem in (A.,9), we know that के = T by Lemma A.5.
17

---

Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
Next, we show that the minimization problem Φ(T, R) be equivalently transformed into maximization problem. Specifically, we will prove that the output policy f for the Algorithm satisfies
E argmax min ((T,, It),
(A.20)
Tenl RER
which is implied by the following theorem.
Theorem A.6. For the policy class II defined (A.5) and the reward model class satisfying Assumption A.2, consider the following policy defined as
" R E argmax o R, m) where Re E argmin max Φ R). well
(A.21)
#Ell
HER
Then the policy "f also solves the following maximin optimization:
argmax min Φ( R).
(A.22)
R
xell
RER
Proof. Under Assumption A.I, know that Φ(T, R) is convex for R E R and strongly concave for II. Applying Theorem 5.6 in Liu et al. (2024b), we prove Theorem A.6.
Suboptimality Decomposition. By the definitions of the optimization objective Φ(π, R) in (A.12) and the suboptimality gap of A w.r.t. a² in (A.6), we decompose the gap as
Gap" (7)
vo you' yorr" (12,99))
[R* (3r, (.)) Be [R' (F.
-Ex-do w ~ (11 ".lin,""),voor wo (111,9*)
Rt" (x, 11. R2' (2, 3- (=' (11,,9") (m
min ( R) + RER
min (, R)
RER
Ex (oo
I1' (z, v1,9") Rx, (o,, -B (L(((-12,,, (Teer)
+ B. +B KL (
KL (0
Term (A) + Term (B) + Term (C),
(A.23)
where we abbreviate Term (A), Term (B), and Term (C) as follows
Term (A) -7 - min ((,, R)
HER
Epwdo e1 non (-12,0 3wo (a) R² (x,81,9") R (2, - 3.
(A.24)
Term (B) = 1j 1 min (1 Rt)
HER
E En # (-1gg. ) Tac R (x, R2 "") -8 (L((f(12..")) (1
(A.25)
and
Term (C) 3 Exwd, KL (-12, (l (1x,")( KL KL(#(-12, ( (=
(A.26)
In the following, we bound Term (A) and Term (B) respectively.
18

---

Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
Analysis of Term (A) in (A.23). Note that
Term (A)
E wn (oo
R' (x,,,,,,9") v1, - (x, ve. B- KLL("" ((ee
min ( R)
RER
< By yo in ~xx "1 hoorta (mgg*) R² (2,81.99)) - R (2, 3 (= (1
min ((T", R)
RER
Rag 21 wet (:12.9)) wo0 (a ( (1,,0,1),")) (x, (R(2,319)) R(x,20099
max RER
- ((R.
(A.27)
where the inequality follows the fact that स solves the maxmin optimization problem in (A.20).
Analysis of Term (B) in (A.23). Note that
Term (B)
1
min ( R) RER
Ex * (.00 (e.0*)
R² (2, (x, (x,,0,0 (7, (1
E Bo ~ ~ "00 (x)g")
R" (2, R² (2,2000.g") 3 KL [r. (
n C((",
En (x,, ).00 (2,81.g") - R² (z. g¹) - B KL (
= = L(R"D))
(A.28)
where the inequality uses the fact that R E R by Assumption A. and the definition of the optimization objective in (A.12).
Concluding the remaining proof. Combining (A.23), (A.27). and (A.28), we have
Gap (7)
Term (A) + Term (B) + Term (C)
Bo xx (1100"))) (R (I, (1.9") R RR (x, (R(x, - R(x, vo. g o*))
RER
vo~ma( (x.g") 1
8 Eg (-( (-12, (-12, KL (L(((-10, [[ ((,g" o¹))]
(A.29)
Next, we upper bound the right-hand side of (A.29) by relating the negative log- likellkld loss difference term to the reward difference term. Recall the definition of the goal-conditionss preference model r in (A.1). Applying Lemma A.7 to give an upper bound of the difference of the negative logelikelihood loss and setting probability least 1 and for any reward model R e R that
(1 + ee) -N-1, it holds with
L(R' D) - C(R,D)
< -2 E No. ~ap Ditetinger (Pre (-1z, Un, ( Un No. 9))
+ 3 log
NN, || )l
(A.30)
N
5
19

---

Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
where No. (R, oloo)) denotes the s-covering number (Zhou, 2002) of the reward model class R. By the relationship between the Hellinger distance and TV distance, we have
Hetlinger (PR- (-1r, yı, yo. 9) [P Vr, No 9)) Diy (PR. (+k,, yr, to 30. yo. g));
By the definition of the goal-conditioned preference model PR in (A.1), have
Drv (1 R-
(-12, 91, yo. 9) | 11, vo.g))
1
R (2,
to. v0. - o of [R(x (1,,,,1199)) R(x, yo, go.g"")) R (2.
2
( (2, (2, (I, - o [R(x, y0. vo.9") R(a, ,1,
+ 2
o R2
(2,"), (2,
R (x, g)) [R(x, vn.9") R(x, yo. 9 ,
(A.31)
where the second equality uses the fact that (-2) = 1- (z)). Applying Lemma A.8 and the condition that R(x, v. BB/ B/2] for any (z, y. g) E & x A Ax7 R G in Assumption A.1, we have
yo. (o, o (R((z.01.') v1.9") R(x, R(2,v0.9")) 30)0")) R o (I, 9²)
(x,",, Rd (x,, (R(x,,n.9")) - R(a,, R(x,No.9")))
(A.32)
where = 1/(1 + exp(pp exp( Therefore, we bound the left-hand side of (A.33)
L(R,DD) DD L(
I =2xx (x vo. | (R* R (z. g)) (R(z, y1. g) R(x, yo 9))
3 N
log Ne, (1axe) (1(...1)11c)))
(A.33)
8
Meanwhile, the reward difference term in (A.29), which is evaluated responses sampled from x² and vaa can be related to the reward difference evaluated the data distribution (bD via Assumption A.3 follows,
((lr,9)) ).v0 walual (10* (1 (x,2i,9")) (x, g")) (R(2,, v1,g") R(a, (o.
Ez was
S CCn (R; x,, naat) / E (R. R (x, (R(x, (1.9) R(x, vo- 9)) /].
(A.34)
Combining (A.33), (A.34), and (A.29) and defining
(1.10,001 - R (x, (R(a,, v1.9) R(x, uo 9)) 9)))
A R
(A.35)
we obtain
Gap
r"
(7)
max RER
Cuo (Ri (Ta) An 2n kx2 122.2221 +
3
log (N:(R. ll
(1oo)
1/
8
+ 3 Eawddo [KT.(T²' ( Trett (-22) (1xrg)))))))))))))))))))))))(( - KT, gz) Tret (-12,,
(Cps (R; T5 man)) 3 nN
NN (loo)
log
8K22
8
[I[ (1121 g") || [re((1 (.,,(((lgg)))) (1)]
B Ezndo
(A.36)
where the second inequality uses the fact that az bz2 < a² (4b) for any z E R and KL-divergence non-negative. As result, selecting
(6 (1 BB) -1 and
2V6 log (R,, | 118))/8)
1
8
(A.37)
N
VN
(1 +++en (1+exp(B))**
20

---

Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
prove that with probability at least 5 that
V6 { exp(())) ((Cup (R; ,aa)) + 1)e
Gap
i
N
+ R Ro
[TT( (+la, ("
(A.38)
where we denote , = v log (N&(R, ll Combining Theorem A.6, (A.20), and (A.38), we conclude the proof of Theorem A.4.
A.6. Technical Lemmas
Lemma A.7 (Uniform concentration). Consider the negative log-likelihood loss in (A.2) and define the approximation
error as = (6- (1+c²) .N) where we assume that R(z, y. y.g) E [-B/2, B/2] for any (R, æ, v. E R x X x x x G.
Suppose that the reward model class R has a finite e-covering number Ne || llso) < co. Then for any 1/e it holds with probability at least 8 that
C(R", D) - D(R,D)
(A.39)
Σ 2 B(x: Ditelinger (P (Pr- (-12, Mn, yo ()((PP((Ieee) U1 Vo. 9))
3
Ne Ne, )lx))
+ N
log
(A.40)
5
Proof. See the proof of Lemma D.1 in Liu et al. (2024b), where the fact that (x, g) follows fixed distribution. Lemma A.8 (Difference of Sigmoid functions). For any real numbers E [-B/2, B/2], it holds that
|21 22| < (0(2)) (=2)) S 21 #2/
(A.41)
where the constant 1/(1 (xxp()))
Proof. See the proof Lemma D.2 in Liu al. (2024b).
B. Experiment Details
B.1. Setup
We use the following prompt during training. Here, the reward values are the quality scores given by the judge models that exist in the preference dataset. The prompt is set as the system prompt whenever the L.MM supports, such as Qwen2-7B- Instruct and Llama-3. I-8B-Instruct and it prefixed before the original prompt when the LLM doesn't 't support system prompting, such Mistral-7B D--nnttrccttt.t and Gemma-2-99-It.
You are assistant that generates responses for the instruction while implicitly achieving the following target score (on scale of 1-10, where 1 is lowest and 10 is highest):
Overall score: (reward_value).
At inference time, we use almost the same prompt, except that the goal score is the highest one, i.e., the overall score is 10.
You are an assistant that generates responses for the instruction while implicitly achieving the following target score (on a scale of 1-10, where 1 is lowest and 10 is highest):
Overall score: 10.
21

---

Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
In our experiments using UltraFeedback, we directly leverage the LLM-as-Judge scores provided by GPT-4 in the dataset, which range from 1 to 10. For our method that applied to on-policy data ranked by external reward models, including PairRM and ArmoRM, we apply linear transformations to normalize the resulting reward scores, ensuring they are scaled within the same to 10 range.
For hyperparameters, we tune the KL regularization coefficient 3 within [0.001, 0.11,0.1] and batch size within [64, 128, 256). We find that = 0.01 and 256 batch size yield the overall best performance for DPO across mod- els. Our method uses the same hyperparameters as DPO. Besides, we adopt the AdamW optimizer (Loshchilov, 2017), with learning rate of 5e 7 UltraFeedback, well Llama-3-8B 3-8B-Instruct on-policy data, both DPO and our proposed method yield improved performance when employing the conservative DPO (cDPO) technique (Mitchell, 2023). Consequently. for these models, we set the label smoothing hyperparameter from the Alignment Handbook (Tunstall et al., 2023a) to 0.3, while keeping it at O for the remaining models.
7 and warmup ratio of 0.1. Furthermore, we observe that for models such
-eees-eB- and
Gemma-2-9B-I
B.2. Full Results
In Table 9, we present the full results on instruction followwig benchmarks, which correspond to the performance illustrated in Figure 2 in the main text.
AlpacaEval 2.0
MT-Bench
Arena-Hard-aae Haaae-Haar--tttt
LC WR
WR
Avg. Len.
Avg.
1st
2nd
Score
Avg. Len.
Mistral-7B-Insttuutt-0.33
19.65
15.40
1503
7.67
8.00
7.34
17.0
494
+DPO (UltraFeedback)
18.76
16.93
1643
7.66
7.92
7.40
17.6
504
+DPO (Reward-Anie (awgard-uugmented)
25.99
28.36
2270
7.69
8.02
7.36
18.3
883
Qwen2-7B-Instrue
20.93
18.22
1788
7.90
8.23
7.56
24.3
617
+DPO (UltraFeedback)
21.46
19.35
1797
8.33
8.72
7.93
21.9
553
+DPO (Reward-Augmentee
31.17
27.58
1789
8.47
8.93
7.97
30.1
644
Llama-3.1-88 --nstrucctttt
24.79
27.38
2081
8.44
8.99
7.90
26.9
831
+DPO (UltraFeeedback
28.67
30.21
2053
8.47
9.01
7.93
33.0
1070
+DPO Rleward-AAugmentee
31.20
35.93
2006
8.47
8.91
8.03
34.4
824
Gemma-2-99-i.
49.20
37.58
1572
8.54
8.81
8.28
42.8
541
+DPO (UltraFeedback)
50.70
35.02
1464
8.54
8.70
8.37
35.8
456
+DPO Rleward-Augmentedd
59.27
54.56
1872
8.59
8.93
8.25
43.9
611
SPPO
55.60
49.61
1822
8,40
8.53
8.26
47.6
578
+DPO (UltraFeedback)
52.75
40.58
1544
8.41
8.78
8.04
40.4
457
+DPO (Reward-Augmmented)
60.97
66.41
2543
8.73
9.06
8.41
49.0
761
Table Results of the DPO models fine-tuned on UltraFeedback and reward-angmented UltraFeedback. We evaluate the instruction- following benchmarks including AlpacaEval 2.0, MT-Bench, and Arena-Hard-AAtto.
We also provide the full comparison results with reward augmeeted methods in Table 10, which corresponds Figure in the main text.
Zephyr-SFT
DPO
DPA
SteerLM
NCA-P
NCA-R
INCA-P
INCA-R
Ours
LC Win Rate
6.21
11.60
11.13
11,50
12.87
13.68
14.83
16.66
Win Rate
3.94
8.58
10.58
8.21
8.43
9.56
11.00
11.34
13.37
Avg. Len.
893
1240
1671
1585
1287
1364
1449
1338
1812
Table 10. Full comparison results with reward-augman augreddd methods.
B.3. More Ablations
Impact the Accuracy of AI Feedback. We consider the 19.8k prompts from a 1/3 subset of UltraFeedback following the setup from Snorkel (Hoang Tran, 2024). Five on-policy responses are first generated from Llama-3 1aama-3-8B-.. 4-8B-Inntrueh An external reward model is followed to rank these responses. We choose the best and worst responses as the chosen and
22

---

Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
rejected ones. DPO is then performed on the resulting preference pairs and the reward-augmented pairs. To ablate how method will be impacted by the accuracy of Al feedback, experiment with reward models as the ranker: PairRM (Jiang al., 2023b) and ArmoRM (Wang et al., 2024b). PairRM is small-sized (0.4B) pairwise reward model, while ArmoRM is 8B model that is state-of the-an art on RewardBench (Lambert et al., 2024) and much stronger than PairRM. We implement a variant (denoted as RA+) of the proposed reward augmentation method that only conditions on the goal rewards of the chosen responses, not those of the rejected ones, leading to same-sized datasets.
Llama-3- 8B-Instruct
PairRM (0.4B)
ArmoRM (8B)
DPO (UF)
DPO (RA+)
DPO (RA)
DPO (UF)
DPO (RA+)
LC WR
22.92
41.76
44.72
48.20
42.32
48.73
WR
23.15
45.79
44.70
53.17
42.79
45.36
Table 11. Ablation on the impact of Al feedback quality on the AlpacaEval 2.0 benchmark.
The results in Table 11 demonstrate that training augmented data conditioned on both chosen and rejected rewards is necessary for PairRM feedback, while relabeling with only the chosen rewards is sufficient to achieve strong performance for ArmoRM feedback. This aligns with our motivation outlined in Section 3.1: in noisy preference data, rejected responses may actually be of high quality, unlearning which can degrade performance. Similarly, low-quality chosen responses may also be reinforeed. This issue does not arise with strong reward models that provide accurate preferences.
More Comparisons with Baselines In Table 12, we compare our method and various baselines under the same setting on the AlpacaEval 2.0 benchmark, including SLiC-HF (Zhao et al., 2023), ORPO (Hong et al., 2024), CPO (Xu et al., 2024a), RRHF (Yuan et al., 2024), KTO (Ethayarajh et al., 2024), IPO (Azar et al., 2023), R-DPO (Park et al., 2024), and SimPO (Meng et al., 2024), where the results are from (Meng et al., 2024), as well as the RPO (Adler et al., 2024) baseline that we implement. Our method outperforms the above algorithms by considerable margin.
SLiC-HF
ORPO
CPO
RRHF
KTO
IPO
RPO
R-DPO
SimPO
Ours
LC WR
26.9
28.5
28.9
31.3
33.1
35.6
40.8
41.1
44.7
48.2
WR
27.5
27.4
32.2
28.4
31.8
35.6
41.7
37.8
40.5
53.2
Table 12. Comparison between our method, i.e., Alama-3-9B-InssruuttDDD (RA) and baselines fine-tuned the same model and
on-policy data ranked by PairRM.
In addition to the comparison with RPO from (Adler et al., 2024) in Table 12, we also report the performance of RPO fine-tuned on additional models including Qwen2-7B-Ing and Gemma2-9B-I.. As shown Table 13, the implemented RPO outperformed by our method across these models.
LC Win Rate
Win Rate
Avg. Len.
Qwen+RPO Qwen+DPO (RA)
20.29
17.34
1704
31.17
27.58
1789
Gemma+ RPO
43.14
30.93
1413
Gemma+DPO (RA)
59.27
54.56
1872
Table 13.. Comparison AlpacaEval 2.0 between method and RPO fine-tuned from the Qwen2-7B-Inntrruc and Gemma2-9B-It models, Our method consistently outperforms RPO across these fine-tuned models.
Benefits of Learning from High-Quality Rejected Responses. Using the UltraFeedback dataset, we construct two reward-augmented preference datasets by filtering out augmented data based on rejected responses with low and high reward values, respectively. Compared to our method, these datasets isolate the impact of excluding low- and high-reward rejected responses as goals. The evaluation results on AlpacaEval 2.0 are presented in Table 14. Learning from rejected high-reward samples demonstrates superior performance compared the approach that excludes these samples.
23

---

Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
+DPO (RA
+DPO (RA
Qwen2-7B-It
+DPO (UF)
+DPO (RA)
filter high)
filter low)
LC Win Rate
20.93
21.46
31.17
29.36
31.81
Win Rate
18.22
19.35
27.58
27.04
27.28
Table 14. Ablation the benefits of learning from high-quality rejected responses.
Impact of the Reward Scale. For the UltraFeedback dataset that contains response rewards in the range of 1-10, we relabel them to be in the range of 1-5 and 1-100 with linear transformation. Our method followed by DPO is then applied these different scaled datasets. The results shown in Table 15. can be observed that our method is robust to the reward scales. Since our main experiments use the default 1-10 scale as in UltraFeedback, it is likely that the performance can be further boosted, e.g., by adopting the 1-100 scale.
Qwen2-7--It 20.93
+DPO (UF)
+DPO (RA, 5)
+DPO (RA 10)
+DPO (RA 100)
LC Win Rate
21.46
29.85
31.17
31.81
Win Rate
18.22
19.35
26.12
27.58
27.96
Table 15. Ablation on the impact of the reward scale demonstrates the robustness of our method.
24


Reviews:
[{"review_id": "Hv94P14GMS", "reviewer": "Reviewer_MN74", "summary": "Direct preferential optimization (DPO) has shown great potential for finetuning language models with user preferences. However, DPO highly depends on positive vs negative samples, and therefore, if some of the relatively good samples are rejected by evaluator model, that can significantly worsen the performance of DPO. In this paper, the authors rather proposed to generate a reward value (e.g., between 1 to 10) for each sample and finetune a reward-conditioned LM for alignment. Experimental results on several academic benchmark dataset (e.g., Alpaca and MT-bench) demonstrate that such reward conditioning can improve the performance of finetuning over traditional DPO ultrafeedback method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for identifying our work's soundness and technical contributions. The valuable comments have greatly helped us improve our manuscript. Below are our specific responses to the raised questions:\n\n**Weakness 1 and Question 2: SOTA works cited in the related work but not compared, such as recent works that improve on the DPO, e.g., RPO (Adler et al., 2024), and RLAIF line of works.**\n\nWe have compared with **15** additional baselines in Figure 4 and Table 10, 12 in the Appendix, including RPO and other SOTA RLAIF methods. We listed the results below for your reference and will incorporate Tables 10 and 12 into the main body of the manuscript:\n\n|               | Zephyr-SFT |  DPO  |  DPA  | SteerLM | NCA-P | NCA-R | INCA-P | INCA-R |  Ours |\n|---------------|------------|-------|-------|---------|-------|-------|--------|--------|-------|\n| LC Win Rate   | 6.21       | 11.60 | 11.13 | -       | 11.50 | 12.87 | 13.68  | 14.83  | **16.66** |\n| Win Rate      | 3.94       | 8.58  | 10.58 | 8.21    | 8.43  | 9.56  | 11.00  | 11.34  | **13.37** |\n\n|         |Llama-3-8B-It| SLiC-HF | ORPO | CPO  | RRHF | KTO  | IPO  | RPO  | R-DPO | SimPO | **Ours** |\n|---------|---------|---------|------|------|------|------|------|------|-------|-------|----------|\n| LC WR   |22.92 | 26.9    | 28.5 | 28.9 | 31.3 | 33.1 | 35.6 | 40.8 | 41.1  | 44.7  | **48.2** |\n| WR      |23.15 | 27.5    | 27.4 | 32.2 | 28.4 | 31.8 | 35.6 | 41.7 | 37.8  | 40.5  | **53.2** |\n\nIf there are other baselines you would like us to consider, please let us know and we would be happy to include them in our comparisons.\n\n\n**Weakness 2 and Question 1: LLM evaluators are typically not that stable when generating scalar rewards.**\n\n- The LLM or reward model (RM) judges are typically trained on human preference data, and the resulting reward scores reflect the expected preferences across annotators—akin to Elo scores in Bradley-Terry models.\n- Compared to direct preference optimization methods such as DPO, our approach is more robust to the instability of scalar rewards. For instance, when only a sample-based estimate of the true preference is available or when using function approximators, it is common for high-quality rejected responses to be favored by more annotators. DPO, which attempts to maximize the reparameterized reward gap, may degrade the model in such cases. In contrast, our method is sensitive to varying response quality and effectively learns from the full spectrum of rewards.\n- We further performed an ablation study to assess the robustness of our method to different scalar reward scales generated by LLM evaluators. Using the UltraFeedback dataset, which provides reward scores in the range of 1–10, we linearly rescaled the rewards to 1–5 and 1–100. We then applied our method to these modified datasets. We observed that our method is robust to the scale of the reward scores. The results are summarized in the following table:\n\n|              | Qwen2-7B-It | +DPO (UF) | +DPO (RA, 5) | +DPO (RA 10) | **+DPO (RA 100)** |\n|--------------|-------------|-----------|--------------|---------------|-------------------|\n| LC Win Rate  | 20.93       | 21.46     | 29.85        | 31.17         | 31.81         |\n| Win Rate     | 18.22       | 19.35     | 26.12        | 27.58         | 27.96         |\n\n**Suggestion 1: Expand the method section with more proofs and guarantees on how the proposed direction can change the preference selection research direction.**\n\nWe thank the reviewer for the helpful suggestion. In the revised version, we will include a proof sketch in Section 4, incorporating the key analysis and lemmas from Appendices A.5 and A.6. Additionally, we will highlight the observed limitations of DPO—such as those illustrated in Figure 3 and similar trends reported on the O.O.D. data HelpSteer2 below—to provide a more comprehensive discussion.\n\n|Rejected score| 8| 9 |10|\n|-|-|-|-|\n| Qwen2-7B-It |-416.7 | -356.5 | -334.8 |\n| +DPO (UF)|-484.5 | -419.4 | -401.7 |\n| +DPO (RA)| -438.6 | -366.4 | -341.1 |\n\n**Question 3: How to identify the optimal goal reward value?**\n\nThe optimal goal reward depends on the value range of the judge model and its scale as presented in the training prompt. For instance, the optimal goal is $1$ when using sigmoid-based reward models, or $5$ when using LLM judges that follow evaluation criteria with a maximum score of $5$. Both types of reward values can be rescaled, e.g., via linear transformations in the training prompt, as demonstrated in Appendix B.3.\n\n---\n\nWe hope the reviewer could consider raising the score if we resolved the reviewer's concerns. We would be happy to have further discussions if the reviewer has any additional questions or comments.", "claims_and_evidence": "Performance of DPO is very sensitive towards evaluator model's preference as it is a binary classification, whereas adding scaler reward metrics can alleviate this sensitivity issue. Authors claimed that with simple tweaking of the loss function and asking the evaluator model to generate scaler goal-conditioned reward value can significantly improve the finetuning performance. Experimental results validated such claim and demonstrate superiority over ultrafeedback method.", "methods_and_evaluation_criteria": "The evaluation metrics and dataset chosen for experiments make sense to. While Ultrafeedback DPO is a good benchmark to compare with, there are also recent SOTA works cited in the related work but not been considered as benchmark methods.", "theoretical_claims": "Theorem 1 shows that the reward-condition formulation is guaranteed to reach optimality, and the detailed proofs are provided in supplement.", "experimental_designs_or_analyses": "Experiments are carefully designed. Performance is validated on well-known benchmark dataset. Several ablation studies demonstrate that reward-conditioning can help improve performance of DPO on standard Q/A tasks, mitigate the issues with unlearning, and also the proposed framework is highly dependent on goal reward setting.", "relation_to_broader_scientific_literature": "Preference optimization and alignment is an important problem and will have broader interest in the community. This work provides an alternative to DPO with reward conditioning for SFT, and therefore, will be interesting broader academic community.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "While DPO has been widely adopted, it fails in many problems due to sensitivity w.r.t. good samples being rejected. Therefore, reward-conditioned alignment propose a good alternative towards DPO. Experimental results also validated that reward-conditioning alignment performs better in many scenarios over traditional DPO. Therefore, this solution would be of interest for the broader community. \nHaving said that, I have a few concerns:\n1. LLM evaluators are typically not that stable when generating scaler rewards (as opposed to generating preferences), and thus many RLAIF works suffer. So, some analysis on how to address the errors or biases from evaluator models would have been great. \n2. The proposed method is only considered with ultrafeedback DPO, but there exists other recent works that improves on the DPO, e.g., RPO (Adler et al., 2024). Also, some comparison with RLAIF line of works is expected.", "other_comments_or_suggestions": "Majority of the space is allocated towards experimental results only. I would recommend expanding the technical method section with more proofs and guarantees on how the proposed direction can change the preference selection research direction.", "questions_for_authors": "1. How do you address the errors in generating reward scores by the evaluator model? Typically, LLMs struggle to identify right scaler rewards. \n2. If we can generate the reward values, then we can directly use RL for the alignment. However, I do see any comparison with wide variety of RLAIF works. Please explain why?\n3. How to identify the optimal goal reward value?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JVakYvVilZ", "reviewer": "Reviewer_Jgw4", "summary": "This paper studies the preference alignment problem in Large Language Models (LLMs) and proposes a Reward-Augmented Data Relabeling method to improve Direct Preference Optimization (DPO). Traditional preference learning focuses only on relative preferences while ignoring the absolute quality scores of responses. This leads to unnecessary unlearning of high-quality rejected responses, indiscriminate reinforcement of low-quality chosen responses, and poor generalization to optimal responses, which are sparse in the preference dataset.\n\nTo address these issues, the authors introduce reward-conditioned policies, leveraging reward scores from AI feedback to relabel preference data and construct a reward-augmented dataset. Experiments on various instruction-following and academic benchmarks demonstrate that training LLMs with DPO on this enhanced dataset consistently improves performance. Additionally, results confirm that this method effectively mitigates the unlearning problem of high-quality rejected responses, making preference optimization more robust and generalizable.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for identifying our work's soundness and technical contributions. Your valuable comments have greatly helped us improve our manuscript. Below are our specific responses to the raised questions:\n\n**Weakness 1 and Question 1: The authors directly apply DPO on the re-labeled dataset, rather than using the training objective introduced in the method.**\n\nWe will incorporate the following clarifications—extending lines 150–219—in the revised manuscript. The training objective introduced in our method is $$\\max_{\\pi\\in\\Pi}\\mathbb{E}_ {x,g\\sim\\mathcal{D},y\\sim \\pi(\\cdot\\mid x, g)}[R(x,y, g)-\\beta_0\\text{KL}(\\pi(\\cdot\\mid x,g)\\| \\pi_{\\text{ref}}(\\cdot\\mid x,g))],$$ where $\\Pi$ denotes the class of all goal-conditioned policies $\\pi(y\\mid x,g)$ and the relabelling distribution for $g$ is\n$$\\mathbb{P}( g= r(x,y_w) \\mid x,y_w,y_l) = P(g=(r(x,y_l) \\mid x,y_w,y_l)=1/2.$$ It has the closed-form solution as follows: $$\\pi_ R(y\\mid x, g)\\propto \\pi_{\\text{ref}}(y\\mid x,g)\\exp(\\beta_0^{-1} R(x,y,g)).$$\nUsing the reward reparameterization trick from DPO, the reward can be written as $$R(x,y,g) = \\beta_ 0 \\log\\pi_R(y\\mid x,g) -\\beta_0 \\log\\pi_{\\text{ref}}(y\\mid x,g) -Z_ R(x,g).$$ Based on the relabeling distribution, we define the augmented preference dataset $\\overline{\\mathcal{D}}$ as $$\\{(x^i,\\tilde{y} _ w^i=y_ w^i,\\tilde{y}_ l^i = y_ l^i,g^i = r(x^i,y^i_ w))\\}_ {i\\in[M]}\\cup\\{(x^i,\\tilde{y} _ w^i=y_ l^i,\\tilde{y}_ l^i = y_ w^i,g^i = r(x^i,y^i_ l))\\}_ {i\\in[M]},$$ which doubles the size of the original dataset $\\mathcal{D}$. The resulting goal-conditioned DPO objective becomes:\n$$\n\\max_{R} \\mathbb{E}_ {x,\\tilde{y}_ w, \\tilde{y}_ l, g\\sim \\overline{\\mathcal{D}}}[\\sigma(R(x,\\tilde{y}_ w,g) - R(x,\\tilde{y}_ l,g))]\n=\\max_{\\pi}\\mathbb{E}_ {x,\\tilde{y}_ w,\\tilde{y}_ l,g\\sim \\overline{\\mathcal{D}}}\\Bigl[\\sigma\\Bigl(\\beta_0 \\log\\frac{\\pi(\\tilde{y}_ w\\mid x,g)}{\\pi_{\\text{ref}}(\\tilde{y}_ w\\mid x,g)}- \\beta_0 \\log\\frac{\\pi(\\tilde{y}_ l\\mid x,g)}{\\pi_ {\\text{ref}}(\\tilde{y}_ l\\mid x,g)}\\Bigr)\\Bigr].\n$$\nThis objective directly corresponds to our implementation, which applies DPO on the relabeled dataset.\n\n**Weakness 2: The experimental setups could be more detailed.**\n\nWe have included our hyperparameters and prompts in Appendix B.1, which we list as follows. On 8 A100 GPUs, the training takes about 5-7 hours.\n| KL regularization | batch size | learning rate | warmup ratio | max prompt length | max completion length | optimizer | lr_scheduler|\n|--|-|-|-|-|-|-|-|\n| 0.01|128 | 5e-7 | 0.1 | 512 | 512 | AdamW | cosine|\n\n**Question 2: Could you provide the prompt settings for the IRA experiments?**\n\nThe prompts used in the IRA experiments are identical to those in the main experiments (as detailed in Appendix B.1). The only difference lies in the rescaling of reward values to the range $[1, 10]$ using the following linear transformation: $\\max(\\min(10*(\\text{reward} - \\text{low}) / (\\text{high} - \\text{low}), 10), 1)$, where high and low denote the maximum and minimum implicit rewards computed from a small subset of the training data.\n\n**Question 3: If possible, could you provide the code for your experiments?**\n\nWe provided the code in the anonymous link: https://anonymous.4open.science/r/anonymous-9208-id.\n\n**Question 4: Figure 3 is quite striking but does not show whether the log probability for low-quality rejected responses also increases.**\n\nWe report the log probabilities for low-quality rejected responses (with scores less than 5) in the following table:\n| |1.0|2.0|3.0 |4.0 |\n|-|-|-|-|-|\n| Qwen2-7B-Instruct | -471.5 | -283.6 | -302.7 | -334.6 |\n|+DPO (UF)| -673.8 | -488.5 | -513.9 | -517.6 |\n|+DPO (RA)| -674.2 | -485.9 | -495.6 | -420.4 |\n\nIt can be observed that the log probability of low-quality rejected responses for our method DPO (RA) has a similar scale as the vanilla DPO (UF) and is smaller than that of Qwen2-7B-Instruct.\n\n**Question 5: Relationship to traditional data-driven robust training.**\n\nWe will incorporate the following discussions to our manuscript:\\\n\"Both our method and robust training techniques are motivated by enhancing the model generalization by leveraging augmented datasets that capture alternative outcomes. However, robust training methods typically focus on generic uncertainty or perturbations, which are not involved in our method. Instead, it concerns the limitations in direct alignment algorithms, such as the unlearning issue, by explicitly conditioning on quality scores to steer the model toward learning patterns associated with varying levels of response quality. Moreover, robust methods often emphasize worst-case scenarios or boundary conditions, which can lead to conservative generalization. In contrast, our method promotes generalization toward sparse, high-quality responses.\"\n\n---\n\nWe hope the reviewer could consider raising the score if we resolved the reviewer's concerns. We would be happy to have further discussions if the reviewer has any additional questions or comments.", "claims_and_evidence": "The paper provides a detailed and convincing explanation of the issues in preference learning. In the experimental section, it conducts extensive testing across multiple models and explores various aspects, particularly dataset size impact (Half RA). The authors perform in-depth experiments on the UltraFeedback dataset, carefully controlling variables such as DPO hyperparameters and benchmark settings to ensure the reliability of results. Additionally, the paper validates the effectiveness of its method for learning from high-quality rejected responses from two perspectives: log probability of high-quality rejected responses and a test where low-quality rejected responses are filtered out before training (reported in the appendix). Overall, the paper presents a meaningful research problem, develops a model to address it, and conducts extensive experiments to verify its effectiveness.\n\nHowever, the paper also has some limitations:In Section 3 (Method), the newly proposed training objective is not fully reflected in later experiments.Instead, in Section 4 (Experiments), the authors directly apply DPO on the re-labeled dataset, rather than using the training objective introduced in Section 3.As a result, the experiments only verify that re-labeling preference data with reward scores improves generalization performance (as the paper claims, allowing the model to “learn from the full spectrum of response quality”). However, the current setup does not directly validate the theoretical claims made in the paper.\n\nIf possible, please supplement the paper with additional experiments or provide more theoretical justification to strengthen the connection between the proposed training objective in Section 3 (Method) and the experimental setup in Section 4 (Experiments).", "methods_and_evaluation_criteria": "- (1) The paper chooses well-established benchmarks (AlpacaEval 2.0, MT-Bench, TruthfulQA, GSM8K etc.), which are widely used to evaluate preference alignment in LLMs. \n- (2) The evaluation metrics (win rate, accuracy) align well with existing literature.\n- (3) Ablation studies effectively analyze different aspects of the method.", "theoretical_claims": "In this paper, a convergence guarantee Theorem (Theorem 4.1) is provided, and it is proved that the method can converge to the optimal strategy under certain assumptions, and the error bound is $O(N^{-1/2})$. The logic and derivation of the proof are correct, clear in form, and consistent with previous DPO-related work.", "experimental_designs_or_analyses": "The paper evaluates multiple LLM architectures (Mistral-7B, Qwen2-7B, Llama-3-8B, Gemma-2-9B, SPPO).\n\nIt controls variables properly, comparing DPO with and without reward augmentation.\n\nAblation studies effectively isolate the impact of reward augmentation.", "relation_to_broader_scientific_literature": "The experiments in this paper are closely related to prior work on Direct Preference Optimization (DPO), particularly the method proposed by Rafailov et al., 2024. The approach in this paper is similar to Reinforcement Learning from AI Feedback (RLAIF), such as LLM-as-Judge (Zheng et al., 2024; Cui et al., 2023) and Reward-Model-as-Judge (Adler et al., 2024; Dong et al., 2024).\n\nCompared to traditional DPO, where the model learns directly from preference data, the method in this paper reconstructs the learning objectives by conditioning on goal rewards rather than directly learning from preference pairs. This reward-conditioned approach aims to optimize the model’s ability to learn from the entire spectrum of response quality.\n\nThe key distinction from RLAIF is that while RLAIF primarily leverages AI feedback preferences, the method in this paper goes further by incorporating AI feedback scores. This allows the model to learn from the full spectrum of response quality, rather than only relying on binary preference information. In other words, the paper’s approach uses the rating scores from the AI judge to enable more nuanced learning across a wider range of response qualities.", "essential_references_not_discussed": "All essential references has been carefully addressed.", "other_strengths_and_weaknesses": "**Strengths**\n- (1) The background of the problem proposed in the paper is very worthy of investigation. The study of the full spectrum of response quality is often overlooked in preference learning, and the reward-conditioned preference learning method proposed in the paper is an effective approach to address this issue. It also holds promise for generalizing LLMs to high-quality output distributions.\n- (2) The paper includes detailed ablation experiments that demonstrate the role of high-quality outputs in model training, which is an interesting finding.\n\n**Weaknesses**\n- (1) The experimental setup section could be more detailed. (For example, the time spent on experiments, more complete parameter configurations, IRA prompt settings, etc.).", "other_comments_or_suggestions": "None", "questions_for_authors": "- (1)\tCould you provide a more complete explanation of the relationship between the theoretical model you proposed in Section 3 and the experiments in Section 4?   From my perspective, the experiments in Section 4 modify the data within the prompt to construct a dataset with reward value information, aligning the training objective with the preset reward goals. However, what is the specific connection to the goal involving R(x, y, g) introduced in Section 3? Alternatively, could you explain how the results from the experiments in Section 4 support the validity of the theory in Section 3?\n\n- (2) For the IRA experiments, since the implicit reward values provided by the model are unnormalized, how did you handle them and use them in your experiments?   It might be helpful if you could provide the prompt settings for the IRA experiments.\n\n- (3) If possible, could you provide the code for your experiments? This would help me better understand your work.\n\n- (4) The experimental results shown in Figure 3 are quite striking, but they only demonstrate that the forgetting of high-quality rejected responses is alleviated. It does not show whether the log probability for low-quality rejected responses also increases.   Could you provide more comprehensive experimental results?   This would help present more convincing findings in your work.\n\n- (5) Regarding your experiments, I believe they can be viewed as augmenting the dataset by swapping the accepted and rejected data pairs and incorporating robust learning with confidence-based parameters. Is my understanding correct? Could you elaborate on the connection and advantages of your work compared to traditional data-driven robust training methods?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tQviSdn1iD", "reviewer": "Reviewer_sRis", "summary": "The paper presents a data augmentation approach for learning on pairwise preference data that doubles the amount of data by modifying the prompt to include a description of the quality (a reward score) of the preferred response and treating each response as the chosen response. DPO is then used to update the parameters of the LLM using the newly constructed preference pairs. The authors cast the data augmentation in the framework of goal-conditioning. To motivate the data augmentation method the authors examine several limitations with DPO. The paper evaluates training with their augmented data on several LLMs, but different LLMs are used for different experiments. The method is compared to a couple of ablations or modifications of the proposed augmentation approach. The only baselines are training without the augmented data and using SPPO. On average the proposed augmentation improves performance according to AlpacaEval2.0 win rates. However, for experiments with results for multiple LLMs the magnitude of the gains is not consistent. Additionally, the gains are small for MTBench. There is a slight, potentially not meaningful, average performance gain on NLP benchmark tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**The only baselines are DPO and SPPO.**\n\nIn addition to DPO and SPPO, we compared with 15 baselines in Figure 4 and Table 12.\n\n**Gains across LLMs are inconsistent. Marginal improvements on NLP benchmarks.**\n\n- The effectiveness of our method is demonstrated on 5 LLMs. It consistently offers improvements, with most gains substantial. Since our method's hyperparameters were not extensively tuned for each model, variability in performance gains is expected. Notably, even SOTA alignment methods [1, 2] reported inconsistent improvements across different models.\n- Alignment tax [1, 3] can reduce common-sense QA performance. So we primarily evaluated on instruction-following benchmarks, where our method yields strong improvements while avoiding the alignment tax.\n\n**Claim of comparable performance between Half RA and RA.**\n\n- While we initially considered RA and Half RA comparable, each outperforming the other on at least one benchmark, we agree that this claim is not essential. We will remove it as the comparison with DPO already supports this ablation: fine-tuning on reward-augmented data yields better performance with half of the prompts and the same compute.\n- We conducted main experiments across five models and selected one or two for ablation studies. Due to resource constraints, we were unable to run all 10 ablations on all 5 models.\n- We extended this ablation to Llama, in addition to Qwen and Gemma. The results are consistent with our original findings.\n\n| | LC|WR|MTB | Arena |\n|-|-|-|-|-|\n| Llama-3.1-8B-It | 24.79 | 27.38 | 8.44  | 26.9  |\n| +DPO (UF)| 28.67 | 30.21 | 8.47  | 33.0  |\n| +DPO (RA) | 31.20 | **35.93** | 8.47  | **34.4**  |\n| +DPO (Half RA) | **31.66** | 34.37 | **8.50**  | 33.6  |\n\n**A version of Figure 2 across all LLMs is provided in Table 9 but not referenced.**\n\nFigure 2 and Table 9 present the same results, both across all LLMs.\n\n**The gains are small for MTBench.** \n\n- Since we did not perform extensive hyperparameter tuning for each benchmark, it is expected that performance gains are modest on some benchmarks.\n- On MTBench, the average gains obtained by our method are **~1.55** more than DPO. In comparison, all four models fine-tuned with SimPO [1] fail to outperform DPO on MTBench.\n\n**The analysis in Figure 3 should also be reported on O.O.D data.**\n\nWe conducted additional experiments on HelpSteer2 and had similar observations as in Fig. 3:\n| | 8| 9 |10|\n|-|-|-|-|\n| Qwen2-7B-It |-416.7 | -356.5 | -334.8 |\n| +DPO (UF)|-484.5 | -419.4 | -401.7 |\n| +DPO (RA)| -438.6 | -366.4 | -341.1 |\n\n**No evidence or citation about how DPO behaves is provided.**\n\n- In introduction, we cited [4] as the first to identify the unlearning issue of DPO, and compared with it in related work and experiments.\n- We offered empirical evidence in Fig. 3. We will also include the new ablation on HelpSteer2 at L384 and add pointers to these results in Sec. 1 and 3.\n\n**Limitations are partially addressed by PPO.**\n\nOur primary motivation is to address the limitations of direct alignment, instead of PPO that exhibit different limitations not covered here (see Sec. 5). We will include a comparison with Llama-3-PPO, which scores $21.27$ on AlpacaEval.\n\n**SFT on the augmented prompt + response is an important baseline.**\n\nPlease refer to Fig. 4, where we compared with SOTA conditional SFT baselines including DPA and SteerLM.\n\n**The strategy for selecting the hyperparameters must be detailed.**\n\nWe will add the following paragraph:\\\n\"We tune $\\beta$ within $[0.001, 0.01, 0.1]$ and batch size within $[64, 128, 256]$. We find $\\beta=0.01$ and batch size $128$ yield the overall best performance for DPO across models. Our method uses the same hyperparameters as DPO.\"\n\n**Additional experiments in Appendix are not referenced.**\n\nWe will add references to these ablation headers in the experiments and move most of them to the main body as space permits.\n\n**The error bars for AlpacaEval should be reported.**\n\nReporting error bars requires training at least three times more models, which was not feasible given our resource constraints. For similar reasons, most prior alignment works [1, 2] also report results from single runs.\n\n**Missing related works.**\n\nWe will add the following paragraph to L262:\\\n\"Pang et al. (2024) addressed DPO’s tendency to reduce the probability of the chosen response by incorporating an NLL loss. In contrast, our work focuses on a different limitation of DPO—its tendency to overlook qualitative aspects of responses—and proposes a data relabeling approach that requires no algorithm changes. It also differs from conditional sequence modeling based on SFT (Chen et al. 2021, Lloret et al. 2024). Due to the lack of textual feedback in UF, we empirically compare with the reward feedback variants of Lloret et al. (2024), including SteerLM and DPA.\"\n\n[1] Meng et al. ''SimPO.''\\\n[2] Wu et al. ''SPPO.''\\\n[3] Askell et al. ''Language Assistant as Alignment Laboratory.''\\\n[4] Adler et al. ''Nemotron-4 340B.''", "claims_and_evidence": "The authors make broad claims about the benefits of the data augmentation method. While the authors include a variety of LLMs in their experiments, very few experiments are conducted across LLMs. Therefore, it is difficult to understand how general the claims apply across LLMs and their various training and data regimes. For example, the authors claim that their Half RA configuration has comparable performance to RA on the full dataset. However, for the performance gaps vary by the LLM (and by task) with a difference of 6 and 11 for AlpacaEval LC WR and WR, which would not be considered comparable. Therefore, it is important to point the reader to results that are reported for all LLMs. A version of Figure 2, but across all LLMs is provided in Table 9 (Appendix B.2). However, this is not referenced anywhere in the main body of the paper. Such references must be made the analysis should reflect that the LLM used influences the performance gains. The analysis should also reflect that there are no meaningful performance gains on MTBench along with an explanation  or hypothesis for why this is the case when AlpacaEval2.0 performance has much higher performance gains.\n\nThe authors claim that DPO is limited in its ability to model preference data because of overfitting and unlearning of high-quality responses, because they are the rejected response in a given pair. For example, in the second paragraph of the introduction. Many claims are made about how DPO behaves, but no evidence or citation is provided. Additionally, it is not mentioned that some of these issues are tested for in the paper.", "methods_and_evaluation_criteria": "The proposed method makes sense for the problem at hand and the evaluation benchmarks are standard for alignment focused tasks.", "theoretical_claims": "I did not assess as they are all in the supplementary material.", "experimental_designs_or_analyses": "1. The authors motivate the structure of DPO and its offline nature as its key limiting factor. However, this is partially addressed by training with PPO+RM. The authors should include this as a baseline to compare against.\n2. The proposed approach is to modify the prompt to be more specific about the quality of the response. Therefore, SFT on the augmented prompt + response is an important baseline.\n3. The hyper-parameters used are listed in Appendix B.1. However, there is a lack of detail about how any parameters beyond the smooth parameter was set. The strategy for selecting the hyper-parameters MUST be detailed. \n4. DPO can be sensitive to the exact hyper-parameters and the best hyper-parameters can vary across LLMs. If the hyper-parameters are not optimized per LLM, the true differences in performance may not be accurately reflected.\n5. The experiments section should discuss the impact of LLM and compare against more baselines (these are mentioned above and in other sections). While additional experiments are included in the Appendix, they are not referenced in the main body and their conclusions are not discussed in the main body. \n6. The paper states that the reward augmentation method improves performance of any direct alignment method. This is done in the appendix, but is not referenced at all in the main body of the paper.\n7. The standard error bars for AlpacaEval2.0 win rate should be reported in Figure 2, Table 9, etc.\n8. A main claim of the paper is that the dataset augmentation helps with generalization. However, all non-benchmark experiments use the same dataset used to train the model. This means that the prompts and responses are reasonably within distribution of the training data, especially as the data comes from GPT. To fully support the generalization claims, the type of analysis reported in Figure 3 should additionally be reported on on at least one dataset from a different distribution, e.g. HH-RLHF or OpenAssistant.", "relation_to_broader_scientific_literature": "The paper has interesting learnings and take aways for different strategies to steer LLM behavior.\n\nThere are contemporary papers that have explored similar methods, e.g. \"Towards Aligning Language Models with Textual Feedback\" (EMNLP 2024), and that attempt to solve similar issues with DPO, such as \"Iterative Reasoning Preference Optimization\" which uses a NLL-loss to help with unlearning. Any final version of the paper should discuss such contemporary literature and help readers to understand how they are distinct.", "essential_references_not_discussed": "The authors position the paper relative to the literature by pointing to a difference in goals. However, a difference is goals is not a strong distinction as a method designed to address a different set of issues may additionally address the goals outlined in the paper. The authors do not go into detail how or why their method addresses is a better solution than the related work that is mentioned.\n\nThe method is similar to Decision Transformer (\"Decision Transformer: Reinforcement Learning via Sequence Modeling\") where generated behaviors are controlled by conditioning on the desired reward the generate behaviors should receive. The similarity and the relationship to Decision Transformer should be discussed.\n\nIf the paper is accepted, the camera ready version should discuss \"Towards Aligning Language Models with Textual Feedback\" (EMNLP 2024) as contemporary work.", "other_strengths_and_weaknesses": "1. The authors make many claims about the weaknesses of DPO and use these claims to motivate their data augmentation approach, however when making those claims, the authors do not present evidence to support them (neither a citation nor experimental result). It isn't until the experiments/results section that the authors begin to provide evidence for their motivating claims. The experiments should be pointed to earlier in the paper when the claims are first raised. Something as simple as \"(see Section X)\" would be sufficient.", "other_comments_or_suggestions": "All comments and suggests are included in previous sections.", "questions_for_authors": "I do not have questions for the authors.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "waA9kmIGq7", "reviewer": "Reviewer_ard6", "summary": "This paper addresses a common limitation in preference-based alignment methods, where only relative preferences are considered, while qualitative aspects of responses are overlooked. It introduces reward-conditioned LLM policies that are trained to generate responses conditioned on rewards. Leveraging a simple relabeling strategy, it constructs preference pairs based on the quality scores to train the reward-conditioned LLMs. Experimental results show that the approach consistently improves the performance of DPO across diverse models and benchmarks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Your valuable comments have greatly helped us improve our manuscript. Below are our specific responses to the raised questions:\n\n**Weakness 1: Analysis of possible optimization outcomes.**\n\n- In Section 3, we analyzed categorical LLM policies, i.e., tabular stochastic policies without function approximations. In this setting, for any prompt $x$ with the chosen and rejected responses $y_w$ and $y_l$, the optimal policy after RLHF must satisfy $\\pi^*(y_w|x)=1$ and $\\pi^*(y_l|x)=0$ in order to maximize the expected reward, i.e., $\\max_\\pi E_{y\\sim\\pi(\\cdot|x)}[r(x, y)]$, since $r(x, y_w) > r(x, y_l)$. When only slightly more annotators prefer $y_w$ over $y_l$, i.e., $r(x, y_w)$ is slightly larger than $r(x, y_l)$, $\\pi^*(y_l|x)=0$ will cause the LLMs unnecessarily unlearn the high-quality $y_l$. Similar limitations are discussed in more detail in Section 3.1.\n- For function-approximated LLM policies, the analysis becomes significantly more complex. To demonstrate that these issues persist in practice, we provided empirical evidence in Figure 3 and will incorporate the newly conducted ablation on the O.O.D. dataset HelpSteer 2:\n\n| | 8| 9 |10|\n|-|-|-|-|\n| Qwen2-7B-It |-416.7 | -356.5 | -334.8 |\n| +DPO (UF)|-484.5 | -419.4 | -401.7 |\n| +DPO (RA)| -438.6 | -366.4 | -341.1 |\n\n**Weakness 2: Missing related works including [1] and [2].**\n\n- The comparison with R-DPO [2] was provided in Table 12. We thank the reviewer for pointing out the missing related work [1], which we empirically compared in the following table. We report the MT-Bench scores of performing our method on the Llama3-SFT checkpoint:\n\n|          | Ours | MMPO | DPO  |\n|------|------|------|------|\n| MT-Bench | 7.66 | 7.58 | 7.41 |\n\n- We will add the above results to experiments and incorporate the following paragraph to related work:\\\n\"Similar to our work, [1] also investigates how preference optimization can overlook qualitative aspects of responses. However, their focus is on overfitting to preference data, and they propose incorporating quality margins into the optimization objective. In contrast, our approach does not involve algorithmic modifications, but rather directly targets the limitations identified in Section 3. Our work also differs from [2], which introduces a constraint-based regularization term specifically aimed at mitigating verbosity bias.\"\n\n\n**Weakness 3: Lack of empirical comparisons with DPO variants such as IPO and alternative approaches that use quality scores.**\n\nIn addition to DPO, we compared our method against **15** SOTA baselines. These include approaches that enhance DPO from various perspectives, such as IPO, as well as methods that incorporate quality scores during fine-tuning, including SteerLM, DPA, and MMPO. The results are presented in Figure 4 and Tables 10 and 12. For your convenience, we summarize the results below and will move Tables 10 and 12 to the main body of the manuscript.\n\n|  | Zephyr-SFT |  DPO  |  DPA  | SteerLM | NCA-P | NCA-R | INCA-P | INCA-R |  Ours |\n|-|-|-|-|-|-|-|-|-|-|\n| LC Win Rate   | 6.21       | 11.60 | 11.13 | -       | 11.50 | 12.87 | 13.68  | 14.83  | **16.66** |\n| Win Rate      | 3.94       | 8.58  | 10.58 | 8.21    | 8.43  | 9.56  | 11.00  | 11.34  | **13.37** |\n\n| |Llama-3-8B-It| SLiC-HF | ORPO | CPO  | RRHF | KTO  | IPO  | RPO  | R-DPO | SimPO | **Ours** |\n|-|-|-|-|-|-|-|-|-|-|-|-|\n| LC WR |22.92 | 26.9    | 28.5 | 28.9 | 31.3 | 33.1 | 35.6 | 40.8 | 41.1  | 44.7  | **48.2** |\n| WR  |23.15 | 27.5    | 27.4 | 32.2 | 28.4 | 31.8 | 35.6 | 41.7 | 37.8  | 40.5  | **53.2** |\n\nIf the reviewer has other baseline methods in mind, please let us know and we will be happy to include them as comparisons.\n\n**Question 1: How do the models perform on the flip task, i.e., evaluating the score of a given response similar to a type of generative verifier?**\n\nOur method, which is designed for direct preference optimization, is not directly applicable to training generative verifiers, which typically involve predicting reward tokens during training. That said, our approach is not incompatible with generative verifiers. For instance, at inference time—when the model is prompted to generate high-reward responses—the probability assigned to a given response can serve as an approximate measure of its quality. More generally, for LLM-as-a-Judge settings, reward-conditioned training can be applied to preference data that reflects the quality of judgments. By conditioning on the highest quality scores, the model exhibits its best judgment capabilities, rather than simply assigning high scores indiscriminately.\n\n---\nWe hope the reviewer could consider raising the score if we resolved the reviewer's concerns. We would be happy to have further discussions if the reviewer has any additional questions or comments.\n\n[1] Kim et al., ''Margin Matching Preference Optimization: Enhanced Model Alignment with Granular Feedback.''\\\n[2] Park et al., ''Disentangling Length from Quality in Direct Preference Optimization.''", "claims_and_evidence": "Regarding the limitations of vanilla direct alignment, the paper highlights two key issues: (a) high-quality rejected responses may be unlearned, and (b) low-quality chosen responses may be reinforced. While these are indeed potential issues, the claims are made more at a conceptual level, based on possible language policies that can be learned. However, the extent to which these issues arise depends on the optimization and the composition of the preference dataset. For instance, assuming overfitting is avoided, the likelihood of a high-quality rejected response can remain unchanged while the likelihood of the chosen response increases to drive down the loss. Similarly, both low-quality chosen and rejected responses can have their likelihoods reduced, with the rejected response being penalized more to achieve a lower loss. While precisely describing all possible optimization outcomes is challenging, this part of the paper could be made more rigorous to strengthen the claims.\n\nThe claim that reward-conditioned policies learn from the full spectrum of responses is sound, as they are explicitly conditioned on the target score of the response to be generated.", "methods_and_evaluation_criteria": "Given quality scores of responses, the paper proposes a simple labeling strategy to implement reward-conditioned alignment. Specifically, given $(x, y_w, r_w, y_l, r_l)$, it proposes to create two new preference pairs with each of the two responses as the target response to generate for the corresponding score. This is a simple, intuitive approach to utilizing quality scores of individual responses that allows direct application of exiting alignment methods such as DPO.\n\nFor evaluation, the paper considers one preference dataset (UltraFeedback) and five different language models to assess the proposed method against vanilla DPO. While the method does not always outperform DPO, in cases where it does, the performance margin is sometimes large depending on the base model. The evaluation could have been strengthened by applying the reward-conditioned alignment to alternative preference-based alignment methods such as IPO, etc.", "theoretical_claims": "Theorem 4.1 shows that under mild conditions, the proposed reward-augmented DPO converges to the optimal policy.", "experimental_designs_or_analyses": "The paper adopts UltraFeedback for preference-based alignment and assesses five open language models across six academic benchmarks. Both the number of language models and benchmarks seem sufficient for a thorough evaluation. However, the experiments could have been strengthened if preference-based alignment methods other than DPO, which have been introduced to address different limitations of DPO, are also evaluated.", "relation_to_broader_scientific_literature": "The paper proposes a simple data relabeling method that, in cases where individual quality scores are available, allows for better use of those scores to train models in a way that mitigates the limitations of vanilla DPO, such as unlearning high-quality responses and reinforcing low-quality responses. This is related to several prior works that attempt to address similar problems, such as IPO (reduces overfitting), conservative DPO (uses label smoothing), MMPO (considers relative quality differences), etc. While the paper demonstrates performance gains over vanilla DPO, additional comparisons with other closely related alignment methods could have further strengthened the study.", "essential_references_not_discussed": "Several prior works, such as [1] and [2], also study closely related problems, including incorporating qualitative aspects of responses into alignment and mitigating overfitting in DPO. These methods should ideally be evaluated alongside vanilla DPO or, at the very least, discussed.\n\n---\n[1] Kim et al., Margin Matching Preference Optimization: Enhanced Model Alignment with Granular Feedback.\\\n[2] Park et al., Disentangling Length from Quality in Direct Preference Optimization.", "other_strengths_and_weaknesses": "While the proposed method is a simple approach to utilizing quality scores of individual responses, it is unclear whether it is the most effective. For example, given individual quality scores, responses from different original preference pairs could be combined to create a significantly larger set of preference pairs. I am curious if the authors have considered or evaluated alternative approaches to using quality scores.", "other_comments_or_suggestions": "None.", "questions_for_authors": "Q. After the proposed reward-conditioned training, how do the models perform on the flip task, i.e., evaluating the score of a given response similar to a type of a generative verifier?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes to improve DPO by making the policy conditioned on the target reward score, which in particular addresses an issue where DPO may "unlearn" good responses that are "rejected" in the preference data. Extensive empirical evaluations validate the benefits of the proposed method over both vanilla DPO and relevant "reward-aware" baselines.

All reviewers are advocating for weak acceptance after the discussion period, where the authors provided several clarifications and important additional experimental results. While there could always be room for more comparisons / ablations, and the improvements might not be consistently major across all benchmarks and models, I personally consider that the authors did a thorough enough work to justify their method and demonstrate its benefits to deserve being accepted for publication. I expect this work to be of significant interest to researchers working on preference-based post-training.
Decision: Accept (poster)