Larger or Smaller Reward Margins to Select Preferences for LLM Alignment?
Authors: Kexin Huang
Junkang Wu
Ziqian Chen
Xue Wang
Jinyang Gao
Bolin Ding
Jiancan Wu
Xiangnan He
Xiang Wang
Venue: ICML 2025 poster (2025)
Keywords: LLM
RLHF
Preference Optimization
Data Quality
Primary area: deep_learning->large_language_models

Abstract: Preference learning is critical for aligning large language models (LLMs) with human values, with the quality of preference datasets playing a crucial role in this process. 
While existing metrics primarily assess data quality based on either *explicit* or *implicit* reward margins, their single-margin focus often leads to contradictory evaluations for the same data.
To address this issue, we propose a new metric of *alignment potential*, $M_{AP}$, 
which integrates both margins to quantify
the gap from the model's *current implicit* reward margin to the *target explicit* reward margin, thereby estimating the model's potential to align on the preference data.
Empirical results demonstrate that training on the data selected by $M_{AP}$ consistently enhances alignment performance, surpassing existing metrics across different base models and optimization objectives.
Furthermore, our method can be extended to self-play data generation frameworks, where we use this metric to identify high-quality data within the self-generated content by LLMs. 
Under this data generation scenario, our method surpasses current state-of-the-art
methods across various training settings and demonstrates continuous improvements
with increasing dataset size and training iterations.

Larger or Smaller Reward Margins to Select Preferences for LLM Alignment?
Kexin Huang' Junkang Wu
Ziqian Chen 2 Xue Wang² Jinyang Gao Xiang Wang'
Bolin Ding? Jiancan Wu
Xiangnan He
Abstract
Ouyang et al., 2022: Stiennon al., 2020), which involves iterative LLM fine-tuning and reward model training. To address the complexity inherent in this multi-stage training process, several offline methods - such as DPO (Rafailov et al., 2024) and SimPO (Meng et al., 2024) - have been proposed. They directly align LLMs using a static offline preference dataset {zz Ye. 31) where yo and yı denote the preferred (winning) and less preferred (losing) responses to the input prompt x, respectively. Nevertheless, due to the offline nature, these methods rely heavily on the quality of the preference dataset, which can substantially impact the alignment process (Liu et al., 2024a; Tajwar et al., 2024; Xiong et al., 2024; Yang et al., 2024).
Preference learning is critical for aligning large language models (LLMs) with human values, with the quality of preference datasets playing a crucial role in this process. While existing metrics primarily assess data quality based on either explicil or implicil reward margins, their single-margin focus often leads to contradictory evaluations for the same data. To address this issue, we propose new metric of alignment pa- tential, MAP, which integrates both margins to quantify the gap from the model's current im- plicit reward margin to the targer explicit reward margin, thereby estimating the model's poten- tial align on the preference data. Empirical results demonstrate that training the data lected by MAP consistently enhances alignment performance, surpassing existing metrics across different base models and optimization objectives. Furthermore, our method can be extended to self- play data generation frameworks, where we use this metric identify high-quality data within the self-generated content by LLMs. Under this data generation scenario, our method surpasses current mrate-ff-the-an methods across various training settings and demonstrates continuous improve- ments with increasing dataset size and training iterations.
Recent research has focused on improving LLM alignment by enhancing the quality of preference datasets, employing strategies such as selecting high-quality data for training (Khaki et al., 2024: Muldrew et al., 2024; Morimura et al., 2024) and re-weighting the loss based on data quality (Xu et al., 2023; Wang et al., 2024b; Wu et al., 2024). These approaches primarily rely on two metrics to assess the qual- ity of preference pair (I, yur y1 in the offline dataset: the explicit reward margin metric (Wang al., 2024b):
M, (x, Ywe 91) = (r(x, tho - r(r,,( 91))
based on the explicit reward r(x, -) provided by an external reward model; and the implicit reward margin metric (Yang et al., 2024; Muldrew al., 2024):
1. Introduction
Ma (2, Yars 3n) = | yar) - fo(a, 31)1.
Learning from human feedback essential for aligning large language models (LLMs) (OpenAI, 2023; Touvron al., 2023) with human preference, ensuring they help- ful, honest, and harmless (Askell al., 2021). A standard method to achieve such alignment is reinforcement learn- ing from human feedback (RLHF) (Christiano et al., 2017;
based on the implicit reward (Rafailov et al., 2024) <e(-(z) derived by the LLM policy mo(- fo 3 log mur((a) and a reference policy (-1z). While it has been demon- strated that selecting data with larger explicit reward mar- gins (Wang et al., 2024b; Ye al., 2024) or smaller implicit reward margins (Yang et al., 2024) for training can improve alignment performance, these two metrics could provide conflicting guidance for the same data. To illustrate this con- flict, we present two examples from the preference dataset used in SimPO (Figure la), where the same preference pair is deemed high quality by one metric but low quality by the other. Such inconsistency in data quality evaluations naturally raises a critical question:
'MoE Key Lab of BIPC, University of Science and Technology of China, Hefei, China 'Independent Researcher. Correspondence to: Xiangnan He <xiingnanhhe gomi, Xiang Wang <xi- angwang 223@ (223@@mmal.comm gmai..com>.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Larger or Smaller Reward Margins to Select Preferences for LLM Alignment?
Metric ① Explicit Reward margin (large better) M,(X, y)) (x( r(xX))1
Metric @ Implicit reward margin (small better) (x, X1) = (P(K.))) - (xxy)))
uniform
(ours)
Apara (%)
Arena
42
Example
Example II
"Tabasco sauce brand of hot sauce...
"Let's play a game. say sentence, then you... ... (game rules). You ready?"
+2.
Is tabasco sauce and hot sauce the same?"
44
N "No, Tabasco sauce brand name." "
reward
Large
explicit
Nu "I'm ready! Let's do it.
reward
Small explicit margin
margin
Appaca LC (%)
Arena
Implictt
Large implicit
Implliik
>/ "Sounds fun! I'm ready. 3
implicit
Y "Yes, In in"
margin
mar gin
Yo
52-
57
Large M,: high quality
Large M,;
Small M,: low quality
Small Mai high quality
46
low quality
Gommaz
Gemma2 9b
(a) Two examples illustrating the conflict between explicit and implicit reward margins
(b) Data selection performance
Figure 1. (Ia) Contradiction of existing metrics. The left example, with large explicit and implicit reward margins, is rated as "high quality" by explicit reward margin but "low quality" by implicit reward margin. The right example, where both margins are small, is rated as "low quality" by the explicit margin but "high quality" by the implicit margin. In both cases, the implicit margins are already aligned with the explicit ones, making both data rated as "low quality" by our metrie, L.e,, no need for further training. Existing metries fail on these examples because they solely focus on a single margin, neglecting the crucial gap between them. (lb) Enhanced performance by data selection. Llama-3-8b 3-bb-nnttrutttt and Gemma-2 2-9bit''s performance on Top-40% data subset selected by different data metries (uniform refers to uniformly sampling 40% data from the dataset), with our proposed metric achieving the highest results.
Is there a more reliable and theoretically grounded metric for evaluating preference data quality in alignment?
Furthermore, MAP not only supports selecting high-quality preference pairs in existing datasets, but also generalizes well to scenarios involving additional data, such as self- play alignment frameworks. In these frameworks, LLMs actively generate new training data through response sam- pling (Munos et al., 2024; Chen et al., 2024) or prompt creation (Ye et al., 2024). Our metric seamlessly integrates into such self-play frameworks, facilitating the identifica- tion of high-quality data within the intricate self geferaenee content. Consequently, it enables efficient expansion of high-quality preference data from a minimal seed dataset, thereby circumventing the constraints of static datasets and enhancing both alignment efficacy and efficiency.
In response to this question, we revisit the alignment ob- jective in RLHF, where the optimum is characterized by Rafailov et al. (2024): (z. y) = r(z, y) + e(z). with c(x) being partition term independent of y- This suggests that the implicit reward margin at optimality should be equiv- alent to explicit reward margin: 1 (2,, Vi) fg (z, u)ll = |r yw) - 1.)l. Building upon this insight, we propose the alignment potential metric, MAP, which quantifies the gap from the current model's 's implicit reward margin to the target explicit reward margin:
MAP(2. yw y1) |r(x,, Mwn) rlz, 90)
We evaluate our proposed MAP metric through extensive ex- periments within such a self-play framework. Using MAP, we first select top-rated subsets from the self-generate dataset for continued policy model training. achieving signif- icant improvements over the current state-of-the-arr results across various base models (e.g., Llama (Dubey et al., 2024), Gemma (Team et al., 2024)) and learning algorithms (e.g.. DPO (Rafailov et al., 2024), SimPO (Meng et al., 2024)). Scaling up training by enlarging the generate-thene theeseetettee--sleeteeddd dataset further yields remarkable performance gains, even surpassing the model trained on twice the size of the UI- traFeedback dataset. Additional experiments demonstrate that increasing training iterations under our metric's guid- ance leads to consistent performance gains over the models trained on the default dataset. These empirical results vali- date the practical utility of our proposed MAP metric across diverse training conditions, providing a robust solution for
(1 Nw.) Ao (2. w0
largel explicil reward margin
current implicit reward murgin
This metric estimates the model's "potential" to align on the preference data by measuring how much the model can improve its preference discrimination: as shown in Equation (1), a preference pair (I, Yw W1) with higher alignment potential score indicates a significant reward improvement of the preferred response yo over yn while the model cannot sufficiently differentiate them (i.e., similar implicit rewards), reflecting the large potential for alignment enhancement. To validate this, we conduct preliminary experiment by benchmarking models trained on the top-rated data subsets evaluated by various metrics within SimPO's preference dataset. As depicted in Figure 1b, our proposed MA metric outperforms other metrics, verifying its effectiveness in identifying high-quality data for alignment.
2

---

Larger or Smaller Reward Margins to Select Preferences for LLM Alignment?
enhancing LLM alignment through optimized preference data selection and utilization.
Simple Preference Optimization (SimPO). SimPO (Meng et al., 2024) builds on DPO by introducing two key modi- fications to achieve SOTA performance. First, it proposes a new implicit reward fiii y) 5 log (o(y))) /el to eliminate the need for a reference model, where ly/ denotes the length of response y. Such a formulation can be seen as special case of DPO's implicit reward with a uniform reference model (Wu al., 2025a). Second, it introduces classification margin term y into the objective. Consequently, the SimPO loss is formulated as follows:
2. Preliminaries
RLHF with reward models. To align a supervised fine- tuned (SFT) model TSFT with human preferences, the stan- dard RLHF pipeline begins with the reward modeling phase (Ziegler et al., 2019; Stiennon et al., 2020; Ouyang al., 2022). A preference dataset D {(r, Mw: M)} is con- structed by sampling response pairs U2) ~ (TST((|x) from prompts x, and human annotators evaluate the prefer- ence y > yr where Yw and y are the preferred (winning) and less preferred (losing) responses, respectively. Although the latent reward r" (a.y) underlying human preference is unknown, the Bradley-Terry (BT) model (Bradley & Terry, 1952) provides an effective proxy to capture human prefer- ence probabilities p" as follows:
Csmpo
-E(,, [log a (r (2r, (I, yw) 1o (s, yz)
Sim
7)]
Data quality metries. Here we study the data quality evalu- ation problem in offline preference optimization. We denote the metric by M (x, yuu yr; 9) to assess the quality of pref- erence data (2r, Yws 31) for the L.LM model zg. To evaluate different metrics, we can benchmark models trained with the top-k subset of the original preference dataset (Yang et al., 2024), denoted as Dx:= [[x, ywu I) Vu Mi; 0) 2 Th)), where тæe is the k-th highest score within the dataset. For model training, we mainly employ DPO and SimPO as optimization methods due to their verified efficacy.
p" (v1 > v2|2) (x, - (1r,
where (--) denotes the logistic function. Therefore, a reward model (RM) (I, y) can be parameterized and trained with the preference dataset via a negative log- ligelhooo loss:
(0) = -B(x.ve. (log (ro(z, yw) - (1,, sn)]] (3) With the learned reward model providing feedback, the language model ne is subsequently optimized via RL using the KL-regularized objective:
3. Alignment Potential Metric
This section introduces our alignment potential metric, de- vised to assess the quality of preference data (z, Me, 31) for aligmment. $3.1 elaborates on the formulation of alignment potential metric. $3.2 empirically evaluates the performance of various metrics by using them to select preference data for training. $3.3 provides a theoretical analysis justifying the proposed metric's ability to identify high-quality data for model alignment.
max E anD [ro(1, v)1 - (Ta (v)x)),
(4) where Teef typically instantiated via the SFT model TSFT, serves as the reference policy and B is a hyperparameter controlling the deviation from the reference model.
Direct Preference Optimization (DPO). DPO (Rafailov et al., 2024) is a widely adopted offline preference opti- mization method. Instead of learning an external reward model, DPO reparameterizes the reward function r(x, y) using a closed-for solution to the KL-regularized reward maximization problem:
3.1. Metric Formulation
Preference optimization methods (e.g.. DPO, SimPO) lever- age preference datasets to transform unaligned LMs into models that align with human values. A natural principle for evaluating the quality of preference data (x, Yor, 3t) is to measure how much the model can improve its alignment with human preferences by learning from such data. This can be essentially quantified by measuring the discrepancy between the current model and an aligned optimal state on that data. For the aligned optimal policy zÿ. Equation (5) describes an optimal condition:
7 + log Zzz),
r(I, 3) = 3 log
(5)
Tree (3) (z)
where Z(-) serves as the partition function independent of
y. Applying this reparameterization to the current L.M
DPO derives implicit reward fa y) = 3lgg (x)(()))))
==(v(z)
r't (z. y) + (z, v) + log Z(x),
By fitting it with the offline preference dataset like Equation
(3), the DPO loss is defined as:
which leads to the following equivalence for any preference data (x, Mu yt):
(!) = - E( D [log (fo yw) - 7e(x,)1)))
(6)
r (x. yw) - r rxx v1) = f, yw) (x, (8)
3

---

Larger or Smaller Reward Margins to Select Preferences for LLM Alignment?
Based on this, the discrepancy between the current model and the aligned optimal any given preference data can be quantified by the fi-distance, resulting in the M1 metric:
"Whett Zuiu soldiers called? not mentioned, output: NN onswerr Text: on (no name glven, thas "No ansiver
M+ metric: (aug=17) 6.2 5.5 11.
M > org.
"Impls." "
11:22)1112
Pa 8899
high quality
-62
-5.5 A
Mi metric: (ang-113)
M1 (Ir :wu y1: 0, rr)
(9)
No answer"
r...)::5.0
Py -34
62 -2--3 5.5 0.7
( (z, yw) - f8 (a, (z, v())
(Fo yw) fo()))-oo(00000000 ( v()) (Fo Mwo) - (z, un) given by cumment model xe
RM noises correct less preferred by RM
RM(r) prefers Incorrect yv
LLM(F prefers correct y1
Mar < avg: low quality
=I (z, Bw) - r (xr H())
Figure 2, Reward noise example from SimPO's dataset, where the less preferred response y by RM is the correct answer. M4 will mislabel this data as high-quality, which the MAP metric avoids.
given by allined optinum 6
Here the term fa (2. Bw) - (z, 31) represents the reward margin (without absolute value) between you and yt as eval- uated by the model mg. indicating the model's preference judgment between you and yr (Rafailov et al., 2024; Muldrew al., 2024). Consequenly, M1 measures the gap between preferences given by the current model Ty and the aligned optimum Th for the data (z, Bw. vt)) With this metric, a larger gap indicates greater potential for improvement (i.e., high-quality data), while smaller gap suggests less need for enhancement (i.e., luw-quality data).
in Table 12 in Appendix E). In this example, the reward model erroneously assigns higher reward to the incorrect answer. In contrast, the implicit reward given by the LLM in this example correctly identifies the preferable answer, reflecting LLM's ability to discern certain preferences, as supported by recent studies (Chen et al., 2025; Xu et al., 2024b; Kim et al., 2025).
This inspires us revisit the M_ metric. Clearly, it tends to prioritize data where r(z. Bw) r(x, v1) >> 0 (i.e., the is strongly preferred by RM) while na (z, Mw:) - V (i.e., Vu is strongly disliked by LLM). While such strong contradictions may suggest mistaken preference judgments by the LLM that necessitate alignment training. they also risk amplifying faulty preference annotations caused by the reward model noise. Considering the aforementioned example again where the reward model assigns incorrect rewards, the RM assigns r(z, yw) r(z 31) = 6.2 while the LLM evaluates fz tho) - fo 31) = 5.5, resulting in an inflated M4 score of 11.7, nearly seven times the dataset's 's average M+ score. As a result, this noisy data would be incorrectly rated as "high quality" by M+.
Nevertheless, direct computing Mj requires access to the latent, inaccessible human preference reward r*. To make this metric practically applicable, we propose using a reward quidanceaand and introduce two key adaptations for practical offline preference learning: umidirectional cali- bration and reward noise regularization.
-f))-ff(zz (2, m) << O
model r(-) for guidance
Unidirectional calibration. In typical offline preference learning scenarios (e.g.. DPO), preferences you > yt jar in the dataset are annotated by an external reward model such that r(x, V > r(z, yt), rather than being sampled from real human preference probabilities as in Equation (2). Given this deterministic nature, objectives like DPO and SimPO adopt a unidirectional update strategy to consis- tently increase the margin fa yw) - fo (2r, 31) (ef. Equa- tion (6)), as opposed to regressing the target aligned opti- mum 7* (1r, Vie) - 7r (Ir, nn) (Rosset al., 2024). In con- trast, the absolute value in M1 (I, Fw: M1 function mea- sures bidirectional gap relative to the centering optimum r* (Ix ) - r' (z, y1). To reconcile this, we apply a unidi- rectional calibration by omitting the absolute value from M1 (Ix Nw W:):
The root issue lies in the contradictions between reward model annotations and LLM judgments, which may signal reward model noise when the LLM can accurately discern certain preferences. To address this, we revise M+ by rein- troducing absolute values as a regularizer, leading to the proposed alignment potential metric:
MAP(I, Bwos yt; rr) |r(x; Mor) r(x, vc) - (x)
(wm) (w)) to (2. 31)
(11)
M+ Yw. ve 0, )*)
(10)
target explicit reward margin (by aligned optimum m,))
current implicit reward margin (by current model ma)
:=(r"(x, (I, ww) - r" (x, v1)) (fo(z, Nw) fa(a mn)).
Large M4 Vee 91) values thus highlight preference data where fo(5r, yn.) - fa y1) is small, implying need for increases consistent with offline preference methods.
This adjustment ensures that data with high MAP scores indicate substantial reward improvements, while the LLM shows uncertainty in preference determination (i.e., similar implicit rewards). It better identifies data with alignment potential and avoids unreliable contradictory annotations.
Reward noise regularization. Moreover, substituting "* with r can introduce reward model noise (Casper et al., 2023), which might lead to erroneous preference annotations (Stiennon et al., 2020; Gao et al., 2023). To illustrate this phenomenon, we present a preference data example from SimPO's dataset in Figure 2 (further details are provided
For the previous example, our alignment potential metric yields a significantly lower score MAP = 0.7, below the dataset's 's average M AP se 1.3. Consequently. this noisy preference data is rated as "low quality" by the MAP met-
4

---

Larger or Smaller Reward Margins to Select Preferences for LLM Alignment?
GPT-4 Agreement with Reward Model on Selected Data
uniform
Me
Mar (ours)
M-
61.3% 3
13.2%
25.5%
Alpaca (%)
46
Alpaca WR (%)
Arena
48
+4.7
34
MaP
72.5%
10.8%
16.7%
42
+5.0
33
+3.0
+4.3
Vo > y: (Agree)
Yu = y, (Equal)
Yw X (Disagree)
40
32
31
Figure 3. GPT-4 agreement with reward model's annotation data selected by M,, or MAP metric from SimPO's dataset. The data selected by MaP has a notably higher agree rate than M+. indicating less reward annotation noise.
36
10.0
30
10.0
29
Figure 4. Performance of Llama- Blama-3-bbinnt 3-8b-instruct model trained on preference pairs selected by different metrics.
multiple benchmarks and models.
ric, offering robust data quality evaluation against reward noise. Furthermore, we validate the regularization term in MAP by applying both M+ and MAP metrics to rate and select top-10% 10% preference pairs in SimPO's dataset and prompting GPT-4 to reassess the selected pairs against the original reward-model annotations. As shown in Figure 3, pairs selected by MA exhibit significantly higher agree- ment with GPT-4, indicating reduced annotation noise', and evidencing the regularization's 's effectiveness in MAP.
Models and datasets. We focus on preference learning for the Llama-3-8b (Dubey et al.. 2024) and Gemma-2-9h (Team et al., 2024) models. Following Meng al. (2024), we employ pre-trained instruction-tuned models as SFT models and utilize the same preference datasets in SimPO: Ilama3- www.:uuaeaeaacc-aaaaooooe armorm, for Ilama and gemma models respectively. These datasets are generated by sampling responses from LLMs for prompts in the UltraFeedback dataset (Cui et al., 2024) and then annotating preferences with the ArmoRM reward model (Wang et al., 2024a). Additionally, SimPO provides another Ilama-based dataset with the preferences annotated using the PairRM reward model (Jiang et al., 2023): Ilama3- ultrafeedback, which we include as an ablation study on different reward models.
and gemmaa-ulttafeeddbacc
Relationship with existing metrics. As shown in Equa- tion (1I), the proposed alignment potential metric can be split into two components: the target explicis reward mar- gin term M, = [r yw) - r(x, 91),, derived from aligned optimum TT. and the current implicit reward margin term M, [,, )w - (z, 31)1, given by current model me- While the MAP metric measures the gap from the current implicit reward margin to the target explicit reward margin, thereby serving as an indicator for alignment potential and data quality, each of these margins has been independently utilized as a data quality metric. Empirical studies have validated that selecting data with larger M, (Wang al., 2024b; Ye et al., 2024) or smaller Ma (Yang et al., 2024) for training improves alignment performance, which is also consistent with the trend described in MAP. Additionally, [a(()
Data quality metrics. Various data quality metrics are utilized to score preference pairs and select the top-40% subset (approximately 24k pairs) from the SimPO datasets for further training. We compare the proposed alignment potential metric MAP against the explicit reward margin M, metric, the negative? implicit reward margin - -2 met- ric, and a uniform bascline that randomly samples 40% of the data for training. Additionally, when calculating data metries before training, the model my used for calculating implicit rewards ńo is initialized from the reference model Trof: making the DPO's 's implicit reward fg Blogg constant zero To address this, we utilize the SimPO's 's im- plicit reward F5im (x, y) = 3 log me (y/xr)/by) for practical implementation across this paper. Further implementation details are provided in Appendix D.
as fz y)= 3 log (aer(ylz) contains a hyper-parameter 3, the ranking induced by MAP converges with M, as 3 - O, and with -M as 33 oc. Therefore, the MAP metric offers an integrated metric unifying both the explicit and im- plicit reward perspectives. More importantly, it underscores the potential for alignment by assessing the discrepancy be- tween target and current reward margins, rather than relying on a singular metric.
Te(yl=))
Tire T (v)a))
Training and Evaluation. We primarily adopt SimPO for preference optimization and include DPO and IPO (Azar et al., 2024) as an ablation study concerning different opti- mization objectives. After preference learning, evaluations are conducted using two widely recognized open-ended instruction-folloiin benchmarks: AlpacaEval 2 (Dubois et al., 2024) and Arena-Hard (Li et al., 2024). These bench- marks assess the model's general conversational abilities
3.2. Empirical Evaluation for Data Metrics
In this section, we conduct a preliminary experiment to evaluate different data quality metrics, using them to select preference data for alignment training. The results demon- strate the superior ability of our metric to select high-quality data, leading to improved alignment performance across
While GPT-4 might also make mistakes, alignment with its preferences suggests at least lower likelihood of anmotation noise.
Sinne Ms prioritizes data with smaller scores, add nega- tive sign for consistency with the top-k selection procedure.
5

---

Larger or Smaller Reward Margins to Select Preferences for LLM Alignment?
uniform
M,
Ma tours)
et al., 2024; Rosset et al., 2024), we reformulate the prob- lem as a contextual bandit, with context (prompt) space X, an arm (response) space Y and reward function r : X' x V + [o, 1]. A policy (LLM) π X + AV maps each context to a probability simplex over the arm space, which is parameterized with 0 X x y + R us- ing the softmax transformation (Mei et al., 2020) akin to exp(0(a.y])
Alpara LC
Alpaca WR (%)
Arena WR. (%)
71-
+2.5
+9.1
70
+7.2
68
69
68
40.0
10.3
66
67-
10.0
66
60
LLMs: To(y]z) = Ey'cy explo(z..']] The optimization objective to maximize the kl.-regularized reward, as ex- pressed in Equation (4), with the optimal condition given by: r(z,)) P-- (x. + log Z(z). To quantify the distance between 0 and 0" for theoretical analysis, we utilize the M1 (zz y. 1'10, r) metric, which measures the discrepancy between wa and optimal policy gg- on any preference pairs (x, y, v"). and define the following error function:
Figure 5. Performance of Gemma-2-99-i model trained on prefer-
ence pairs selected by different metries.
across diverse queries and are extensively used in the re- search community. Our results report the win rate for Arena- Hard and both length-controlled (LC) and raw win rates (WR) for AlpacaEval 2.
Our alignment potential metric con- sistently outperforms existing data quality metrics: As de- picted in Figures 4 and 5, while all three metrics can enhance the alignment performance, our alignment potential metric MAP achieves the highest performance across all bench- marks and models. Particularly, models trained on data selected via our metric significantly outperform the uniform baseline by 2.5 and 6.2 points on Alpaca LC, and by 4.7 and 7.1 points on Arena WR. Such notable performance enhancement underscores the effectiveness and robustness of our metric in identifying high-qualtty preference data for alignment training.
Empirical Results.
D0st V EIDN
Σ Σ MI,r.,, DEE
Optimization methods. We investigate convergence rates of different sampling strategies when optimizing the policy my using the DPO loss, whose population-form is defined as (Azar et al., 2024):
min (a.. ply > )'1x) log (fo((, fo(a, v)),, me
where s: X )22 + [0, 1] represents the sampling proba- bility, and p(u > y'(x) = a(r(z,v) v) - r(x, v()) denotes the preference given by BT model. We apply stochas- tic gradient descent to optimize the DPO loss and com- pare the performance of two samplers: a uniform sampler su (Z. y, y.y') = アلī and an adversarial sampler that se- lects preference data (z, y, y') with the highest M (z, M, v') score. Note that we directly employ the original M, metric within this theoretical setting, with no need for other cali- brations specially designed for practical offline preference learning scenarios (i.e., M+ or MAP).
Ablation Study. We conduct in-depth ablation studies to rigorously evaluate various data quality metrics under dif- ferent reward models, training objectives, and amount of selected data (cf. Appendix B.1). As illustrated in those ex- periment results, our alignment potential metric consistently surpasses existing baselines even with these variations, fur- ther validating its robustness and efficacy.
3.3. Theoretical Analysis
Convergence rates. Our theoretical analysis reveals that by selecting preference data (ær y. v/) with the largest gap, Le., the M1 (a, y: v/) score, the adversarial sampler achieves more than 2 times faster convergence than the uniform sam- pler. Formally, we state the following theorem (the proof is provided in Appendix C.I):
Building upon the empirical results showcasing enhanced alignment with data selected by our alignment potential metric, we now provide theoretical explanation for its effectiveness. With the preference learning process trans- forming unaligned LLM Ty into the aligned optimum Tg. our proposed metric, which selects preference data with a larger gap between Ta and vy. serves as an adversarial sam- pler prioritizing "hard" examples for training. Compared with uniform sampling from the dataset, such philosophy can intuitively speed up preference learning by prioritizing data with the largest optimization gaps, which requires the most update, to train on. Below, we rigorously substantiate this intuition through the lens of stochastic optimization.
Let Tu(e) and Tedu(e) ) be the (expected) it- erations required for the error Dist (f", 8*) to reduce to =Dist(0", 0²)) under uniform and adversarial sampling, re- spectively. With optimal learning rates, we have:
Theorem 3.1.
Tado (e) (.).
We also conduct numerical experiments to validate this theo- retical result. As illustrated in Figure 10, the error reduction rate with the adversarial sampler significantly outperforms
Contextual bandit setting. Following the standard theo- retical frameworks in RLHF (Rafailov et al., 2024; Munos
6

---

Larger or Smaller Reward Margins to Select Preferences for LLM Alignment?
that of the uniform sampler, thereby confirming the theoret- ical acceleration. More importantly, such an acceleration provides theoretical justification for selecting data with higher alignment potential scores for training.
offering more rigorous validation of our metric in the data generation scenarios. Additionally, this data generation and selection process can be iteratively conducted akin to eva, which we formally detail in Algorithm 1.
4. High-Quality Data Generation
Algorithm Eva with Evolve-then-Seled Pipeline
SFT model TSFT, reward model т, existing prompts X' and data quality metric M.
Input:
The proposed data metric extends beyond merely selecting high-quality data subsets in existing datasets; it also applies to additional data scenarios, i.e., to generate more high- quality data for alignment training. To facilitate this, we integrate our proposed alignment potential metric within the evolving alignment (eva) framework (Ye et al., 2024), a self-play data generation technique that employs LLM to generate new prompts x and associated response pairs (Nws 31) augment high-quality preference data.
Initialize Te, with TSFT.
for iteration t= 1,...,7 , - do
preference dataset: D + GenDataset (X, TO, (99-1177)) r1)).
evolved prompts: X" <- Evollnstruct A
additional dataset: DD' <- GenDataset (X', TBc-1:
top-k dataset selected by M metric:
Di. = {(1r, yw: v1) E DD (z;, Bwr, yt; 0t-1) 2 Tx), with Tx being the k-th largest M score in D), preference optimization
Similar to SimPO, the eva framework begins by constructing a preference dataset D = {(x, Vu, }1}} by sampling LLM's responses (y1, 12) to an existing prompts set x E X (e.g. UltraFeedback (Cui et al., 2024)), and annotating prefer- ences with a reward model such that r(z, yw) > r(z, n). Then eva augments a high-quality subset within this dataset through a seleet-then-evolve pipeline, which can be delin- cated follows:
D U D/:
0g < 0 1--0--1-62000-11 (o D()) end for
Return: optimized policy 'er
5. Experiments
In this section, we present the experimental results utilizing our alignment potential metric within the context of the eva self-play data generation framework. Following eva, we use the UltraFeedback dataset (Cui et al., 2024) as our source of initial prompts and adopt the ArmoRM (Wang al., 2024a) reward model for preference annotation. Similar to the setup in $3.2, we employ SFT models from the Llama-3-8h and Gemma-2-0 model families, apply SimPO and DPO loss for alignment training, and utilize AlpacaEval and Arena Hard benchmarks for model evaluations.
1. Initially, eva evaluates each preference pair (at, Dwn Vr) in D using the explicit reward margin metric M,. The top-k subset D is then selected based on their M, scores.
2. Subsequently, eva leverages the LLM itself to rewrite selected prompts Xg in Pk. yielding a set of newly gen- erated prompts X². Therefore a new preference dataset D' = {(2,, Vie s/)) } can be generated with similar sam- ple and annotation process as DD
To generate new prompts X², eva employs the off-the-shell instructions from EvolInstruct (Xu et al., 2024a), a popular method for autonomous prompt synthesis with LLM, which involves querying the [.LM for in-depth and br-brehdth evolving of existing prompts (cf.. Appendix D). The data generation methodology above allows eva to augment data selected by M, and generate additional preference datasets for alignment training.
5.1. Main Results
We first compare models trained with different datasets un- der the single-iteration setting. Utilizing the procedure out- lined Algorithm 1, we begin with 10k prompts from the UltraFeedback dataset generate the initial uf-10k dataset 'D. Then evolve new prompts X" and construct the addi- tional preference dataset D'. Based on this new dataset, we select the top- 10k subset using eva and our alignment po- tential metric, resulting in the D' D'va-10k and Diun. 10% datasets. To evaluate the effectiveness of evolve-selected data against existing datasets, we create an additional dataset, Dur. using supplementary 10k prompts from Ultraeeedback.
Our proposed MAP metric can be seamlessly integrated into this framework for identifying high-quality data, by replac- ing the M, metric during the prompts selection phase of eva. Nonetheless, the subsequent data evolving process does not currently involve data quality evaluation through any metric, thus failing to guarantee the quality of the evolved dataset. To address this shortcoming, we modify this self-play proce- dure by reordering the two phases of eva: We first evolve the existing prompts X' in D to derive additional dataset D' and then select a high-quality subset D'2 from the newly evolved dataset. This evolve-then-select modification ensures that the resultant dataset D' aligns with our data quality metric,
Table 1 illustrates the alignment performance across models trained on various datasets. Here, "uf-10k" refers to the initial dataset D, while "+uf-10k', "+eva-10k" and "+ours- 10k" denotes models trained on the union of D with Dur. D' l0k. and D'ars 108 respectively. As shown in the table, while all three additional datasets contribute to improved
7

---

Larger or Smaller Reward Margins to Select Preferences for LLM Alignment?
Table 1. Main results under the evolve-then-sele select self play setting. Selecting data with the alignment potential metric for training, our method achieves notable aligmment improvements, outperforning the UltraFecdback data (uf) and eurrent SOTA method (eva).
UltraFeedback data
Full data
Evolve-selected by MaP
Alpaca LC
Arena WR
75
(4.41-22.)
62.8 (+344
70-
60-
65
Llama-3-88 --ana---8b--nssie Sbinntructttttt
Gemma-2-9b-it
50-
60
Method
45
AH
AE
2.0
AH
AE
2.0
55
50
WR
I.C
WR
WR
LC
WR
10k
20k 30k
40k
50k
60k
04
10k
30k
40k
50K
60k
Dataset Size
Dataset Size
SFT
21.4
23.25
23.50
40.7
48.75
36.49
DF
23.6
29.80
28.32
46.0
54.00
46.11
Figure 6. Performance of Gemma models the dataset size in- creases.
+uf- 10k
35.0
42.67
41.67
56.4
62.56
62.17
+eva-10k 10k
31.4
39.97
40.74
56.0
63.32
62.10
+ours-10k
35.4
42.82
45.00
56.6
63.88
64.52
prompts, comprising approximately 60k data, is randomly sampled with varied sizes for training.
SimPOOu--oo
24.2
28.45
26.23
45.2
56.90
43.80
+uff-1k
28.5
35.75
32.82
53.5
66.43
64.55
Figure 6 illustrates the model performance relative to dataset size. In contrast to the gradual performance increase ob- served with the full UltraFeedback dataset, datasets selected using our metric demonstrate significantly faster improve- ments. Remarkably, training on 30k subset containing only half the data leads to superior performance compared to the full UltraFeedback dataset, improving 2.3 points in Alpaca LC and 3.4 points in Arena Hard. These substantial gains highlight the efficacy of our metric for selecting high- quality data, suggesting a potent avenue for LLM alignment with superior models even using smaller datasets.
seva-10k
32.6
41.12
39.94
61.0
66.78
66.86
+ours-10k
34.6
43.76
42.00
60.9
66.85
68.34
aligmment performance, datasets selected via our metric con- sistently achieve superior results across different base mod- els and training objectives, surpassing both existing datasets and the current SOTA method, eva. These consistent im- provements underscore the effectiveness and robustness of our alignment potential metric in discerning high-quality data for alignment training.
Furthermore, we conduct additional experiments employing pipeline, which initially
eva's original select-then-evie selects a high-quality subset D2 for data evolving into the additional dataset D (more details can be found in Ap- pendix B.3). While the resultant Dj dataset's quality may not match that of Dr due to the intricate data-evolving pro- cess, our experiments, as shown in Table 7, reveal that evolved datasets based on our metric's 's selected subsets still achieve the best overall performance. This suggests that high-quality initial subsets might positively influence the quality of the evolved data, thus enhancing the model align- ment. Specifically, we illustrate this correlation between model performance and the alignment potential score of their corresponding datasets in Figure 9 (cf. Appendix B.3). The selected subset Dr using our metric yields a higher MAP score in the resultant D',, and the model performance consistently improves with this score, affirming enhanced alignment through high-quality data generation.
5.3. Scaling with Training Iteration
Our metric is also applied to multi-iteration settings. Like the main experiment, we utilize a fixed 10k subset X from UltraFeedback to generate the initial dataset 'D for each iter- ation, and augment it with 10k additional evolve-selected data D' Dous-10kk or supplementary data from UltraFeedback Duf-1ok. for SimPO training. We report results for gener- ating Dur-iux by both fixed and additional prompts from UltraFeedback. Figure 7 illustrates the continuously in- creasing performance with more iterations when using our metric, which consistently outperforms the UltraFeedback data, even when extra data are sampled.
Fixed Uttraeedback data Alpaca LC (%)
Add uf data
Evolve sele-etllctdd by
Alpaca WR (%)
Arena WR (%)
40-
5.2. Scaling with Dataset Size
Since the main experiments involve only 20k data, we ex- tend our investigation to larger datasets assess the scala- bility of the proposed metric. Within the woolee-then-seler framework, we leverage LLMs to generate additional pref- erence data based on 20k seed prompts from UltraFeedback and select a top-k subset from the combined dataset with our alignment potential metric for SimPO training. For compar- ison, a full-size dataset built upon the entire UltraFeedback
Iteration
Iteration
Iteration
Figure 7. Perforraacc of Llama models trained under iterative preference learning setting.
6. Conclusion and Future Work
In this work, we propose the alignment potential metric, MAP, to evaluate preference data quality in alignment. By
8

---

Larger or Smaller Reward Margins to Select Preferences for LLM Alignment?
measuring the gap from the model's current implicit reward margin the target explicit reward margin, MAp quantifies the discrepancy between the current model and the aligned optimum, thereby indicating the potential for alignment enhancement. Extensive experiments validate the efficacy of MA across various training settings under offline and self-play preference learning scenarios.
Statistics, pp. 4447-4455. PMLR, 2024.
Bai, Y., Jones, A.. Ndousse, K., Askell, A.. Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training a helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint arXiv:2204.0586. 2022.
Limitations and future work. Despite the performance improvements, MAP requires tuning a parameter 3 to com- bine the explicit and implicit margins; future work could explore how to set this ratio automatically. As for the mod- els, we mainly investigate the alignment performance of instruction-tuned models, i.e., the Gemma-2-8h-in models. Although these models have been widely applied in existing works, their post-training proce- dure (SFT or RLHF) is not fully publicly disclosed. Addi- tionally, while our experiments mainly focus on the widely applied DPO and SimPO objectives, a broader investigation with alternative models and preference learning methods is crucial in future work.
Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-344 345, 1952.
Casper, S., Davies, X., Shi, C., Gilbert, T. K., Scheurer, J., Rando, J., Freedman, R., Korbak, T., Lindner, D., Freire, P., et al. Open problems and fundamental limitations of re- inforcement learning from human feedback. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. Survey Certification, Featured Certification.
Instruct and
Chen, C., Liu, Z., Du, C., Pang, T., Liu, Q. Sinha, A., Varakantham, P., and Lin, M. Bootstrapping language models with dpo implicit rewards. In The Thirteenth International Conference on Learning Representations (ICLR), 2025.
Acknowledgments
This research is supported by the National Science and Tech- nology Major Project (20232D0121102) the National Nat- ural Science Foundation of China (U24B20180, 62121002). This research was also supported by the advanced comput- ing resources provided by the Supercomputing Center of the USTC.
Chen, Z., Deng, Y., Yuan, H., Ji, K., and Gu, Q. Self- play fine-tuning converts weak language models to strong language models. In Forty-first International Conference on Machine Learning (ICML), 2024.
Christiano, F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information pro- cessing systems (NeurIPS), 30, 2017.
Impact Statement
Our work introduces a data metric that enhances the align- ment performance of large language models (LLMs) by selecting high-quality data for preference learning, poten- tially reducing misalignment risks deployed Al systems. Our approach can also reduce computational resources dur- ing training by enabling more efficient training on high- quality data subsets, making responsible AI development more accessible. While our method improves data quality and efficiency, continued monitoring remains important to ensure Al systems are fair and beneficial.
Cui, G., Yuan, L., Ding. N., Yao, G., He, B., Zhu, Ni, Y., Xie, G., Xie, R., Lin, Y., et al. Ultrafeedback: Boosting language models with scaled ai feedback. In Forty-first International Conference on Machine Learning (ICML), 2024.
Dong, H., Xiong, W., Goyal, D., Zhang, Y., Chow, W., Pan, R., Diao, S., Zhang, J., KaShun, S., and Zhang, Raft: Reward ranked finetuning for generative foundation model alignment. Transactions on Machine Learning Research, 2023.
References
Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2 (tt::2//2.000 008611, 2021.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Ilama 3 herd of models. arXiv preprint arXi::2407.2113 2024.
Azar, M. G., Guo, Z. D., Piot, B., Munos, R., Rowland, M., Valko, M., and Calandriello, D. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and
Dubois, Y., Galambosi, B., Liang, P., and Hashimoto, T. B. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.0444 2024.

---

Larger or Smaller Reward Margins to Select Preferences for LLM Alignment?
Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. Model alignment as prospect theoretic optimiza- tion. In International Conference Machine Learning (ICML), pp. 12634-12651, PMLR, 2024.
Morimura, T., Sakamoto, M., Jinnai, Y., Abe, K., and Ariu, K. Filtered direct preference optimization. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 12722-227779 2024.
Gao, L., Schulman, J., and Hilton, J. Scaling laws for reward model overoptimization. In International Conference Machine Learning (ICML), pp. 10835-10866. PMLR, 2023.
Muldrew, W., Hayes, P., Zhang, M., and Barber, Ac- tive preference learning for large language models. In Forty-first International Conference on Machine Learning (ICML), 2024.
Hong. J., Lee, N., and Thome,, J. Orpo: Monolithic prefer- ence optimization without reference model. In Proceed- ings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 11170-11188 1189,, 2024.
Munos, R., Valko, M., Calandriello, D., Azar, M. G., Row- land, M., Guo, Z. D., Tang, Y., Geist, M., Mesnard, T., Fiegel, C.,, et al. Nash learning from human feedback. In Forty-first International Conference on Machine Learning (ICML), 2024.
Jiang, D., Ren, X., and Lin, B. Y. LIm-blender: Ensembling large language models with pairwise comparison and generative fusion. In Proceedings of the 61th Annual Meeting of the Association for Computational Linguisties (AC1. 2023), 2023.
Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., et al. Webgpt: Browser- question-answeriig with hu- man feedback. arXiv preprint arXiv:2 2021.
Khaki, S., Li, J.. Ma, L., Yang, L., and Ramachandra, P. Rs-dpo: A hybrid rejection sampling and direct prefer- ence optimization method for alignment of large language models. In Findings of the Association for Computational Linguistics: NAACI 2024, pp. 1685-1680, 2024.
OpenAL. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Ouyang, L., Wu, Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., al. Training language models follow instructions with human feedback. Advances in neural information processing systems (NeurIPS), 35:27730-27744 2022.
Kim, D., Lee, K., Shin, J., and Kim, J. Spread preference annotation: Direct preference judgment for efficient Iim alignment. In The Thirteenth International Conference on Learning Representations (ICLR), 2025.
Pattnaik, P. Maheshwary, R., Ogueji, K., Yadav, V., and Madhusudhan, S. T. Enhancing alignment using cur- riculum learning & ranked preferences. In Findings of the Association for Computational Linguistics: EMNLP 2024, November 2024.
Li, T., Chiang, W.-L., Frick, E., Dunlap, L., Wu, T., Zhu, B., Gonzalez, J. E., and Stoica, From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406.1 1939, 2024.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimiza- tion: Your language model is secretly reward model. Advances in Neural Information Processing Systems (NeurIPS), 36, 2024.
Liu, T., Zhao, Y., Joshi, R., Khalman, M., Saleh, M., Liu, P. J., and Liu, J. Statistical rejection sampling improves preference optimization. In The Twelfth International Conference on Learning Representations (ICLR), 2024a.
Liu, W., Zeng, W. He, K., Jiang, Y., and He, J. What makes good data for alignment? comprehensive study of auto- matic data selection in instruction tuning. In The Twelfih International Conference on Learning Representations (ICLR), 2024b.
Rosset, C., Cheng, C.-A., Mitra, A., Santacroce, M., Awadal- lah, A., and Xie, T. Direct nash optimization: Teaching language models to self-improve with general preferences. arXiv preprint arXiv:2404.037115 2024.
Santacroce, M., Lu, Y., Yu, H., Li, Y., and Shen, Y. Efficient rlhf: Reducing the memory usage of ppo. arXiv preprint htt::2309.007... 2023.
Mci, J., Xiao, C., Szepesvari, C., and Schuurmans, D. On the global convergence rates of softmax policy gradient methods. In Proceedings of the 37th International Con- ference on Machine Learning (ICML). PMIR, 13-18 Jul 2020.
Shi, R., Zhou, R., and Du, S. The crucial role of sam- plers in online direct preference optimization. In The Thirteenth International Conference on Learning Repre- sentations (ICLR), 2025.
Meng, Y., Xia, M., and Chen, D. SimPO: Simple prefer- ence optimization with reference- reward. In The Thirty-eighth Annual Conference on Neural Information Processing Systems (NeurIPS), 2024.
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, F.
10

---

Larger or Smaller Reward Margins to Select Preferences for LLM Alignment?
Learning summarize with human feedback. Advances in Neural Information Processing Systems (NeurIPS), 33: 3008-3021, 2020.
Wu, J., Xie, Y., Yang, Z., Wu, J., Chen, J., Gao, J., Ding, B., Wang, X., and He, X. Towards robust alignment of language models: Distributionally robustifying direct preference optimization. In The Thirteenth International Conference on Learning Representations (ICLRR, 2025b.
Tajwar, F., Singh, A., Sharma, A., Rafailov, R., Schneider, J., Xie, T., Ermon, S., Finn, C., and Kumar, A. Prefer- ence fine-tuning of LLMs should leverage suboptimal, on-policy data. In Forty-first International Conference on Machine Learning (ICML), 2024.
Wu, Y., Sun, Z., Yuan, H.,, Ji, K., Yang, Y., and Gu, Q. Self-play preference optimization for language model alignment. In The Thirteenth International Conference on Learning Representations (ICLR), 2025c.
Team, G., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahri- ari, B., Ramé, A., et al. Gemma Improving open language models at practical size. arXiv preprint arXiv:2408.00116 00118. 2024.
Xiao, T., Yuan, Y., Zhu, H., Li, M., and Honavar, V. G. Cal-dpo: Calibrated direct preference optimization for language model alignment. In The Thirty-eighth Annual Conference on Neural Information Processing Systems (NeurIPS), 2024.
Touvron, H., Martin, I.., Stone, K., Albert, P., Almahairi, A., Babaci, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint ar arXiv:2307.09288 2023.
Xiong, W., Dong, H., Ye, C., Wang, Z., Zhong, H., Ji, H., Jiang, N., and Zhang, T. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. In Forty-first International Con- ference on Machine Learning (ICML), 2024.
Wang, H., Xiong, W., Xie, T., Zhao, H., and Zhang, T. Inter- pretable preferences via multi-objective reward modeling and mixture-of-experts In EMNLP, 2024a.
Xu, C., Sun, Q. Zheng, K., Geng, X., Zhao, P., Feng, J., Tao, C., Lin, Q., and Jiang, D. WizardLM: Empowering large pre-trained language models to follow complex in- structions. In The Twelfth International Conference on Learning Representations (ICLR), 2024a.
Wang, S., Zhang, Z., Zhao, R., Tan, F., and Cam-Tu Tu, N. Reward difference optimization for sample reweighting in offline rlhf. In Findings of the Association for Com- putational Linguistics: EMNLP 2024, pp. 2109-2123, 2024b.
Xu, G., Xu, K., Sudalairaj. S., Wang, H., and Srivastava, A. Cdr: Customizable density ratios of strong-over-weal. 1lms for preference annotation. ArXiv, abs/2411.02453 2024b.
Wang, S., Tong, Y., Zhang, H., Li, D., Zhang, X., and Chen, T. BPO: Towards balanced preference optimization between knowledge breadth and depth in alignment. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies ( (Volume 1: Long Papers), April 2025.
Xu, J., Lee, A., Sukhbaatar, S., and Weston, J. Some things are more cringe than others: Preference opti- mization with the pairwise cringe loss. arXiv preprint arXiv:2312. 16682, 2023.
Xu, Z., Jiang, F., Niu, L., Deng, Y., Poovendran, R., Choi, Y., and Lin, B. Y. Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing. arXiv preprint wrXivv2406.08466 2024c.
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Aligning language models with self self-generated instructions. In Pro- ceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13508, 2023.
Yang. S., Cui, L., Cai, D., Huang, X., Shi, S., and Lam, W. Not all preference pairs are created equal: A recipe for aunotation-cfffiieen iterative preference learning. In Find- ings of the Association for Computational Linguistics: EMNLP 2024, pp. 1549-656) 2024.
Wu, J., Xie, Y., Yang, Z., Wu, J., Gao, J.. Ding, B., Wang, X., and He, X. B-DPO: Direct preference optimization with dynamic B. In The Thirty-ightth Annual Conference on Neural Information Processing Systems (NeurIPS), 2024.
Ye, Z., Agarwal, R., Liu, T., Joshi, R., Velury, S., Le, Q. V., Tan, Q., and Liu, Y. Evolving alignment via asymmetric self-play, arXiv preprint MxXiv:241100000 2024.
Wa, J., Wang. X., Yang, Z., Wu, Gao, J., Ding, B., Wang, X., and He, X. a-dpo: Adaptive reward margin is what direct preference optimization needs. In Forty- International Conference on Machine Learning (ICML), 2025a.
Yuan, H., Yuan, Z., Tan, C., Wang, W., Huang. S., and Huang, F. Rrhf: Rank responses to align language models with human feedback. Advances in Neural Information Processing Systems (NeurIPS), 06:110935-10955 2023.

---

Larger or Smaller Reward Margins to Select Preferences for LLM Alignment?
Yuan, W., Pang, R. Y., Cho, K., Li, X., Sukhbaatar, S., Xu, J., and Weston, F.. Self-rewarding language models. In Forty-first International Conference on Machine Learning (ICML), 2024.
Zheng, R., Dou, S., Gao, S., Hua, Y., Shen, W. Wang, B., Liu, Y., Jin, S., Liu, Q., Zhou, Y., al. Secrets of rlhf in large language models part i: Ppo. arXiv preprint arXiv:2307.04900 2023.
Ziegler, D. M., Stiennon, N., Wu, J., Brown, B., Radford, A., Amodei, D., Christiano, and Irving, G. Fine-tuning language models from human preferences. arXiv preprint arXiv: /909//999000 08593, 2019.
12

---

Larger or Smaller Reward Margins to Select Preferences for LLM Alignment?
A. Related Work
RLHF and Preference Learning Algorithms. Despite RLHF's effectiveness in aligning language models with human preferences (Ziegler et al., 2019; Stiennon et al.. 2020; Bai et al.. 2022; Ouyang et al., 2022). the multi-stage RL training makes it computationally complex and hard to optimize (Santacroce al., 2023; Zheng et al., 2023). Researchers have been exploring more efficient and simplified alignment algorithms, by simplifying the RL training process (Dong al., 2023; Yuan et al., 2023) or utilizing only the offline preference dataset, with DPO (Rafailov et al., 2024) being notable example. In addition to DPO, various offline preference learning objectives have been proposed, such as IPO (Azar et al., 2024), KTO (Ethayarajh et al., 2024), ORPO (Hong et al., 2024), and SimPO (Meng et al., 2024), Moreover, these offline preference optimization methods have been extended to iterative settings, with new preference pairs continuously sampled or reference policy updated using models trained in the previous iteration (Xu et al., 2023; Yuan et al., 2024; Rosset et al., 2024; Xiong et al., 2024).
Data Quality in Alignment. The importance of data quality in alignment processes has been well-documented, both in RLHF (Nakano et al., 2021; Bai et al., 2022) and offline preference learning methods. A significant body of research focuses on the distribution for response sampling during preference dataset construction (Liu et al., 2024a; Tajwar et al., 2024; Xiong et al., 2024), while another line of work examines the quality of different preference pairs within the preference dataset (Wu et al., 2024; 2025b; Morimura et al., 2024; Pattnaik et al., 2024; Wang ct al., 2025). Of particular relevance to this study are efforts to explicitly define and utilize data quality metrics, such as leveraging the explicit reward margin to select high-quality data (Khaki al., 2024; Ye al., 2024) or reweighting loss (Wang al., 2024b), and using the implicit reward margin to prioritize training data (Muldrew et al., 2024; Yang et al., 2024) or calibrating loss functions (Xu et al., 2023; Xiao et al., 2024). Despite the demonstrated effecttiveness of these metrics, their often conflicting properties (as shown in Figure la) necessitate the development of a more universal data quality metric, which motivates us to propose the alignment potential metric.
Additionally, current works based on implicit reward margin are limited to iterative preference learning settings (Muldrew et al., 2024; Yang et al., 2024), requiring the model xy to be different from Tee to avoid constant- implicit reward fg. To overcome such a shortcoming, we propose to utilize the SimPO-based implicit reward ASim resulting a new version of implicit reward margin applicable for standard offline preference learning scenarios.
Self Play Alignment and Prompt Synthesis. The paired comparison nature of preference learning has inspired range of self-play methods based on two-player games (Munos et al., 2024: Chen et al., 2024; Rosset et al., 2024; Wu et al., 2025c), with both players being LMs generating responses to given prompts. Diverse from these works, eva (Ye et al., 2024) proposes an asymmetric alignment game involving a prompt creator and a response solver to augment preference data with higher reward differences (aka. informativeness in eva). To generate new prompts, researchers have developed various prompt synthesis methods such as SelfInstruct (Wang et al., 2023), EvolQuality (Liu et al., 2024b), EvolInstruct (Xu et al., 2024a), Magpie (Xu et al., 2024c) and so on Our work adopts the asymmetric self- pelf-play framework from eva generate additional preference data, with a focus on evaluating various data quality metrics within this evolving data context.
B. Additional Experiments
B. Ablations study for Data Metries Experiments
Additional Reward Models. For the ablation study concerning different reward models, SimPO provides an additional preference dataset generated using the -amai---b-insttuutt model and annotated with the PairRM reward model, available at llama--ltrafeeddack. Given such convenience, we directly utilize this dataset and select top-40% subsets with different metries for subsequent SimPO training. The performance of trained models with different metrics is depicted in Figure 8a. Consistent with the results discussed in $ 3.2, our proposed alignment potential metric MAP still achieves the highest performance across all benchmarks under varying reward models, demonstrating the efficacy and robustness of our metrics.
Additional Training Objectives. For the ablation study focused on different training objectives, we opt for DPO loss as a substitute for SimPO and employ the Gemma-2-9b-it model, noted for its high performance. We again apply various metrics to select top-40% data for training. The results, depicted in Figure 8b, showcase that our proposed metric consistently outperforms existing baselines with notable improvements. These consistent performance gains certify the MAP metric's 's ability to identify high-quality data for alignment training. demonstrating stable improvements for varying
Our contribution orthogonal to different prompt synthesis techniques, and we employ Evollnstruct to generate new prompts
following eva.
13

---

Larger or Smaller Reward Margins to Seleet Preferences for LLM Alignment?
uniform
M,
M
Map (ours)
uniform
M,
Mn
MaP (ours)
1%6)
Arena (%)
Alpaca (%)
Alpaca WR (%)
WR
38
37
34
66
42.9
36
26
34
+2.1
30
3.44
+4.2
+3.7 13.8
45.7 +5.
54
+1.3
62
53
32
28
+0.0
+0.0
-0.0
-0 0
30
(a) Ablation on different reward models
(b) Ablation on different training objectives
Figure 8. Ablation study for data metries. (8a) Performance of Llama-3-8b-instrrut model trained with SimPO loss on preference pairs selected by different metrics. The preference dataset is based on the PariRM reward model, instead of the ArmoRM model considered in other settings. (8b) Performance of Gemma-2-4 2-9biit model trained with DPO loss on preference pairs selected by different metries.
training objectives. Additionally, we include IPO (Azar et al., 2024) as another ablation training objective, and include both PairRM and ArmoRM-based Llama datasets for experiments. Consistent with previous configurations, we select the top 40% of subsets based on various metries for further training. As shown in Table 2, our proposed MAP metric still outperforms existing baselines under this alignment method, further certifying its effectiveness.
Table 2. Performance of Llama-3 1iima-3-8b-Instruee model trained with IPO on datasets selected by different metrics.
(a) Results on PairRM annotated dataset.
(b) Results on ArmoRM annotated dataset.
Metries
Uniform
Mp
M,
MAP
Metrics
Uniform
Mr
Ma
M AP
AE 2.0 LC
30.49
34.15
35.04
35.44
AE 2.0 LC
33.17
33.92
38.
40.35
AE 2.0 WR
34.15
34.08
35.40
36.08
AE 2.0 WR
30.63
30.17
35.73
38.14
Table 3. Performance of Llama-3-8b- Instruct model trained with SimPO on datasets selected with varying selected ratio.
Varying Data Amount. In previous data selection ex- periments, we focused solely on a fixed proportion of the dataset subset, specifically 40%. In this ablation study, we explore the impact of varying the quantity of selected data. Utilizing the -lama-3-8b-Insteee
Ratio
Top-20% %
Top-40% %
Top-60% %
AE 2.0
LC
WR
LC
WR
LC
WR
model along with its PairRM-based dataset, perform SimPO training with selection ratios of 20%, 40%, and 60%. Table 3 shows that our MAP metric consistently outperforms existing methods across different proportions of selected data.
Uniform
26.02
28.06
30.72
32.64
33.94
34.72
M,
28.95
30.24
36.58
36.12
40.77
38.14
Mr
28.46
31.01
33.53
34.75
38.29
37.28
MAP
30.66
30.62
37.07
36.58
42.86
39.60
Additional Selection Strategies. Beyond the existing M, and Mr metrics, we broaden the data selection baselines by introducing strategies: (1) reversed data selection: In this approach, we select the bottom-k data ranked by various metrics. (2) ranking-based metric: Here, propose selecting preference data where the ranking of implicit and explicit rewards for th and yt is opposite. Specifically, this occurs when 1(r(x, ywo) > r(z, vi)) + Il (z, Mwr) > fa(x, in)) 1. This intuitive approach mirrors the M+ metric as discussed in Section 3.1, both of which encourage the contradictions between the current L.M and the reward model's preference judgment. In contrast, our MAr incorporates a reward noise regularization design to avoid such contradictions, thus mitigating the reward noise problem.
For reversed data selection, we employ the Llama-3-8b- Instruct model and its ArmoRM-based dataset for SimPO training. As demonstrated in Table 4, the performance of resultant models across all metrics-wher selecting the bottom 40% based on M, -Mg. M-P-was poorer than the uniform baseline, indicating lower data quality through reverse selection. Notably, the lowest performance observed was with our MAP metric, verifying its efficacy in identifying high-quality datasets from an opposite standpoint.
Table 4. Performance Llama-3-8b ---srrutu model trained with SimPO on datasets reversely selected with different metrics.
Reversed Metrics
Uniform
M,
M,
MAP
AE 2.0 LC
41.61
40.66
37.07
34.64
AE 2.0 WR
36.26
33.36
32.95
29.37
14

---

Larger or Smaller Reward Margins to Select Preferences for LLM Alignment?
For the ranking-based selection, we select 40% subsets of Llama-based datasets where the ranking of explicit reward and implicit reward Yho and y is opposite. As anticipated, this selection method clearly underperforms compared to the regularized MAP metric, reinforcing the justification of our reward noise regularization design.
Table 5. Additional comparisons with ranking- bastnd selection. Experiments are conducted by training Llama-3-88 1aamma---Bb-ae 8b--nntrrcct model on different datasets using SimPO.
aa) Results on PairRM annotated dataset.
(b) Results on ArmoRM annotated dataset.
Metrics
Uniform
Ranking
MAP
Metries
Uniform
Ranking
MAP
AE 2.0 LC
30.72 32.64
31.53
37.07
AE 2.0 LC
41.61 36.26
43.48
47.83
AE 2.0 WR
30.40
36.58
AE 2.0 WR
38.47
45.14
B.2. Significance Testing
We present detailed results corresponding to the figures in the main body of this paper (i.e., Figure and Figure 5), including the standard deviation of AlpacaEEval 2 and the 95% confidence interval of Arena Hard, As shown in Table 6, all benchmark results are reasonable and do not display significant outliers noise.
B.3. More Comparison with EVA
Algorithm 2 Eva with Select-then-Evolve Pipeline
Input:
SFT model TSFT, reward model r, existing prompts and data quality metric M. Initialize no, with TSFT- for iteration I,..., do
Generate initial preference dataset: D < GenDataset(Xx To. 11
Select top-k dataset using M metrie:
Dr = {2x, Yw 31) E DD (ar, Vw. 02-1) 2 Jk), with being the k-th largest M score in DD Evolve prompts: X + Evollnstruct(Ak, where Nr. denotes the prompts of D2 dataset. Generate additional preference dataset: D' < GenDataset :.. Conduct preference optimization on D 0r + 0:-1 - -Vo((0--
end for
Return: optimized policy wer
Our main study employs the evolve-then-select pipeline for preference data generation, detailed in Algorithm 1, where streamline the dataset generation process using the GenDatasct(A' ("otassttt"""or) function. This function takes the prompts X', samples responses with LLM me, and annotates pref- erences with reward model r to construct a preference dataset.
Performance and Map Score of Different Datasels
Avg win rate
Ma score
DPO win rate
1.00
62-
SimPO
0.75
Potential Score
0.50
Arena-Hard WR
The wvolve-then-seled pipeline ensures the final dataset D/ aligns with the evaluation of data quality metrics.
0.25
-0.00
In contrast, the original framework operates under select- then-evolve pipeline", detailed in Algorithm 2. Within this pipeline, data selection precedes the evolving data process. Therefore the data quality of the final evolved dataset D/ may not be consistent with that of D; due to the intricate prompt evolving and preference data generation process in data evolving. Nevertheless, we still include comparisons under this select- then-evolve setting to ensure comprehensive and equitable evaluation against eva.
-0.50
50
0.75
data
seloct-evalved
select evolved ours
by
Figure 9. The MAP scores of different datasets and the corre- sponding Gemma models' performance. The dataset select- evolved by methods shows the highest alignment potential scores and produces the best-performing results.
*The original eva paper employs M, scores as sampling weights for Dj selection but lacks implementation guidance. Thus substitute this sampling with a similar top-k selection.
15

---

Larger or Smaller Reward Margins to Select Preferences for LLM Alignment?
Table 6. Standard deviation and 95% confidence intervals of benchmark results.
(a) Results of
model (Figure 4)
(b) Results of Gemma-2-9b-in model (Figure S)
Metries
Uniform
M,
M,
MAP
Metries
Uniform
M,
M+
MAP
AE 2.0 LC
41.61
44.60
47.15
47.83
AE 2.0 LC
67.95
68.63
68.25
70,45
AE 2.0 WR
36.26
40.53
41.22
45.14
AE 2.0 WR
61.40
68.64
69.45
70.45
AE 2.0 std
1.45
1.44
1,43
1.46
AE 2.0 std
1,45
1.39
1.38
1.40
Arena WR
29.9
33.5
33.0
34.6
Arena WR
55.7
59.6
61.9
62.8
Arena CI
(-2.1, 1.7)
2.44
(-2.3, 2.0)
(-2.0, 2.4)
Arena CI
(-2.6, 2.3)
(-2.7,3.0)
(-2.7, 2.3)
(-2.3, 1.8)
Using the experimental setup detailed in $5.1, but reversing the selection and evolution order, assess models trained on various datasets as reported in Table 7. The results indicate that datasets select-evolved with our metric continue to deliver better overall performance than the supplementary UltraFeedback datasets and those derived using the eva method. To corroborate that these performance improvements stem from the ability to select high-quality data, we present the average performance (Arena Hard win rate) of Gemma models trained with either DPO or SimPO,, alongside the alignment potential scores of the respective datasets, Figure 9. As illustrated, the model performance continually improves as the dataset's alignment potential score increases, with the dataset select-evolved by our metric exhibiting both the highest alignment potential score and performance. This correlation between alignment potential score and performance further underscores our metric's capability of discerning high-quality data for training.
Table Main experiments under the select-then-evolve self-play setting
J
Gemma-2-9bbit
Method
AH
AE 2.0
AH
AE
2.0
WR
LC
WR
WR
LC
WR
SFT
21.4
23.25
23.50
40.7
48.75
36.49
23.6
29.80
28.32
46.0
54.00
46.11
+uf-10k
35.0
42.67
41.67
56.4
62.56
62.17
+eva-10k
35.0
40.38
42.10
56.2
61.73
62.94
+ours-10k
35.5
41.98
44.29
57.3
62.07
62.42
SimPOor-rok
24.2
28.45
26.23
45.2
56.90
43.80
+uf-10t
28.5
35.75
32.82
53.5
66.43
64.55
+eva-10k
32.7
40.78
38.68
55.9
64.96
62.88
+ours-10k
33.3
42.75
41.72
60.9
65.29
65.47
C. Convergence Rates of DPO with Different Samplers
C.1. Proof of Theorem 3.1
Here we restate the DPO loss in its population form:
E (( [[xx 0)] = ((..n//>> 1- p(y
s²) logo fo(1))) E(..y.,' (-o (-o(r
- (og(ou) f
Under stochastic gradient descent, data (ar, y ( is sampled to update the parameter He each iteration t with the empirical loss. Since the sampled context xf and arms n',"" can be utilized to compute both L(x', v² 2/ 0²) and C(x', y" "'".v' y²;0"), we utilize both terms to derive the stochastic update rule, like previous works (Azar et al., 2024; Rosset et al., 2024):
g++l =²
2" n'- [[x' v v 0¹) + [(x', V." v'''''0'))
(12)
16

---

Larger or Smaller Reward Margins to Select Preferences for LLM Alignment?
where 1f denotes the time-varying learning rate. Without loss of generality, assume uniform reference policy (y)ar) = [ like Shi et al. (2025). Therefore fø(z, y) log 1/ = (0 y) - log ((x) and the optimum 0" characterized by r(z, v) = P-- (x,y) + log (z), turns into 30 (z, v) = rr(z, v) + c(z), where c(x) is a constant independent of y. The proof of Theorem 3.1 is as follows:
xe(u]=)
Proof of Theorem 3.1. For case of annotation, we first consider the single context setting | = so that we can denote 9, = ((.,) y),7 Ty = r(z, y) with x omitted for simplicity. We first investigate the stochastic update rule of 0 given by Equation (12). Let the arms chosen at iteration t being y. y', we have:
BBB 3
0++1
0g
(o Tyr (o))((33) 30%) or - ryy)o 30
17'
8
02 +
[o - r.y) - (30g 30t
r 3
0
0g +
[o(ri- - ryy) - or (30 - 30t}
2
n 3
By 2
[o(ry - ryy - o -0
where the second equality holds by the fact that (z)o(-9) o(-z)o(y)) (xx) - -yy).
Since the optimal target is 30 = r + c, where e is a constant, we denote a new vector 5² = 30" - -r. Let:
bu =rr - -ry) -0(03 - 30
The update rule can be rewritten
T A At uy 4/
0, = y +
= =0
2
2
(13)
n 3²
n A Av
<t Ey +
A vb Eog
Eg.
2
Note that 0t+ o f + or and gt+l + t E ey. which indicates the mean of and & remains unchanged during optimization.
For optimal 0", the === 30 should be a constant vector 1. Therefore, we can measure the variance of g as an indicator of convergence:
v
1
Σ c - where Et
Σ
|
|
VEY
NEY
Using Vt, the distance function Df 8*) satisfies that:
Dis (0)0 = D
Σ
MI = D/2
1
Σ
(c),
)² 2V'
y.W/ V.W'EY
Now we examine how Vc changes at each iteration. Assume the sampled pairs are yf at iteration t, then we have:
pe+1
Vt_ +
[est]
E2 + (E - 12 (E)) - -2 (E) - 2)0]
= V* +
[15tt12 + (5 - (E,) - )²]
|>
V' +
[(Et + a (yy )² + (ES - 1 - (E5) - -(E))
(let a
y
2
2a
V* +
(E)
Ety + a Any''
y
where the second equation holds with g++ Ex and ct+1 + E = <t + Ey.
17

---

Larger or Smaller Reward Margins to Select Preferences for LLM Alignment?
Using the mean value theorem, have:
bo o' (Nyy 1(r rg (30, -0yy)) (N Egy),
where Ayy is between ry Ty and 30g 30yr Note that d'()) (0, 4),, and the value of d'() can be further lower bounded using the update rule.
The change in V can thus be written as:
2(1 (N ((r)))ao((X (N (E) -
AV = - Vt= -
[2]
Under the optimal learning rate ry² ( vy/)
(we adjust the learning rate at each iteration, similar to the line search
methods), such that ao'(N = 2. the change in V satisfies:
1 A V
2/3/ (5) ct
(14)
Now we can examine the impact of different samplers, which determine the choices of y, y at each iteration. For the uniform sampler, under the optimal learning rate ry,, we have:
Ely = V² + E|AV
Vt 1
Ey..'rss [(5] -Ety (
2/3/
V 1
Σ
Σ
(5/ 32
2|
DP
yEy y'EY
1
Vt
2V
2
= (1 Vv
y
So have the following convergence result for the uniform sampler:
E V (1
vvo
(15)
D
In contrast, the adversarial sampler, which chooses the pair y² y² to maximize the Mj score:
ya, 3 arg max M2 (z. U. 8²)
v.y'Ey
arg max ry N'y (30 -30,,, wa'ey
arg max Eg
wayey
which means to select the maximal and minimal elements from ?º. Given this definition, the selected pairs at each iteration "//" satisfy that:
(Ey- - < (E - + - V, V.
M' = Ey. - E² and st = E - E.. the inequality above can be rewritten Let Let = E E
(m' (M */)) 5,0. e.
18

---

Larger or Smaller Reward Margins to Select Preferences for LLM Alignment?
Summarize over all y E V/ we have:
2
Σ (m² - ,ly)
MEy
| m' M' -(m² + M)
Σ
S Σ
(2g)
DEy
yEy
| -immmmM M a + Σ (00))
yEy
= DII [(Ey- - [5 - - - v4]
where the second equality (line 2 line 3) holds by Eyey sg 0. Combing with the fact that (a (0-b)) 6)2 -4ab, have:
( - 3 2 - -((/ (E) - gt)
2 2V..
Therefore, when updating using the selected pair y", 3 at iteration i, we have (under the optimal learning rate in Equation (14)):
V++1 Vt
(5)
5
2
2
< VV' Vt_
V
2
= (1
V
|
So have the following convergence result for the adversarial sampler:
2 vvo D
v s (1
(16)
Eventually, we can compare the iterations required to converge to the same level of error, when applying different samplers. Since Dist(@", )² 2V', to reach Dis 8*) = =Dist(0", 8*) for some small 1, or equivalently, V =2 the
uniform sampler would require (u(e) log log(e) 1/ iterations according to Equation (15). In comparison, the adversarial
sampler would require Tado (s) log(1 D/ iterations by Equation (16). As one may verify that for |)] 2:
2 [c)
2 log([1
1
log(1
[ -
therefore the adversarial sampler requires less than half the iterations of the uniform counterpart:
(e) < (c)
(17)
Furthermore, the above results can directly generalize to the contextual setting where |X > 1, which we briefly discuss
here: With slight abuse of annotations, we can re-define V | Exex Ege ) and the update of V with preference data (I, y, y²) t would be:
2a (C',, Ety + )
V++ V
|
Then the convergence results immediately become E[V'] (1 ( V'v for uniform sampler and V < (1 for the adversarial sampler, leading to the same result with Equation (17).
2 V
19

---

Larger or Smaller Reward Margins to Select Preferences for LLM Alignment?
C.2. Numerical Experiments
Theorem 3.1 establishes that the adversarial sampler requires less than half the iterations of the uniform sampler to reach the same error level. Here we conduct numerical experiments to validate this result.
We examine two scenarios: the standard bandit setting with a single context (i.e., [X] = 1) and a contextual setting with |X| = 5. In both scenarios, the arm space is set to | 10. Rewards are initialized uniformly using U|0, 1J and the initial parameter 0 is set to O. We consider a uniform reference policy, with the Kl-parameter 3 fixed at 0. For the learning rate, while our proof uses the optimal learning rate ry [P 2 13z a fixed learning rate is set to 4/ for simplicity.
Figure 10 presents the error, i.e., the distance to optimum Dist(O, 0*), averaged across 10 random initializations. As depicted, while both samplers achieve linear convergence, the adversarial sampler converges significantly more rapidly due to its strategic selection of the data with the most potential for alignment. Notably, in our experiments, the uniform sampler required approximately six times more iterations to reach the same level of error as the adversarial sampler, which corroborates the theoretical prediction of speed-up by factor of more than two.
Convergence different samplers (X]=1.|MM=10)
Convergence different samplers (XX|55 10
10
advarsarall (aurs)
uniform
10"
uniform adversancl (ours)
10
10"
Dell8 81)
Det(f
10
10"
10"1
18"5
12'44
slep
step
(a) Convergence results for | = 1, [3]]
10
(b) Convergence results for [ = 5, | = 10
Figure 10. Numerical experiments for DPO under contextual bandit setting. This figure compares learning convergence using two samplers: uniform sampler and an adversarial sampler that selects pair data to maximize our M1 metric. The plots display the error [0, 0*) across random initializations. Figure 10a addresses the single context scenario, while Figure 10b pertains to multiple contexts. The error lower bound of approximately le-15 is attributed to floating-poin precision limitations.
D. Implementation Details
D.1. Codes & Hyperparameters
Our code is open-sourced in: http://////gatttooo///////////// hotp////esse////eeeeeeteeeeeaaaa
Data quality metries. We mainly investigate three metrics in this paper: (1) the explicit reward margin metrie:
M, Vue 3/) = |r yw)) - r(x, st]l: (2) the implicit reward margin metric: M (z, Bu. 31) = |fa(z, yw) - fe(x, w)];; and (3) our proposed alignment potential metric: MAP = |r(x, Bhe) r(x, 30) - (Z, yw) - fo( 31)l. While explicit rewards r(z, 2) are directly given by reward model, computing the implicit rewards fo 3) 3 log )e
[.(x)) requires the LLM
Ty, a reference policy Tref, and a parameter / > 0. As discussed stated $3.2, since the implicit reward fo y/ becomes constant-zero when me is identical with Tref. we propose implementing the implicit reward via SimPO's length-normalized reward: ASim (x. y) ,3 log Te T which can be seen special case of As with uniform reference model (Wu et al., 2025a). Regarding the parameter 33, while it doesn't 't influence the ranking determined by M,. it does affect the score produced by our MAP metric. To effectively choose value for 8, let and Ø denote the standard deviations of |r(x, vw) r(x, 31]] and [log To (y l/ - log e (3n |=)\/>||| across the dataset, respectively. Utilizing these standard deviations, we rescale the two margins akin to Z Zcoore normalization and introduce a parameter a e [0.25, 0.3, 1.0, 2.5, 5.0} for computing the MA metric:
1 MAr(E, Bhe v1) = or
a log Te ( (z) log ne (v1 [x)
|r(z, vw) - rIx, v1)
dn
V/
20

---

Larger or Smaller Reward Margins to Select Preferences for LLM Alignment?
Here use a distinguish it from the 3 hyperparameter used in the DPO and SimPO loss function.
Preliminary experiments. The preliminary experiments, outlined in Section 3.2, employ the three metrics to select the top 40% subsets from existing preference datasets within SimPO. While the explicit rewards r(x, y) are present in SimPO's s datasets, they didn't record the prob- abilities of LLM To (y]2)) when sampling the responses. So we use the corresponding models from huggingface: Gemma-2-9b-in and (aama-3-bb-instie
Table 8. Hyperparameterss used in our preliminary experiments.
Gemma (ArmoRM)
Llama (SimPO)
Hyperparameter
SimPO
DPO
ArmoRM
PairRM
learning rate
8e-7
5e-7
le-6
le-6
B
10
0.01
10
2.5
input the gener- ated responses y and prompts y. and compute To (v( to
7/B
0.5
\
0.3
0.55
derive implicit rewards log अe (y/l)//bll We use three datasets for data selection: two Llama-based datasets annotated by and PairRM: ulaaaa-llrafeedbbckk, and one Gemma- based dataset annotated by ArmoRM: gemma2- gemmaa-ultrafecdbaaee www.aaeeedacc-aaanaeae armormm For the hyperparameter in MAP. we set a = 1.0 for Llama' s PairRM anmotated dataset and set a = 2.5 for the remaining two datasets. After data selection, we apply the tuned hyperparameters reported in SimPO's s paper for both SimPO and DPO training. Specifically, we set a batch size of 128, with max sequence length of 2048 and max prompt length of 1800. All the models are trained using cosine learning rate scheduler with 10% warmup ratio, via AdamW optimizer for one epoch. Table 8 details the other hyperparameters for different datasets and objectives
ArmoRM: wtwaaaallaaaaaeeaaaaaa-aaoaaooe
Main experiments. In the main experiments, we integrate our metric MAP into the evolving alignment (eva) framework, which involves iterative preference dataset generation and prompt evolving processes. To generate the preference datasets D and D', we use top-p sampling (p=0.95) with a temperature of 0.8 and max sequence length of 4096 to sample responses y for prompts x. Following SimPO, we use five distinct random seeds to sample 5 responses (. .//} for each prompt x, and use the responses with max/min reward r(x,, y). annotated via ArmoRM, construct the preference pair (1r, Mhes M1). The previous sampling parameters are also used for prompt evolving, i.e., query the model to rewrite existing prompts X X' into evolved prompts X". Regarding the instruction for LLMs to write prompts, we employ the five prompts in EvolInstruct (Xu et al., 2024a) and make some modifications for better instruction-follene revised prompts are available in Table 9. Incorrectly formatted prompts per the evolving instruction are filtered out when constructing X².
capabilities. Detailed descriptions of these
As for the subsequent data selection and training, we use a = 2.5 for gemma and a = 5.0 for llama for our main results (Table 1 and Table 7). For llama with DPO, we set learning rate of 7e-7 and B 0.01, and other hyperparameters remain consistent with Table 8. In the data-size scaling experiment (Figure 6), the number of EvolInstruct prompt templates is reduced from 5 to 2, ensuring the processed prompt set is less than 30kx2, maintaining smaller prompt set than 60k. The hyperparameter a = 0.5 and 7/3 = 0.3 is employed for MAP metric and subsequent SimPO training. For the multi-iteration experiment (Figure 7), we reduce the 3rd iteration's learning rate to 3e-7 to prevent overfitting. The 7/B parameter is tuned within {0.1, 0.3, 0.5, 0.8). and the berf-performing results based on the Arena Hard WR are reported.
D.2. GPT-4 Annotation
In 1, we use M+ and MAP metrics to select the top-10% subsets from SimPO's preference dataset and measure GPT-4's agreement on the preference annotation in data (I, Vw: 3/). Specifically, we input the prompt æ along with two responses yw: M1 for GPT-4, ask choose the better one, and check if it aligns with the preference data's judgment, i.e., y > 3 lz. For feasibility and financial consideration, 1,000 preference data from the selected top subsets is sampled for annotation. We employ the prompt template from Arena-hard (Li al., 2024) to prompt the GPT-4-1 1 106-preview model for preference annotation. The prompts can be accessed via their config file.
In addition to the agreement results in Figure 3, which is selected and annotated on the Gemma dataset of SimPO, we also include the agreement results of the Llama dataset of SimPO in Figure 11. Here data selected by MAP metric also results in a higher agree rate than M4, underscoring the effectiveness of noise reduction strategies in M MAP-
E. Illustrative Examples of Preference Data
This section provides detailed examples of preference data utilized in this paper.
Contradiction of existing metrics. Prior research indicates that prioritizing data with large explicit reward margins or smaller implicit reward margins can improve alignment performance. However, these two metrics often give conflicting
21

---

Larger or Smaller Reward Margins to Seleet Preferences for LLM Alignment?
GPT-4 Agreement with Reward Model on Selected Data
M
53.8%
13.5%
32.7%
MAP
57.4%
13.1%
29.5%
Yu yı (Agree)
Yv * y) (Equal)
ya <), (Disagree)
Figure 11. GPT-4 agreement with reward model on data selected by different metrics from SimPO's dataset with Llama model. The data selected by Mae higher agree rate than M.. indicating less reward annotation noise.
guidance, demonstrated by two examples from the SimPO preference dataset geenma--utraffeddhck-aaaa depicted in Figure la: The first example, where the data produces both large explicit and implicit reward margins, is rated as "high-quality" by the explicit reward margin but "low-quality" by the implicit reward margin. Detailed information for this example is provided in Table 10. The second example, where the data produces both small explicit and implicit reward margins, is rated "low-quality" by the explicit reward margin but "high-quality" by the implicit reward margin. This data is elucidated in Table 11.
Our proposed alignment potential metric differentiates itself by classifying both examples as "low-quality." This is due to the small disparity between explicit and implicit rewards, indicating that the LLM's preferences are already well-aligned with the data, thus negating the need for further training.
Noisy reward model annotation. Reward models can introduce noisy preferences in annotation. An illustrative example from the gemma2-ultran aatammaauuttaaeeedaaaaaaaaaa aarmoom dataset is presented in Table 12, with its simplified version illustrated in Figure 2 in the main body of our paper.
As shown in the example, the prompt asks LLM about the name of Zulu soldiers and requires it to output "No answer" if the answer cannot be determined by the provided text. Since the provided text does not state what Zulu soldiers are called, the correct answer would be "No answer", corresponding to the y response. However, the reward model predicts lower reward for the correct answer and incorrectly assigns higher reward for the wrong answer Yu-- This highlights the limitations of reward models and the resultant annotation noise.
Morcover, this preference data also implies that the implicit reward, given by the LLM itself, could provide correct preference annotations: the implicit reward (a, 3t) = -3.4 for the correct yet mislabeled response yı, is much higher than the wrong answer (2, ywo) -899. This observation is supported by recent studies recognizing the LLM's capacity to annotate preferences (Chen et al., 2025; Xu et al., 2024b).
22

---

Larger or Smaller Reward Margins to Select Preferences for LLM Alignment?
Table 9. Prompts to evolve instructions, adapted from EvolInstruct (Xu et al., 2024a).
In-Depth Base Prompt with an adaptive "strategy" identifier: <STRATEGY>, which will be replaced by the following 4 strategies
I want you act as a Prompt Rewriter. An Your objective is to rewrite a (Given Prompt) into a **more complex version** to make those famous AI systems (e.g., ChatGPT) a bit harder handle. An =*Requirements An The (Rewritten Prompt] cannot omit the input in the (Given Prompt). An You SHOULD complicate the given prompt using the following method: \n <STRATEGY> \n The (Rewritten Prompt) } must be reasonable and must be understood and responded to by humans. ** Constraints that must be followede<:'n - The (Rewritten Prompt) can only add to 20 words into the (Given Prempt). in The (Rewritten Prompt] should be seli-contained, with **all necessary information provided, so that it can be responced without needing refer back to the (Given Prompt). An Your respense should contain 4xenly ** the |Rewritten Promptl, **without any** additional formatting or introductory phrases such as 'Here is the rewritten prompt:' or 'The zewritten prompt is:' An The (Given Prompt]: \л<PROMPT><n ========)
nBased on the prompt above, rewrite a prompt \n(eerittRReertteen Prompt): AAn
Adding Constraints
The strategy is: Please add more constraint/requiireeen into the (
Given Prompt)
Deppening
The strategy is: If the (Given Prompt) contains inquiries about certain
issues, the depth and breadth of the inquiry can be increased.
Concretizing
The strategy is: Please replace general concepts with more specific concepts.
Increasing Reasoning
The strategy is: If the (Given Prompt) can be solved with just few simple thinking processes, you can rewrite it to explicitly request multiple-step reasoning.
In-Breadth Prompt
want you act Prompt Creator. An Your objective is take inspiration from the (Given Prompt) ) create **oe*** brand new prompt. An *Reeiuuireenntse: \n This new (Created Prompt] } should belong to the same domain as the {Given Prompt) but with di different details. An The LENGTH and complexity of the ICreated Prompt] | should be similar to that of the (Given Frompt). An - The (Created Prompt } must be reasonable and must be understood and responded by humans An the (Given Prompt) includes specific input as part of its instructions, create
input for your (Created Prompt when The (Created Prompt] should be seli-contained, with +*all necessary information ** provided, so that it can be responded to without needing to refer back to the (Given Prompt). An Your response should contain *xonyy ** the (Created Promptl, *wwithout any* additional formatting or introductory phrases such as "Here is the created prompt:' 'The created prompt is:".An The (Given Prompt): \л<РRомPT>Аn ===..s==) nBasee on the prompt above, create your prompt: AnCreatedd Prompt);
applicable. \n *Constraints that must be follovedr
23

---

Larger or Smaller Reward Margins to Select Preferences for LLM Alignment?
Table 10, Example from SimPO's preference datasct, in which the explicit reward margin and implicit reward margin are both large. We rescaled explicit reward value to comparable range implicit rewards.
Prompt
Given the task definition and input, reply with output. In this task you given wikipedia articles on range of topics as passages and question from the passage. We ask you to answer the question by classifying the answer as 0 (Falsc) or 1 (Truc) in\nPassage: Tabasco sauce -- Tabasco sauce is a brand of hot sauce made exclusively from tabasco peppers (Capsi cum frutescens var. tabasco), vinegar, and salt. It is produced by MeIlhenny Cempany Avery Island, Louisiana. nQuestion: is tabasco sauce and hot
the same
Chosen Response Yo
(Halse) \nAnwhile Tabasco sauce * *is* type of hot sauce, the passage states that Tabasco sause is a brand name, Therefore, not all hot sauce is Tabasco sauce. \n
Rejected Response
An
Explicit Rewards
Chosen: r(2, 10.3, rejected: r(z, yt) 3.4, explicit reward margin: Mr, yw Bi) 6.9.
Implicit Rewards
Chosen: F8 (x, Vvc)
= -1.4, rejected: (
(z,81) y1) = -7.7, implicit reward margin:
Ma (z, Vw y1) = 6.3.
Table Example SimPO's preference dataset, which explicit reward margin and implicit reward margin are both small. rescaled the explicit reward value to comparable range of implicit rewards.
Prompt xc
let's play game. say a sentence, then you make a sentence that follows up my sentence then i give continuation to yours and so on. you ready?
Chosen Response Yw
Sounds fun! I'm ready. Give me your first sentence. \ud83d\ude0a Antn
Rejected Response Yt
I'm ready! Let's do it. \udB3d\ude0a \n\n**Give me your first sentence An'n
Explicit Rewards
Chosen:
r(z, vw) = 13.7, rejected: r(x, 3t) = 13.0, explicit reward margin: M, Bes Wt)
0.7.
Implicit Rewards
Chosen: ASim (I, yw)
-3.7, rejected: Sim (z, vi) -2.9, implicit reward margin: M (z. Vw, vt) = 0.8.
24

---

Larger or Smaller Reward Margins to Select Preferences for LLM Alignment?
Table 12. Example from SimPO's preference dataset, in which the reward model's judgment incorrect. In this example, the text provided in the prompt does not explicitly state what Zulu soldiers are called, thus the correct answer to the question "What are Zulu soldiers called?" would be "No Answer", which corresponds the "rejected" response by the reward model. Note that the response with higher implicit reward is correct in this example (we use SimPO's implicit reward in implementation as explained in $3.2). Prompt x
Please answer the following question: Please answer the following question about this movie plot. If it's un-answerable, please output \NN answer\". Question: What are Zulu soldiers called? Movie plot title: Zulu Dawn Movie plot: The film set in British Scuth Africa in the province of Natal, in January 1879. The first act of the film revolves around the administrators and officials of Cape Colony notably the supremely arrogant Lord Chelmsford and the scheming sir Henzy Bartle Frere, who ooth wish to crush the neigabouring Zulu Empire, which is perceived as threat to Cape Colony's emerging industrial economy. Bartle Frere issues an impossible ultimatum to the Yulu king, Cetshwayo, demanding that he dissolve the Xulu Empire Cetshwayo refuses, providing Cape Coleny with a pretext to invade zululand. Despite objections from leading members of Cape Colony's high society and from Great Britain itself, Bartle Frere authorises Lord Chelmsford to lead a British invasion force into Zululand. The rest of the film focuses on the British invasion of Zululand and the lead-up to the Battle of Isandlwana. The invading British army laden with an immense network of supply wagons, invades Zululand and marches in the direction of Ulundi, the Zulu capital. British forces, eager to fight large battle in which they can unleash their cutting-eage military technology against the vast Zulu army, become increasingly frustrated as the main Zulu army refuses to attack the British, and Eighting is restricted to a few small skirmishes between British and Zulu scouts. Concerned that their supply lines are becoming overstretched and that the main Zulu army is still at large, British troops begin torturing captive Zulu warriors in effort to 1earn the location and tastics of the Xulu army. Hal fway to Ulundi, Chelmsford halts his army at the base of Mount Isandhlwana, ignoring the advice of Boer attendants to entrench the camp and laager the supply wagons, Leaving the camp dangerously exposed. During the night Colonel Durnford and an escort of fifty mounted Basutos approach the camp. Lord Chelnsford then orders Durnford to return his unit, bringing them to the сamp immediately reinforce Colonel Pulleine. Lt. Vereker should join Durnford as...
Chosen Response yo
Impis An
Rejected Response yI
This plot description does not state what Zulu soldiers are called. No answer. \n
Explicit Rewards
Chosen: y(x,yw) = 11.2, rejected: r(z, 91) 5.0, explicit reward margin: M1, ywr vi) 6.2.
Implicit Rewards
Chosen: (iim(r, vw) = -8.9, rejected: -Slm (I, v1) = -3.4, implicit margin: M (2, Bos W1) 5.5.
25


Reviews:
[{"review_id": "cVbmRUEs1C", "reviewer": "Reviewer_HZK1", "summary": "- This paper proposes a new way to select good preference data jointly using two different reward signals which are captured through the external reward model and training model’s implicit DPO reward, respectively. Start from the mathematical derivation of DPO, the authors suggest the revised score function to alleviate the problem from the original score. Through two different practical scenarios of preference learning (offline and online), the proposed method is demonstrated compared to the existing baselines.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Ablation I: absolute values\n\nAlthough Eq.11 incorporates two additional absolute values compared with Eq.10: $|r(x,y_w) - r(x,y_l)|$ and $|\\hat r_\\theta(x,y_w) - \\hat r_\\theta(x,y_l)|$, only the latter absolute values (on implicit rewards) actually changes the metric's value.\nThe reason is that, when constructing the preference dataset $\\{(x,y_w,y_l)\\}$, two responses $y_1,y_2$ are annotated by the reward model $r$ to determine the winning/losing responses: $y_w,y_l \\in \\{y_1,y_2\\}$, such that $r(x,y_w) > r(x,y_l)$  *(line 182-185)*.\nTherefore, by definitation we have $r(x,y_w) - r(x,y_l) \\ge 0$ and $|r(x,y_w)-r(x,y_l)| = r(x,y_w) - r(x,y_l)$.\nGiven that **the absolute value applied to the external reward gap does not alter the outcome**, there will be no need to conduct additional ablations.\n\nWe will explicitly clarify this point in our revised manuscript to prevent potential confusion.\nThank you for pointing out this problem!\n\n## Ablation II: Varying-k & Bottom-40%\n\nThanks for your insightful suggestions! \nWe have conducted additional experiments to address your feedback on varying selection ratios and the impact of selecting bottom-40% samples. \n\nFor varying k values, we expanded our experiments by selecting different top-k on the default SimPO's Llama v1 dataset using various metrics, and evaluated the resulting models using Alpaca Eval 2.0.\nWe reported both Length-Controlled (LC) win rates, which are more preferable to reduce length bias, and raw Win Rates (WR):\n\n| **Select 20%** | Uniform | $M_r$ | $M_\\pi$ | $M_{AP}$ (ours) |\n|---|:---:|:---:|:---:|:---:|\n| Alpaca LC | 26.02 | 28.95 | 28.46 | **30.66** |\n| Alpaca WR | 28.06 | 30.24 | **31.01** | 30.62 |\n\n| **Select 40%** | Uniform | $M_r$ | $M_\\pi$ | $M_{AP}$ (ours) |\n|---|:---:|:---:|:---:|:---:|\n| Alpaca LC | 30.72 | 36.58 | 33.53 | **37.07** |\n| Alpaca WR | 32.64 | 36.12 | 34.75 | **36.58** |\n\n| **Select 60%** | Uniform | $M_r$ | $M_\\pi$ | $M_{AP}$ (ours) |\n|---|:---:|:---:|:---:|:---:|\n| Alpaca LC | 33.94 | 40.77 | 38.29 | **42.86** |\n| Alpaca WR | 34.72 | 38.14 | 37.28 | **39.60** |\n\nAs shown in the table, our $M_{AP}$ metric continuously outperforms other methods **across different proportions of selected data**.\n\nFor the bottom-40% selection, data was reversely chosen using various metrics from the Llama v2 dataset for SimPO training:\n\n| **Reverse 40%** | Unifrom | $M_r$ | $M_\\pi$ | $M_{AP}$ |\n|---|:---:|:---:|:---:|:---:|\n| Alpaca LC | 41.61 | 40.66 | 37.07 | _34.64_ |\n| Alpaca WR | 36.26 | 33.36 | 32.95 | _29.37_ |\n\nAcross all metrics—when selecting the bottom 40% based on $M_r, -M_\\pi, M_{AP}$—the performance of resultant models was poorer than the uniform baseline, indicating lower data quality through reverse selection.\nNotably, **the lowest performance** observed was with our $M_{AP}$ metric, verifying its efficacy in identifying high-quality datasets **from an opposite standpoint**.\n\nThese experiments affirm the effectiveness of our proposed method, and we will incorporate the findings in our revised version. \n\n## More Iterative Baselines & References\n\nThank you for your valuable suggestions!\n\nIn the iterative preference learning experiments, we augment the default iterative preference learning pipeline by introducing an additional data selection procedure guided by the proposed metric $M_{AP}$.\nThe role of our proposed metric in this process is to **select high-quality preference data** for subsequent training.\nTherefore, it serves as an **orthogonal strategy** alongside existing iterative preference optimization techniques [1,2,3].\nWhile integrating our data selection process with these optimization methods poses interesting prospects, we're afraid the current timeframe does not allow us to explore all potential directions.\n\nWe sincerely appreciate you for highlighting these relevant works. \nWe will include them in our references and discuss potential integrations of our $M_{AP}$ metric with various iterative preference learning methods in the future work section of our revised version.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "The key contributions are about new ideas and results.", "essential_references_not_discussed": "Current references are sufficient, but it would be nice if some works are additionally referred and discussed. Those works are mentioned in the below comments.", "other_strengths_and_weaknesses": "### Pros\n\n1. **Clarity**. Overall, the writing is clear and easy to follow. In addition, the organization of the main draft is well-established.\n2. **Well-motivated problem and intuitive approach.** The selection of good preference data is an interesting problem, and the proposed approach seems to be intuitive and effective. \n\n### Cons\n\n- **More ablation study**. While the authors provide some ablation studies to demonstrate the effectiveness of the proposed ideas, more experiments are required for a rigorous ablation.\n    - In Figure 3, the proposed score (Eq. 11) is only compared with the score in Eq. 10. However, as there are two modification (taking absolute value in external reward gap & taking absolute value in implicit reward gap), it is unclear whether they are really necessary. It would be nice if the authors can provide two additional results for each modification (e.g.,  absolute external reward gap like Eq. 11 & implicit reward gap like Eq. 10 and vice versa).\n    - In section 3.2, all the experiments are conducted across different selection methods under a fixed selection ratio (top-k%, k=40). For more extensive demonstration of the proposed method, it would be nice if the authors provide the additional results by (1) selecting bottom-40% samples and (2) varying k values such as 10% and 70%.\n- **More baselines for iterative preference learning setup**. In figure 6, the authors conduct the experiments about the iterative preference learning setup. As many prior works for online DPO have been proposed for this problem [1,2,3], it would be nice if the authors can demonstrate the effectiveness of the proposed method compared to them. Also, the baselines in Figure 5 (uniform, $M_r$, $M_{\\phi}$ are directly applicable for this experiment, too.\n\n[1] Xiong et al., Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint., ICML 2024  \n[2] Kim et al., Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment., ICLR 2025  \n[3] Chen et al., Bootstrapping Language Models with DPO Implicit Rewards., ICLR 2025", "other_comments_or_suggestions": "Please address the above concerns.", "questions_for_authors": "Please address the above concerns.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7jHR1oNXJu", "reviewer": "Reviewer_gdV3", "summary": "The submission addresses the challenge of selecting high-quality preference data for aligning large language models with human values. It introduces the metric of alignment potential. This metric quantifies the gap between a model’s current implicit reward margin and the target explicit reward margin to identify preference pairs with high potential for improving alignment. By integrating this metric into both offline preference learning and self-play data generation, the methodology effectively selects data that accelerates convergence and enhances overall performance, achieving superior alignment and robust performance gains compared to traditional data quality metrics.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Suggestions & Questions Regarding Figure 1\n\nThanks for your valuable suggestion!\n\nIn Figure 1, we present two examples to compare the existing metrics:\n- **Explicit reward margin**: $M_r = |r(x,y_w) - r(x,y_l)|$, which quantifies **how much $y_w$ is more preferable than $y_l$**. \n- **Implicit reward margin**: $M_\\pi = |\\hat r_\\theta(x,y_w) - \\hat r_\\theta(x,y_l)|$. \nWith $\\hat r_\\theta(x,y)$ indicating the current model $\\pi_\\theta$'s preference evaluation on $y|x$, this measures **how well the model discerns the preference between $y_w$ and $y_l$**. \n\nWhile existing works select data with *larger explicit reward margins* or *smaller implicit reward margins* for training,\nwe illustrate how these metrics can yield contradictory evaluations for the same data in Figure 1—demonstrating inaccuracies since at least one evaluation must be incorrect.\n1. In **Example I**, the chosen response is clearly correct and more preferable, resulting in a substantial explicit reward margin (indicated by the blue bar), and thus deemed \"high-quality\" by $M_r$.\nHowever, the large implicit reward margin (shown by the orange bar) suggests that the current model $\\pi_\\theta$ can already distinguish the preferences between $y_w$ and $y_l$, thus being  \"low-quality\" by $M_\\pi$.\nConsequently, despite what $M_r$ indicates, \nthis data cannot further improve preference learning as the model is already well-aligned on this sample—**a scenario where relying on $M_r$ is misleading**.\n2. **Example II** presents two nearly identical responses $y_w$ and $y_l$, resulting in a very small explicit reward margin, and thus being \"low-quality\" by $M_r$.\nSimilarly, the model cannot tell the preference between such similar responses and produces a small implicit reward margin, which will be evaluated as \"high-quality\" by $M_\\pi$.\nGiven the negligible preference between the responses, this data should be considered as low-quality—**highlighting how reliance on $M_\\pi$ can also be fallacious**.\n\nThese examples underscore the shortcomings of relying solely on a single margin, and motivate us to derive a more propoerly designed metric:\n- Since $M_r$ meaures the difference between $y_w$ and $y_l$, it serves as an **alignment target** indicating **how the preference should be** on the specific data.\n- In contrast, $M_\\pi$ corresponds to the model's **current state**, indicating **how the preference of the current model is** on this data.\n- As shown in the two examples, evidenced by the examples, data quality cannot be determined merely by a large target value ($M_r$) or a small current value ($M_\\pi$); rather, **it is the gap between the current and target preferences that holds significance**.\n- From this insight, our proposed $M_{AP}$ metric evaluates preference data quality by quantifying the gap from the current implicit margin to the target reward margin, thereby measuring the potential for further alignment training.\n\nWe acknowledge that the connection between the contradictions presented in Figure 1 and the core concept of the proposed $M_{AP}$ metric is not sufficiently clear in the current figure. \nTo address this, we will enhance Figure 1 by including more informative text within the figure or title to explicitly highlight that existing metrics fail on these two examples because they solely focus on either the current model or the target value, neglecting the crucial gap between them.\nThank you once again for your insightful suggestion!", "claims_and_evidence": "The primary contribution of the submission is the introduction of the Alignment Potential (MAP) metric as a method to select preferences effectively for model alignment. The authors present a clear and logical derivation of this metric, progressing from the initial metric formulation ($M_1$) through subsequent refinements ($M_+$) until the final $M_{AP}$ formulation. Furthermore, the paper provides empirical evidence demonstrating the superiority of $M_{AP}$ over baseline methods, confirming its practical benefits.", "methods_and_evaluation_criteria": "The proposed $M_{AP}$ metric represents a novel approach to me for selecting high-quality data for LLM alignment. This derivation shows how combining the explicit reward margin with the implicit reward margin results in a unified metric that quantifies the model’s potential for alignment improvement. Furthermore, the empirical evaluation employs benchmark datasets and evaluation metrics that are common practice in the field.", "theoretical_claims": "I have carefully reviewed the proofs presented in the main text as well as those in Section C.1 of the Appendix and have found no errors.", "experimental_designs_or_analyses": "I reviewed the experimental design and analyses presented in the paper. The methodology is sound, with appropriate evaluation protocols across the relevant benchmarks. The authors provide clear comparisons between their proposed approach and existing baselines in both offline and online settings.", "relation_to_broader_scientific_literature": "From my understanding, this paper makes two key contributions in broader scientific literature:\n- Advances in Data Selection for Preference Optimization: The $M_{AP}$ metric refines data selection by effectively distinguishing high-quality from lower-quality preference data, which streamlines the alignment process.\n- Integration of Self-Play Mechanisms: Incorporating self-play supports scalable model refinement through self-generated content, beyond static data selection. The metric enables the effective identification of high-quality preference pairs within self-generated content, supporting continuous model refinement in dynamic training scenarios.", "essential_references_not_discussed": "Not applied", "other_strengths_and_weaknesses": "The clarity of the writing in this submission makes it an engaging and enjoyable read. The logical progression from $M_1$ to $M_+$ and ultimately to $M_{AP}$ is smooth. Additionally, the authors provide an example for calculating the various metrics.", "other_comments_or_suggestions": "I strongly recommend that the authors enhance Figure 1 with more insightful examples to better illustrate the potential impact of this work. The current examples fall short in conveying the broader implications, making it challenging to fully appreciate the significance of the proposed approach. To be more specific, Example I demonstrates a case with a clear ground-truth label that can be easily handled by other methods. By contrast, Example II features two very similar responses, rendering it difficult to discern which one is superior—even with human evaluation.", "questions_for_authors": "Could you please explain in more detail the potential impact of addressing the issues presented in the two examples in Figure 1?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B7VOIn5W2l", "reviewer": "Reviewer_JxER", "summary": "This paper investigates the techniques of preference data selection, particularly between the explicit reward margin by reward models and the implicit reward margin by the SFT models. This paper proposes a new preference pair quality metric (MAP) based on the discussion between the above two reward margins, enforcing the high explicit reward margin and the low implicit reward margin simultaneously. The experiments show that, using MAP as the preference data selector, alignment quality is improved compared with other selection methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## The $M_+$ Metric\n\nThe $M_+$ metric is a modified version of the proposed $M_1$ metric designed to adapt to the **unidirectional update nature of existing preference optimization methods** (e.g. DPO, SimPO).\n\nAs described in section 3.1, we first quantify the discrepancy between the current model and the aligned optimum on the preference data $(x,y_w,y_l)$ via the $M_1$ metric: \n$$M_1(x,y_w,y_l;\\theta,r^*) = |(r^*(x,y_w) - r^*(x,y_l)) - (\\hat r_\\theta(x,y_w) - \\hat r_\\theta(x,y_l))|.$$\nTo ensure practicality, we then introduce two key adaptations to the $M_1$ metric,  and the first one is termed **unidirectional calibration** (Lines 182-202). The rationale behind this adaptation is explained as follows:\n\nIn the optimization process of DPO, the loss function $-\\log\\sigma(\\hat r_\\theta(x,y_w) - \\hat r_\\theta(x,y_l))$ tends to **unidirectionally increase the implicit margin term** $\\hat r_\\theta(x,y_w) - \\hat r_\\theta(x,y_l)$ during optimization. \nSo we would like to select data with a low implicit margin term requiring to be increased through training.\n\nHowever, the original $M_1$ metric measures a **bidirectional gap** between $\\hat r_\\theta(x,y_w) - \\hat r_\\theta(x,y_l)$ and $r^*(x,y_w) - r^*(x,y_l)$, which can lead to selection of data with either a very high margin or a very low margin.\nWhile data with a low margin term, $\\hat r_\\theta(x,y_w) - \\hat r_\\theta(x,y_l)$, is desirable,\ndata with a high margin term will be further increased by the DPO training process, which is counterproductive.\n\nTo address this, we opted to remove the absolute value from $M_1$, implementing what we refer to as unidirectional calibration, and formulated a revised metric, $M_+$:\n$$M_+(x,y_w,y_l;\\theta,r^*) = (r^*(x,y_w) - r^*(x,y_l)) - (\\hat r_\\theta(x,y_w) - \\hat r_\\theta(x,y_l)).$$\n\nThis new metric effectively selects data with a minimal implicit margin term $\\hat r_\\theta(x,y_w) - \\hat r_\\theta(x,y_l)$ to increase, **aligning with the unidirectional nature of preference optimization methods**.\n\nWe hope this elucidation clarifies the intent and application of the $M_+$ metric within our study. \n\n## Readability\n\nThank you for highlighting this issue.\nWe will strive to enhance the readability in our revised version.\nSpecifically, we plan to: (1) merge the two $M_{AP}$ equations (Eq. 11 and Line 236) to reduce redundancy and enhance clarity; (2) include a concise overview of our metric formulation at the beginning of Section 3 to guide readers through our methodology with greater ease.\nThese changes aim to streamline the presentation and make the content more coherent to readers.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "The proposed MAP data selector is clear and easy to follow. The evaluation designs are convincing.", "theoretical_claims": "I did not check the correctness of DPO convergence, since I am not an expert in this field.", "experimental_designs_or_analyses": "1. The comparison between different preference data selection methods.\n\n2. GPT-4 agreement on the selected preference pairs.\n\n3. The DPO and SimPO alignment results in the self-play setting.", "relation_to_broader_scientific_literature": "This paper provides an easy yet effective preference data selection method, which is essential for constructing good preference datasets.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "1. Although the scope of this paper is quite narrow, this paper still provides an in-depth discussion of the reward margins, which I think is helpful for readers to understand the literature.\n\n2. I am surprised by the simplicity of the proposed method. I think that is the reason why this paper focuses on too many technical details and is a little bit hard to read.", "other_comments_or_suggestions": "In L216, what is the M_+ method? It is not clearly stated before this line.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uVkDljBsO8", "reviewer": "Reviewer_66Hn", "summary": "This paper examines how reward margins influence preference data selection in LLM alignment. It introduces a novel metric, Alignment Potential (AP), which integrates both explicit reward margins (provided by the reward model) and implicit reward margins (derived from the policy and reference models). Furthermore, it extends AP to self-play data generation. Extensive experiments demonstrate that AP outperforms leading alignment baselines across various settings.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Additional Alignment Approaches\n\nThanks for your suggestion! \nAside from the DPO and SimPO methods, we conduct additional experiments using the IPO method for alignment training.\n\nLike the setting in Section 3.2, we utilize different metrics to select top-40% subsets from existing preference datasets for subsequent IPO training. \nWe utilize the two Llama-based datasets for experiment, benchmark the trained models on Alpaca Eval 2.0, and report both Length-Controlled (LC) win rates and raw Win Rates (WR):\n\nOn the default Llama dataset of SimPO (based on the PairRM reward model):\n\n| **Llama+IPO** | Uniform | $M_r$ | $M_\\pi$ | $M_{AP}$ (ours) |\n|---|:---:|:---:|:---:|:---:|\n| Alpaca LC | 30.49 | 34.15 | 35.04 | **35.44** |\n| Alpaca WR | 34.15 | 34.08 | 35.40 | **36.08** |\n\nOn the Llama v2 dataset of SimPO (based on the ArmoRM reward model):\n\n| **Llama-v2+IPO** | Uniform | $M_r$ | $M_\\pi$ | $M_{AP}$ (ours) |\n|---|:---:|:---:|:---:|:---:|\n| Alpaca LC | 33.17 | 33.92 | 38.19 | **40.35** |\n| Alpaca WR | 30.63 | 30.17 | 35.73 | **38.14** |\n\nAs shown in the table, our proposed $M_{AP}$ metrics still **outperform existing baselines under this new alignment method**, further certifying its effectiveness.\n\n**Reference**\n- IPO: Azar, Mohammad Gheshlaghi et al. “A General Theoretical Paradigm to Understand Learning from Human Preferences.” ICML 2024.", "claims_and_evidence": "The claims are well-supported by both theoretical analysis and experimental results.", "methods_and_evaluation_criteria": "The proposed method is well-aligned with the LLM alignment problem, focusing on contrast strategies in preference data selection—specifically, whether larger or smaller reward margins should be used. AP is a theoretically and empirically grounded metric for assessing preference data quality.\n\nTo evaluate AP’s effectiveness, the paper employs well-established preference optimization benchmarks (ApacaEval2, Arena-Hard) and diverse model architectures (Llama3-8B, Gemma2-9B).", "theoretical_claims": "The proofs, including those in the appendix, have been reviewed, and no issues were found.", "experimental_designs_or_analyses": "The authors first conduct preliminary experiments to assess different metrics (explicit reward margin, implicit reward margin, and AP) in the context of LLM alignment, supported by theoretical analysis. The experiments are then extended to a self-play data generation framework, with clear comparisons against baseline methods (DPO, SimPO with different self-play strategies) across various model architectures (Llama3-8B, Gemma2-9B) and dataset sizes (10k–60k). The experimental design is thorough and convincingly demonstrates AP’s effectiveness.", "relation_to_broader_scientific_literature": "1.\tThis paper builds on research in LLM alignment, particularly in direct preference optimization (DPO) and self-play preference generation.\n2.\tIt explores a critical problem in LLM alignment—how contrast strategies (explicit and implicit reward margins) in preference data selection impact alignment. AP is proposed as a novel strategy that integrates these reward margins to provide a more effective metric.\n3.\tBeyond empirical validation, the paper provides theoretical analysis to justify AP’s ability to identify high-quality alignment data.", "essential_references_not_discussed": "To my knowledge, this paper has included sufficient references.", "other_strengths_and_weaknesses": "Strengths\n\n-\tThe paper is well-written, with clear motivation and an intuitive structure.\n-\tIt establishes connections between existing data selection strategies (explicit and implicit reward margins).\n-\tPreliminary experiments, paired with theoretical analysis, support AP’s effectiveness.\n-\tExtensive experiments consistently show AP’s superior performance across LLM architectures, benchmarks, and dataset sizes. Notably, models trained on only AP-selected data (e.g., 30k samples) outperform those trained on full-sized datasets.\n\nWeaknesses\n\n-\tAP is tested only with DPO and SimPO. While the experiments are thorough and convincing, evaluating AP with additional alignment approaches would strengthen the results.", "other_comments_or_suggestions": "See weaknesses.", "questions_for_authors": "See weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rUovXjRvOY", "reviewer": "Reviewer_ZHMc", "summary": "This paper proposes the new \"alignment potential metric\" to evaluate the quality of (and select) data for offline preference optimization. This metric quantifies the gap between the model's implicit reward margin and the target explicit reward margin, and thus aims to estimate the model's potential to align on the preference data by measuring how much the model can improve its preference discrimination. Using standard DPO and SimPO preference learning setups, the paper compares this preference data selection metric against existing baseline metrics (e.g., using only the model's implicit reward margin, and only the explicit reward margin), and shows that this alignment potential metric consistently outperforms. The paper also provides a theoretical justification for why training converges faster when selecting data using the alignment potential metric, versus random data selection.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Due to space limits, we put all tables in this anonymous link: https://anonymous.4open.science/r/tables-11915/ZHMc.md.\n\n## Large/Small Margins\nTo reiterate, the primary claim of our paper is **not** that \"large explicit reward margin\" and \"small implicit reward margin\" **individually** imply high-quality data.\nInstead, the key motivation for our work is that **relying solely on either of these margins is insufficient** for assessing data quality. \n\nAs demonstrated in Figure 1, we evaluate instances with high explicit reward margins or small implicit margins as low-quality data (Line 71-74).\nThe main idea of our proposed metric is to consider the **gap between the model's current implicit margin and the target explicit margin** to measure the potential for alignment learning.\n\nHowever, if assessing data quality using a single margin is *required*, we acknowledge that a higher explicit margin or smaller implicit margin can be beneficial. This notion is supported by recent works (Line 40) and our experiments (Figures 4, 5, and 8).\nIn response to your suggestions, we've conducted additional experiments by following the setup of Figure 4, but **reversely selecting data** via these metrics.\nAs shown in **Table I of the linked page**, reversely selecting training data indeed results in poorer performance, thus verifying these metrics' efficacy from an opposite standpoint.\n\n## Too Large Gaps\nWhile a larger gap might require more intensive alignment training, the direction of the update remains correct. \nMoreover, the common practice in preference dataset construction for alignment training, such as SimPO, involves **sampling responses $y_w$ and $y_l$ using the model that is being trained**. \nThis practice ensures that both the chosen and rejected responses are **within the model's generation capabilities**, thus mitigating the label exposure bias problem. \n\n## Different Reward Models\nIn our study, we have indeed considered the impact of different reward models by incorporating two RMs: **PairRM (0.4B) and ArmoRM (8B)**, for preference annotation and metrics evaluation. \nFigures 4 and 8a in the paper illustrate the results based on datasets annotated using ArmoRM and PairRM, respectively. \nAs shown in the figures, **our proposed $M_{AP}$ metric outperforms existing baselines under both RMs**, attesting to the effectiveness of $M_{AP}$ metric under different reward models.\n\n## Reward Noise & Ranking\n**The ranking disagreement might not be the key determining the data quality**, due to the reward noise issue.\nAs suggested, we select 40% data subsets in which the ranking of explicit and implicit rewards are opposite. \nWe compare this strategy with existing metrics on both Llama datasets as described above.\n\nAs shown in **Table II of the linked page**, although selecting data with **contradictory ranking** between explicit and implicit rewards can improve performance to some extent, it **cannot outperform any of the existing baselines**.\n\nSuch an observation is consistent with our motivation of \"Reward noise regularization\".\nAs discussed in Line 212-215, given the LLM's ability to discern certain preferences, **the opposite ranking** between explicit and implicit rewards—suggesting contradictory preference judgments between the RM and the LLM—may indicate **a higher risk of noisy reward model annotations**, and thus resulting inferior model performance.\n\n## Other Directions\nWe sincerely appreciate your suggestions for exploring the *non-linear mapping, LLM-as-Judge, weighted loss and curriculum learning* settings.\nWhile we believe they hold promise for future research, the limited timeframe for the current study does not permit us to explore all these directions, so we plan to investigate them in future work.\n\n## Novelty\nWhile existing methods focus solely on either explicit or implicit margins and often lack in-depth explanations, our work introduces a key innovation by measuring the data quality with **the gap between the model's aligned optimum and the current model's preferences**.\nThis gap serves as a **theoretical basis** for evaluating data quality in alignment training, distinguishing our approach from prior work.\n\nAlthough the final form of $M_{AP}$ incorporates the two existing metrics ($M_r$ and $M_\\pi$), **its core idea of gap quantification and the logical derivation of our approach are novel aspects not explored in existing metrics**., which contributes significantly to the advancement and originality of this work.\n\n## Training Details & Significance Testing\nWe report the training time, standard deviations and confidence intervals of current evaluations in **Table III & IV of the linked page**.\n\n## References & Presentation\nThanks for your suggestions, we will make the following revisions:\n1. Include relevant references of data selection in general LLM tasks\n2. Merge repetitive equations and figures\n3. Revise sections by splitting the Discussion section into separate Conclusion and Limitations sections", "claims_and_evidence": "A major claim is that \"large explicit reward margin\" and \"small implicit reward margin\" both imply \"high quality\" datapoints for preference optimization. These claims are substantiated by citations in the introduction, but since they are both very general claims and are central to the motivation for the paper's new \"alignment\" potential metric\", it would be more convincing if the results and/or theoretic basis supporting these claims were at least summarized in the paper, in the context of the paper's own setup. Especially the claim that data for preference optimization should be selected via the \"small implicit reward margin\" metric is not obvious; e.g., perhaps a curriculum learning setup where the allowed implicit reward margin increases over the course of training would outperform. It would be informative to include an ablation to show that selecting datapoints for preference optimization using \"small explicit reward margin\" and \"large implicit reward margin\" metrics does not work well.\n\nAnother related and central claim in the paper is that a larger gap between the target explicit reward margin and current implicit reward margin indicates a greater potential for improvement. But the paper does not address the potential limitation that if the gap is too large, learning the correct alignment may be out of reach of the model (label exposure bias problem).", "methods_and_evaluation_criteria": "- The M_{AP} metric crucially depends on (the quality of) an external reward model, and measures the extent to which the model's preference margin matches the ground truth margin as given by this external reward. The effect of the reward model used for data selection is not explored in the main text. \n  - To what extent does the quality of the M_{AP} metric depend on the quality of the explicit reward model? (This model is being considered as a proxy for the gold rewards.)\n  - An assumption in the formulation of the M_{AP} metrics is that all reward margins of equal magnitude imply an equal quality gap. But in practice, the mapping between quality and reward model score is often non-linear, and a large reward margin at the upper versus lower end of the quality spectrum may imply different actual differences in quality.\n  - Can the same LLM generating the responses be used as a prompted LLM-as-a-Judge reward model to get the \"explicit rewards\"? If not, what are the required characteristics of the external reward model for this method to yield gains?\n- The \"Reward noise regularization\" subsection (Section 3.1) is poorly motivated. The regularization simply consists in taking the absolute value of the explicit and of the implicit margins. In the extreme (for a certain explicit reward model), this metric could select examples for which the model always agrees with the external reward about the ranking of the preferred and dispreferred responses, which is intuitively the opposite of what M_{AP} purports to do. So the effectiveness of this regularization is likely highly dependent on the characteristics of the external reward being used.\n- The benchmarks and evaluation metrics used were reasonable, and are standard in the preference optimization literature.", "theoretical_claims": "Theoretical results (Theorem 3.1, proof in Appendix) show that training converges faster when using M_{AP} for data selection, vs random sampling. This result is novel and insightful.", "experimental_designs_or_analyses": "- The paper compares using M_{AP} against using only the model's implicit reward margin, and only the explicit reward margin, as data selection metrics. The paper does not address whether the relative magnitude of the implicit versus explicit margins is important, though, or whether it is primarily the ranking (dis)agreement that matters. Another insightful baseline would be to compare using M_{AP} for data selection against using a ranking (rather than margin) metric, which selects all examples for which the explicit and implicit rewards disagree on the ranking of the preferred versus dispreferred response. (Within this subset, examples could be randomly chosen to match the expected dataset size.)\n- How does data selection with M_{AP} compare to weighting the loss by the implicit margin? And can train-time loss weighting provide incremental improvements on top of offline data selection with M_{AP}?\n- The proposed M_{AP} metric selects a fixed set of examples offline before preference learning (or during multiple iterations of training, as in Section 5). However, the paper does not explore a curriculum learning setup. How does performance from selecting a fixed set of examples with a large M_{AP} margin compare to (smoothly) varying the reward margin of the examples over the course of training?\n- How long were the DPO/SimPO models trained for, and how was the checkpoint to use for evaluation chosen?", "relation_to_broader_scientific_literature": "This paper contributes to the body of literature on data selection for post-training/alignment. There has been a lot of recent interest and work in this area, not only for selecting datasets for preference optimization (e.g., DPO), but also for selecting SFT and RL datasets.", "essential_references_not_discussed": "While the paper cites related work in preference optimization dataset selection, it does not cite other works which investigate metric-guided data selection for training LLMs more generally. For example, [this paper](https://arxiv.org/pdf/2311.05350) (among others) shows that using QE metrics for data filtering improves the quality of machine translation models. The proposed M_{AP} method also uses metric-guided filtering, just in the (paired) preference data setting.", "other_strengths_and_weaknesses": "Strengths:\n- The results look strong. The proposed M_{AP} method for preference data selection consistently outperforms other metrics, across several experiments in which data is selected both from fixed and iteratively evolving datasets.\n- The paper shows that by selecting data with M_{AP}, training converges faster than with baseline data selection methods. Moreover, the performance gains are consistent across dataset sizes.\n\nWeaknesses:\n- The proposed M_{AP} metric is extremely simple and not very novel (very small incremental contribution on top of existing work).\n- There is no significance testing to accompany the experimental results (and, for example, the results using M_{pi} versus M_{AP} in Figure 5 look very close).\n- The Introduction is repetitive and introduces concepts/intuitions which are redundant with those presented in the Methods section (Section 3).\n- The entire Related Work section is in the Appendix.", "other_comments_or_suggestions": "- The M_{AP} equation is repeated multiple times throughout the paper. Is this necessary, or can equation (11) just be referenced?\n- Heavily overlapping results between Figure 1b and Figures 4/5\n- The Discussion section contains nothing more than the Conclusion + Limitations and future work.", "questions_for_authors": "No additional questions besides those posed in previous sections (especially see the \"Methods And Evaluation Criteria\" and \"Experimental Designs Or Analyses\" sections).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PMq1KluC1o", "reviewer": "Reviewer_bqyD", "summary": "This paper identifies that existing metrics for selecting preference data, which rely on either explicit reward margins or implicit reward margins, often yield contradictory evaluations for the same dataset. To address this issue, the authors propose a new metric called the alignment potential ($M_{AP}$), which quantifies the gap between a model’s current implicit reward margin and a target explicit reward margin. Empirical results demonstrate that training on data selected by $M_{AP}$ consistently enhances alignment performance, outperforming existing metrics. Furthermore, the proposed metric can be integrated into self-play data generation frameworks.\n\n## Update After Rebuttal\n\nThanks for the authors' effort during the rebuttal. I still have concerns regarding the computation of the implicit reward margin. As the authors replied, “The implicit margins, $M_\\pi = |\\hat r_\\theta(x,y_w) - \\hat r_\\theta(x,y_l)|$, are indeed computed based on the initial model (i.e., the SFT model $\\pi_\\text{SFT}$).” However, based on prior work and empirical observations, an SFT model typically assigns very similar implicit rewards to different responses, indicating a lack of meaningful discrimination. Thus, I remain skeptical about this formulation. While I understand that the implicit reward may gradually become more distinguishable as training progresses, the initial SFT model itself is generally not capable of such fine-grained reward estimation. In fact, upon revisiting the cited paper [1], I found that the implicit reward in that work was not computed using the SFT model, but rather using a trained model, with the SFT model serving only as a reference. A raw SFT model, not yet trained via DPO or other preference optimization methods, lacks the necessary discriminatory ability. This remains my primary concern with the paper. I would appreciate a discussion on this issue in future revisions, particularly addressing how the results might differ if a trained model were used to compute the implicit reward instead of an untrained SFT model, especially for both the baseline and the proposed method. \n\n[1] Not All Preference Pairs Are Created Equal: A Recipe for Annotation-Efficient Iterative Preference Learning", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Contradictions\nWe have conducted an analysis to measure the contradictions between explicit and implicit margin metrics ($M_r$ and $M_\\pi$) by comparing the Jaccard similarity of subsets selected by each metric:\n- **Subset selection**: Using $M_r$ and $M_\\pi$, we select the top-rated k% subsets from existing datasets, denoted as $D_r$ and $D_\\pi$.\n- **Jaccard similarity**: We calculate the similarity between the selected subsets using: \n$$\nJ(D_1,D_2) = \\frac{|D_1 \\cap D_2|}{|D_1 \\cup D_2|}.\n$$\n- **Comparison**: We compare the resultant similarity $J(D_r, D_\\pi)$ with two uniformly selected subsets $J(D_{u1},D_{u2})$. A lower similarity $J(D_r, D_\\pi) < J(D_{u1},D_{u2})$ would indicate contradictions between $M_r$ and $M_\\pi$.\n\nWe measure the similarities on all three datasets in our paper:\n|Jaccard (%)|Top-5%|Top-10%|Top-20%|\n|-|:-:|:-:|:-:|\n|Gemma|1.28|3.16|7.67|\n|Llama v1|2.12|4.82|10.59|\n|Llama v2|1.42|3.46|8.25|\n|_Uniform_|2.62|5.44|11.08|\n\nAs shown in the table, the similarities $J(D_r, D_\\pi)$ are **consistently smaller than those of uniformly selected subsets** across varying top-k ratios and datasets. This observation validates the claim that explicit and implicit margin metrics can **indeed provide contradictory evaluations for preference selection**.\n\n## Smaller Implicit Margins\nExplanation of implicit reward margins:\n- **Computation**: The implicit margins, $M_\\pi = |\\hat r_\\theta(x,y_w) - \\hat r_\\theta(x,y_l)|$, are indeed computed based on the initial model (i.e., the SFT model $\\pi_\\text{SFT}$).\n- **Motivation**: **The implicit margin shows the LLM's ability to discern preferences**. A small implicit margin indicates the model initially lacks the ability to distinguish such preference, making the data suitable for alignment training. \n\n**Empirical evidence**: We measure the distributions of implicit margins before and after the alignment training on the Llama v2 dataset.\nThe results, as shown in Figure I in this [anonymous link](https://anonymous.4open.science/r/tables-11915/bqyD.md), demonstrate a notable increase in $M_r$ after alignment training. \nSpecifically, the average $M_r$ rises from `0.072` to `0.742`, indicating **improved preference recognition ability** of the model.\n\nGiven these insights, data with relatively smaller initial implicit margins generally indicates that **the LLM cannot tell the preference, thereby requiring further training** to align on such data.\n\n## Method Implementation\nOur SimPO-based implementation of the implicit reward is because:\n1. The implicit reward $\\hat r_\\theta^{Sim}(x,y) = \\beta\\log\\frac{\\pi_\\theta(y|x)}{|y|}$, has been demonstrated to be effective in SimPO's paper.\n2. The formulation does not require additional conditions (e.g. reference model), allowing for its direct application in different optimization methods, e.g., DPO, IPO.\n\nThroughout our paper, **the implicit reward in our $M_{AP}$ metric is consistently calculated using the SimPO formulation** $\\hat r_\\theta^{Sim}(x,y),$  regardless of whether the training method is SimPO or DPO.\nAnd the empirical results also validate its effectiveness across different training settings.\n\n## More Baselines\nIn response to your suggestions, we conduct additional experiments to include uniform and $M_\\pi$ baselines under Table 1's setting, and here are the results of Llama models on Alpaca Eval:\n\n|Llama+DPO|Uniform|$M_\\pi$|$M_{AP}$|\n|-|:-:|:-:|:-:|\n|Alpaca LC|40.86|42.00|**42.82**|\n|Alpaca WR|42.66|44.71|**45.00**|\n\n|Llama+SimPO|Uniform|$M_\\pi$|$M_{AP}$|\n|-|:-:|:-:|:-:|\n|Alpaca LC|40.69|42.39|**43.76**|\n|Alpaca WR|38.77|41.62|**42.00**|\n\nAs shown in the tables, our $M_{AP}$ metric consistently outperforms existing baselines for both optimization methods.\n\n## Hyperparameters\nThe value of hyperparameter $\\alpha$ can be found in Line 991 and Line 1008 of Appendix D.1, where we generally set the value within {0.5, 1, 2.5, 5.0}.\n\n## Explain Margins & Motivation\n\nWe would like to clarify that our paper is not centered on the notion that \"large explicit reward margin\" and \"small implicit reward margin\" **individually** imply high-quality data.\nInstead, one key motivation for our work is that **relying solely on either of these margins is insufficient** for assessing data quality. \nAs demonstrated in Figure 1, the two instances with high explicit reward margins or small implicit margins are evaluated as *low-quality* data by our metric (Line 71-74).\n\nThe core idea of our metric is to quantify the data quality by measuring **the discrepancy between the current model's preference and the aligned optimum** on the data.\nThe alignment optimum is captured through explicit reward margins (Equation 8), and the model's preference is indicated by implicit reward margins.\nSo the proposed metric $M_{AP} = |r(x,y_w) - r(x,y_l)|-|\\hat r_\\theta(x,y_w) - \\hat r_\\theta(x,y_l)|$ quantifies the **gap from the current implicit margin to the target explicit margin**, thereby indicating the potential for alignment on the given data.", "claims_and_evidence": "1. The authors claim that \"while existing metrics primarily assess data quality based on either explicit or implicit reward margins, they often provide contradictory evaluations for the same data\" as the main motivation of this paper. However, this claim is only illustrated through two specific cases in Figure 1(a). There is a lack of quantitative analysis to determine how frequently such contradictions occur in commonly used preference datasets. Conducting a quantitative study on the proportion of \"contradictory\" data points in widely used datasets would strengthen the motivation of this work.\n\n2. It is unclear whether the implicit reward margins used in this work are computed from a policy that has already been optimized on the preference dataset, or if they are all derived directly from an SFT model as mentioned on Page 5. If the latter is the case, it raises concerns, as an SFT model is expected to be relatively neutral in its preferences for all the preference data. This leads to the question of how the distribution of implicit reward margins differs before and after optimization? Additionally, in line with the first concern, it would be useful to quantify the proportion of data points where implicit reward margins contradict explicit reward margins. The choice of implicit reward margins is crucial to the paper’s claims—why should smaller implicit reward margins be preferable?", "methods_and_evaluation_criteria": "Is the proposed method limited to SimPO, or can it generalize to other approaches? For instance, in standard DPO, the implicit reward margins computed from an SFT model are always zero. How would this affect the applicability of the proposed $M_{AP}$ metrics in such settings?", "theoretical_claims": "Most of the theoretical results appear to be correct upon review. However, the detailed proofs provided in the appendix were not fully verified.", "experimental_designs_or_analyses": "1. In Table 1, the primary experiments only compare $M_{AP}$ against baselines that use explicit reward margins $M_r$ for data selection. However, there is no baseline that selects data based on implicit reward margins $M_\\pi$. \n\n2. The experiments mention the hyperparameter $\\alpha$ for the proposed $M_{AP}$, but do not report the specific values used across different experiments. Providing this information would improve the reproducibility of the results.", "relation_to_broader_scientific_literature": "Can you provide a theoretical or intuitive explanation of why a larger explicit reward margin is always better, while a smaller implicit reward margin is preferable? A clear justification for this assumption would strengthen the correlation to previous works and strengthen the motivation of this paper.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Overall, this paper is well-organized and easy to read. However, as noted in the Claims and Evidence and Experimental Designs or Analyses sections, there are still some concerns regarding the lack of quantitative validation for key claims and missing baselines.", "other_comments_or_suggestions": "NA", "questions_for_authors": "Please see above. I am very happy to discuss with the authors during the rebuttal process and look forward to having the above concerns and questions addressed.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper received 6 reviews since some originally-assigned reviewers submitted the review after the deadline. After carefully checking the paper, the reviews, the rebuttal, and the author-reviewer discussions, I think the strong points outweigh the weak points, which can be fixed in the camera-ready version. Thus, I recommend accepting this paper.
Decision: Accept (poster)