Less is More: Improving LLM Alignment via Preference Data Selection
Authors: Xun Deng
Han Zhong
Rui Ai
Fuli Feng
Zheng Wang
Xiangnan He
Venue: NeurIPS 2025 spotlight ()
Keywords: data selection
DPO
alignment
Primary area: deep_learning

Abstract: Direct Preference Optimization (DPO) has emerged as a promising approach for aligning large language models with human preferences. While prior work mainly extends DPO from the aspect of the objective function, we instead improve DPO from the largely overlooked but critical aspect of data selection. Specifically, we address the issue of parameter shrinkage caused by noisy data by proposing a novel margin-maximization principle for dataset curation in DPO training. To further mitigate the noise in different reward models, we propose a Bayesian Aggregation approach that unifies multiple margin sources (external and implicit) into a single preference probability.  Extensive experiments in diverse settings demonstrate the consistently high data efficiency of our approach. Remarkably, by using just 10\% of the Ultrafeedback dataset, our approach achieves 3\% to 8\% improvements across various Llama, Mistral, and Qwen models on the AlpacaEval2 benchmark.  Furthermore, our approach seamlessly extends to iterative DPO, yielding a roughly 3\% improvement with 25\% online data, revealing the high redundancy in this presumed high-quality data construction manner. These results highlight the potential of data selection strategies for advancing preference optimization.

Less is More: Improving LLM Alignment via Preference Data Selection
Xun Deng" 14, Han Zhong"; Rui AI Fuli Feng"; ZZeng Wang', Xiangnan He University of Science and Technology of China, "Peking University, Massaahusettss Institute of Technology, "Alibaba Cloud Computing Moo Key Lab of BIPC, University of Science and Technology of China dx9812280mail ustc. edu. cn, hanzhong@stu, pku edu. cn, ruiai @mit edu [fulifeng93, aeariiiiiinnnnnaa xiangnanne))@gaill com, vz3887790alibaba innaa-inn...e com
Abstract
Direct Preference Optimization (DPO) has emerged as promising approach for aligning large language models with human preferences. While prior work mainly extends DPO from the aspect of the objective function, we instead improve DPO from the largely overlooked but critical aspect of data selection. Specifically, we address the issue of parameter shrinkage caused by noisy data by proposing novel maaiiiizimiiiiiiiiionn principle for dataset curation in DPO training. To further mitigate the noise in different reward models, we propose a Bayesian Aggregation approach that unifies multiple margin sources (external and implicit) into single preference probability. Extensive experiments diverse settings demonstrate the consistently high data efficiency of our approach. Remarkably, by using just 10% of the Ultrafeedback dataset, our approach achieves 3% to 8% improvements across various Llama, Mistral, and Qwen models the AlpacaEval2 benchmark. Further- more, our approach seamlessly extends to iterative DPO, yielding a roughly 3% improvement with 25% online data, revealing the high redundancy this presumed high-quality data construction manner. These results highlight the potential of data selection strategies for advancing preference optimization.
Introduction
Reinforcement Learning from Human Feedback [RLHF: 6 65 has emerged as crucial technique for aligning Large Language Models (LL.Ms) with human preferences and values. Traditional RLHF im- plementations involve two-stage process; reward model training based preference data followed by reinforcement learning optimization. However, this approach presents significant computational challenges, requiring loading multiple model instances and extensive hyperparameter tuning.
As an alternative, 39 introduced Direct Preference Optimization (DPO), which streamlines the aligmment process by directly optimizing the LLM policy from preference data. DPO has demon- strated comparable effectiveness while substantially reducing computational requirements compared to classical RI.HF, Following DPO's introduction, numerous studies have proposed improvements through modified learning objectives (22 [5] and iterative learning schemes (53). While these algorithmic advances have shown promise, there remains a critical gap in our understanding of the data- data-centric aspects of preference learning: what characteristics of preference data contribute most to model alignment?
This work thoroughly studies the impact of preference data quality on DPO training, which is crucial for developing more efficient training strategies. In particular, we achieve both improved performance
"Corresponingg author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

and reduced computational costs through strategic data selection. Our research makes three primary contributions:
(1) We prove in theory the necessity of data selection in the presence of exogenous noise. Specifically, the noise in the reward model may flip the preference between response pairs, leading the emergence of the parameter shrinkage issue. Furthermore, we demonstrate that margin based selection criteria can effectively address this issue by inducing parameter inflation.
(2) Driven by the theoretical results and the derived margin- maximizaaiiinnnn principle, we propose a Bayesian Aggregation for Preference data Selection (BeeS) strategy. BeeS incorporates signals from both external rewards and DPO implicit rewards, and deprioritizes a preference pair it exhibits a low reward margin from any single reward source to mitigate potential noise. Through extensive experiments across diverse datasets and models, we show that this selection strategy shows two consistent advantages: it substantially reduces computational overhead via efficient data selection and improves model performance compared to training on the full dataset. In particular, on the UltraFeedback dataset and its variants, our method identifies 10% data subset for DPO training on LLama, Mistral, and Qwen series models, consistently achieving 3% to 8% point improvements on the AlpacaEval 2.0 benchmark relative training on the complete dataset.
(3) Finally, we extend our data selection framework to iterative DPO settings, showing that selectively sampling online data can simultancously lower computational costs and improve performance. In particular, we achieve 48.499 49.49 win rate and 54.99% length-conte win rate on the AlpacaEval 2.0 benchmark using only 25% of the online data for training.
Our findings provide both theoretical insights into the dynamics of preference learning and practical guidelines for more efficient DPO implementations. This work bridges an important gap between algorithmic innovation and data quality considerations the context of LLM alignment.
1.1 Related Work
Preference learning algorithms. Reinforcement Learning from Human Feedback also known as dueling RL (37 or preference-based RL [5), has become a crucial component recent Large Language Models (LLMs) such as ChatGPT (36]. While the classical RLHF pipeline traditionally uses Proximal Policy Optimization, several alternative approaches have been proposed. These include but not limited other RL-based training algorithms 28 63), rejection sampling (1I conditional supervised fine-tuning [31] [56 60), and Direct Preference Optimization [39] Among these alternatives, DPO has gained significant attention due its simplicity and robust performance.
Following the introduction of DPO, numerous works 662 I 15 32 to improve its performance by modifying the DPO objective.
18
19
51 have attempted
Data selection in LLM Fine-tuning. Data selection is crucial in LLM post-training 48 for two key observations: post-training typically converges rapidly, and excessive data can degrade model performance through overfitting or exposure to toxic content 43 TO) Recent research has focused on enhancing instruction tuning efficiency by identifying high-quality subsets from large instruction datasets [4, 25 (2), often adapting active learning query strategies (40) to assess sample uncertainty and diversity. However, data efficiency in preference learning remains relatively unexplored. Prior studies have studied reducing annotation costs in preference dataset creation 35 [7) and on scenarios involving numerous ranking annotations 45 34] Other research aims improve model's ability to distinguish between two responses by adding a margin to the loss term [32 38 [ Additionally, concurrent work highlights the importance of margins for preference data filtering, though there is
debate on whether hard samples help or hinder preference learning [50 22
59 [6]
Our work firstly provides clear criteria for identifying informative samples while filtering toxic ones, thereby improving both DPO's efficiency and performance. Furthermore, our method extends to iterative DPO (53] and its variants (61), wherein training data is dynamically generated by the model during its iterative training process.
2 Background
Reinforcement Learning from Human Feedback (RLHF) has emerged as a key method for aligning LLMs with human preferences. It leverages training data of the form D {I, ywo y1 ), where x rep-

---

Preference Dataset
Preference Aligned model
Target model
Randemly select 2k samples
DPO training
Good Subset
Subset
Selecting margin samples
DPO training
Normalization + Bayesian aggregation
LLM A (small)
LLM B
Fyla (q) xx(a19)
Implicit roward model: r(g.a)
Margin calculation: r(g. r(g.a)
Denoised Margin distribution
External reward models: r(q.a)
Figure 1: The workflow of the BeeS method.
resents the input prompt, and Ye and y denote the preferred and dispreferred responses, respectively, The RLHF pipeline typically involves two stages: reward learning and policy optimization.
Reward Learning. In the reward learning stage, a reward model is trained to approximate human preferences based preference data. By adopting the Bradley-Terry model [ capture human preference, reward training involves minimizing the loss:
CRM (r) = -E -E log a (r(z, Mw) r(x, u)))
where (.) is the sigmoid function.
Policy Optimization with Reinforcement Learning. Once the reward model r is trained, it is used to guide the optimization of policy To (v(r),, where 9 denotes the parameters of the model. This stage often employs reinforcement learning techniques such Proximal Policy Optimization [PPO; 41 to optimize the policy by maximizing the expected reward.
To (o(y)r))
max E Egw (. r(xx,) v) 1000--logg -Blog Tref (y/ xx)
70
where 3 0 the regularization parameter. However, this RI. approach be computationally expensive, sensitive to reward misspecification and require careful hyperparameter tuning.
Recently, as alternative to the RL-based policy optimization in RLHF, Direct Preference Opti- mization [DPO; (9) has been proposed. DPO simplifies the reward alignment process by directly incorporating human preference data into supervised training. Instead of defining and optimizing reward function explicitly, DPO minimizes
To (H x) Tref (Nec
log Tin (1) (z)
CDPO (0) = - log (Blog
Troe
By bypassing the intermediate step reinforcement learning, DPO offers more stable and compu- tationally efficient alternative to standard RLHF, while still aligning models effectively with human feedback.
Methodology
3.1 Parameter Shrinkage and Inflation Analysis
We follow the model from to illustrate why data selection can improve model performance. assume that reward model r r(z, y) (ő(z, y), w*) with some feature function o(.,)) For reward

---

Table Symbols used in the formulation.
Bu/u
o/
Reward Input prompt
Preferredddisprefeedd Response (relative) feature function
Exogenous
Learnable parameters
learning, our reward model can be an explicit r(z, y) [36] while for DPO, / log =e(v(z)
s plays the
role of reward model implicitly [9]. Based on observations in previous literature, we can derive such features by removing the last layer of the pre-trained model. However, both humans and other large models may inaccurate reward functions generate labels, where the labels represent the ranking of two responses. We say the preference between Y/u and yi is generated by r(æ, Foe - r(x, 31) + Ç where Ç is an exogenous error. We use AΦ(x) to denote ((x; (/w) Φ(0, vi) for simplicity.
Parameter Shrinkage. Here, we hope to find w to minimize
LuM((w) = -Ba.s (ite (A+ log( 1 (AM()) eA
1
+ e(Sa(e),
log(
+ e(Apla)
)] (1)
It holds that the first-order condition is e (( Ad(x)) = Be.s 11+ (A(()))))) 1 t (aم()))))
1
(Ao(s)))) Aō(x))
Bu.c 1 elad))))))) 1 c(Aa(a),m))
(2)
Since we know that ( is positive, when Ç is small comparing to the margin, it holds that 14ci400) is convex with respect to Ç. Due to Jensen's inequality, it holds that
e(A0())))) Ad(x)
((A(())))) Ap(z)
Ex, 1 1 (((ot))))= 1 e(do(r)
E. 1+ cldo(z),") 1 c(do(t)).)
Similarly, we have
(A0()))))) Ao(Z)
(A0())))) AΦ(z)
Ex, e (A)0)),-"- 1 (Ap(s),,))
R2 ( 1 e (Ao(a)..))
Since the optimal w is w" without Ç, plugging w" in Equation ( will cause the left-hand side to be greater than the right-hand side. Therefore, the optimal w with the existence of Ç intends to shrink to the original point compared to w so that the first-order condition is still satisfied.
We provide the underlying intuition with an extreme example. If V()) goes to infinity, the preference between Vw and y mainly depends on Ç. approaching a Rademacher distribution, then w could be a good solution to Equation (I). In other words, Ç offsets part of the information provided by the reward model, causing the model's parameters to shrink toward zero. Thus, data selection is essential for acquiring policies with good performance. Finally, we remark that Ç can come from multiple resources, including human classification errors, different embeddings or reward models from other
LLMs and
Parameter Inflation. We next explain why selecting data points based on the margin can lead to parameter inflation, thereby offsetting the parameter shrinkage caused by errors.
First, when the margin is large, namely, (AΦ()), w²) + Ç large, from the S-shaped graph of (+)), we know that the slope is very small in this area. As a result, the probability of preference reversal caused by Ç is low, which means the likelihood of incorrect samples also low. Secondly, given prompt x, as we select data with large (Aő(x), + Ç. the posterior distribution of Ç is skewed toward the positive side. Therefore, the preferences corresponding pronounced, leading to inflated estimates of w in Equation (T). Finally, we point out that if realized and y are all separable, proportional scaling of can reduce the value of Equation (I) continuously. Hence, some techniques like regularization or early stopping when training are indispensable.
this kind of data are more
In summary, inaccuracies in the reward model cause the parameters of LLMs shrink toward zero. By selecting data with larger margins, we can compensate for the performance degradation caused by this shrinkage. The balance between parameter shrinkage and inflation offers the potential to enhance the performance of LLMs. Driven by this theoretical result, our main idea is to let the model reach an overall high margin during preference learning. We realize this by providing a robust estimation of reward margins for the entire dataset ahead of full-set training, which then allows efficient maggin data filtering.
3.2 Multi-source Margin Aggregation
Building on our previous analysis, we aim to develop a data selection strategy based on the margin- maximization principle, with the calculation of reward margin being the critical component. We

---

Implicit Reweard Margin
Thend Ine
3B Impllch Reward Margin
Trend
Armorm Rewwrrd Margin
Tend line
8 13
20-
20-
410
20
40
-40
240
External Reward Margin
10 Implicit Reward Margin
Skywork Reward Margin
Figure 2: Visualization of joint margin distribution on FltraFeedback. (Left) Joint distribution of
external and implicit reward margin values. (Middle) Joint distribution of implicit reward margins computed using models of IB and 3B scales. (Right) Joint distribution of two different external reward margin values ooline-generated data.
examine two distinct types of reward margin calculations: external reward margin and implicit reward margin. The external reward margin provided by an external reward function, while the implicit reward margin is derived from the implicit reward log _u_, where Te represents the policy trained by Teo!." DPO [39] (see Section 4.1 for details on margin calculation). We visualize the joint distribution of different reward margin sources using the UltraFeedback dataset 7 in Figure
The left and middle panels of Figure 2 reveal several key phenomena: (1) The correlation between implicit and external reward margins notably weak. In particular, samples exhibiting high positive implicit margins span a broad range of external margins, including strongly negative values, and vice versa. This divergence highlights the distinct preference patterns captured by these reward types. This underscores the need to combine both reward types for a reliable margin estimation. (2) In contrast, we observe a strong correlation between implicit reward margins calculated by models of different sizes (Llama-3.2 3B and IB). Notably, these two patterns remain consistent across other datasets as well (see Appendix B.2).
Online RLHF [3 employs the target model to generate multiple responses for given prompts iteratively and uses an external reward model identify the response pair with the largest margin for DPO training. The right panel of Figure 2 iillssrates that a max-margin pair construction method, even when derived from one strong reward model, can still yield ambiguous preferences when evaluated by another reward model (which shows similar performance on RewardBench Leadboard). This ambiguity preference signal indicates that the online data generation process can still cause high redundancy, which may offer little to no benefit, or could even be detrimental, online-DPO training.
These observations highlight the need for multi-source margin aggregation to achieve a more robust margin estimation, thereby enhancing data selection and preference learning. To this end, propose a strict aggregation strategy. Bayesian Aggregation for Preference data Selection (BeeS), that deprioritizes preference pair if it exhibits low reward margin from any single reward source. We implement this method through a general three-step procedure.
Step 1: in-distribution pre-DPO training. Our objective is to obtain the in-distribution implicit reward model with low computational and GPU-memory cost. Given the strong correlation that margin across different models (See Figure 2). this weak-to-strong guidance is feasible. To achieve this, we randomly select a small seed dataset Do from D and employ (or several) small model to perform preference learning this seed set. The high sample and training efficiency of the DPO loss 23 ensures the feasibility of this approach.
Step 2: margin calculation. We calculate external and implicit reward margins as Mex Tex (x', y'.) - "==c(=,,)) vf) and mim Tim (x', vln) - (2m ,v}) for each datum D, where we directly calculate Tim (I, y) as log [v)=( preference learning. We assume that there are K margins involved from these two typical sources.
ma(ul=) Here, Tref and To denote the small model before and after
Step 3: Bayesian aggregation. To mitigate noise from individual reward sources, we utilize Bayesian probability theory for their robust aggregation. Given that reward margins from different sources often vary in their underlying distributions and value ranges, we propose projecting these diverse margins into a unified probability space. This projection serves to quantify the confidence that specific preference direction you > y1 is correct, formally expressed as (P(y > yr w , m2.... .m*). Assuming that these sources are conditionally independent, then following the Bayesian formula
5

---

transformation in previous work [0 19, the preference probability can be expressed as:
I K [P y1 /m²)
P( > yt Imm :..... m K)
(3)
Il K (y > 31/m²) + II:= (1 P(y > 3 u(mm))
The typical absence of well-defined preference labels (e.g.. Yhe y! for clear preference, or Vw = Y/ for indifference) renders the rigorous estimation of single-source preference proba- bilities challenging Consequently, we approximate the probability using linear projection: F( > Yt m²) = clip(m' clip(m",L.U)-L where clip(m, L, U) = min(max(m, L), U)) and L, U are U-L
tuning parameters. This adaptive approach mitigates the adverse effects of outlier samples with unusually high margin values. See more implementation details and discussion about the derivation approximation of Eq. (3) in Appendix A.I
Sample selection. We consider the data selection for both the one-pass DPO training and the iterative DPO workflow. For the former, we directly select the samples with the highest aggregated preference probability to construct Dtrsin- The threshold depends on how many preference samples we prefer to use (but should guarantee that samples with negative margins are excluded). For the latter, we only need to train implicit reward models in the first iteration, and BeeS three-step procedures are applied for each iteration.
4 Experiments
We organize the experiments as follows: we explain the experimental setup in Section 4.1: we compare BeeS with various sample selection baselines on diverse preference tasks and present the detailed results in Section[4.2) then we focus on the important chat task, and explore the effectiveness of BeeS in enhancing comprehensive dialogue ability in Section 4.3 Lastly. we perform diverse ablation studies for the BeeS in Section
4.1 Experimental Setup
Preference Datasets. We evaluate our approach using three established preference datasets: (I)
Reddit TL;DR summarization dataset 47 42 that contains human-written summaries and human- rated results, (2) Anthropic Helpful and Harmless dialogue dataset (HH) (2), and (3) UltraFeedback
77.. which comprises guality-scores model responses across diverse prompts from multiple sources. To explore how models react to on on-policy data, we leverage two modified versions of the UltraFeed. and Distral-UUtraFeeebbae
(32] In the variants, the original chosen and rejected responses are replaced with the highest and lowest scored responses, respectively, sampled from five candidates generated by the corresponding Instruct model. The scores are given by 21 reward model. Statistics about these datasets in Appendix A
back dataset, Caima-Ultraaeediie
the PairRM
Models. Our experiments are conducted across four model series: Llama-3.2 3.2 [33 Llama-3 14]
Mistral-7B-v2 20 and Qwen-2.5 (55] under Base and Instruct setups. For the Base model (Llama 3.2-3B and 3-ama-3-8B), we first establish fundamental cestructiio-followweee capabilities through supervised fine-tuning on the setup, we directly use them as the start of DPO training. Regarding the external reward model, adopt the recent -wwwork--eearddlee.... 1aowaard-llam-311--880 -8B-v0.. (29 that is the best reward model at this scale according to the RewardBench leadboard. As for the implicit reward model, we employ the Llama-3.2 3.B-3B Base and its DPO-tuned model (on 2,000 randomly selected samples from the complete dataset) for Reof and Tg.
2.5-ndadd datasets. For the Instruct
Implementation and Evaluation. For DPO training, we follow 39 and use a fixed value of 0.1, except for TL;DR where B = 0.5. We run each training for two epochs, with a learning rate 5f x 10-7, and a 0.1 warmup ratio. Following [39], we evaluate the models using 400 randomly sampled test sets from the validation/test pools the TL;DR and HH datasets, separately. For
models trained on Ferraeed we employ AlpacaEval and AlpacaEval 2.0 27 our evaluation benchmark, which consists of 805 diverse questions As the ground truth oracle is unavailable,
"This estimation typically relies on methods such isotonic regression or histogram analysis.
'They use the same set of questions, and differ in their reference response generation: AlpacaEval uses Text- next-Davinci-003 (58). whereas AlpacaEval 2.0 employs GPT4- 1 11111106veiii preview

---

Table 2: GPT-4 judged win rates for Llama-3.2-3B models fine-tuned with DPO subsets selected by various data selection strategies. For every strategy and benchmark (TL;DR, HH. UltraFeed- back (UF)), 2,000 preference samples were selected. Performance is highlighted as follows: bold numbers denote the best results, blue numbers indicate the significantly degraded results, and underlined numbers represent runner-up performances to the best number. P. Z. and N denote the most positive, near-zero, and most negative selection principles.
Strategy
Rand
External Margin
Implicit Margin
IFD Margin
BeeS
Fullset
Region
P
Z
N
P
Z
N
P
Z
N
P
TL;DR
46.50
66.25
42.00
22.00
30.75
43.00
19.75
1.75
41.25
55.25
83.25
36.75
HH
84.25
82,25
76.50
69.75
92.25
81.25
32.00
11.25
90.00
64.50
90.25
92.00
UF
82.86
91.18
73.29
25.84
89.81
77.14
37.02
72.05
83.60
54.53
91.68
80.99
employ GPT-4 as proxy for human judgment across three distinct settings: summarization, helpful or harmless completion, and single-turn dialogue. We utilized a fixed decoding temperature (T = 0.7) for all model generation in the experiments. More details are presented in Appendix A
Baselines. For the offiine data selection setting, compare our method with three types of methods: (1) Random, a simple yet effective strategy in many domains (c.g.. Instruction tuning (52]), (2) IFD (66 (i.e., exponential form the Point-wise Mutual Information), which measures semantic overlap. We use the difference in IFD scores between chosen and rejected responses for preference data selection. (3) External/Implictt Margin (M-Ex/Im) computes the gap between chosen and rejected responses using either external reward models or implicit DPO rewards. For (2) and (3), we segment the data into (most positive pairs), Z (close to zero pairs), and N (most negative pairs) subsets according to margin values. Specifically, previous work [50] posits that "hard" preference pairs (where chosen and rejected samples are highly similar) are more beneficial for training, and we use the IFD-Z to quantify this scheme and call it Low-Gap. For the iterative DPO setting, we compare our approach against the standard online iterative DPO baseline established by (53] [2] and run for three rounds, each using 20k prompts sampled from UltraFeedback. We provide source code wtwpttuuboooo/iiinnaaaai/i/////a PaPP--aataSSllecton.
of our paper in
4.2 Win Rate Comparison with Baselines Classic Preference Datasets
First, we compare BeeS and baseline strategies on three widely-used preference datasets: TL;DR, HH, and UltraFeedback. Using Llama-3.2-1 3.2-3B model as our Base architecture, we evaluate different selection methods, each sampling 2,000 training examples for DPO training. We use AlpacaEval as the test sets of UltraFeedback as it better reveals the degree of improvement. The results, measured We summarize the findings below.
by GPT4 win rates, are presented in Table 2
Our method, BeeS, consistently achieves optimal or near-optimal win rates across all eval- uated tasks, while all baseline methods show weak performance at least one task. This outcome highlights BeeS's superior robustness to noisy or detrimental samples across diverse task environments. Further, more data in DPO training does not always yield better results. Using just 2-5% of carefully selected data can surpass the performance achieved with the full dataset. Insights from Table also reveal the existence of toxic samples and potential pitfalls of certain selection strategies. For instance, results highlighted by blue numbers show that models trained on data selected using external, implicit, or IFD margins can sometimes perform significantly worse than models trained on randomly chosen subsets. Such outcomes highlight the critical need for rigorous data quality assessment and effective filtering mechanisms in DPO training pipelines.
Among all methods, only implicit margin-N consistently identifies toxic samples, emphasizing the value of incorporating DPO implicit reward margin into the BeeS' strategy. Despite its strong performance in RewardBench, the Skywork reward model's 's margin signals prove less effective than random selection on HH. highlighting the coat-o--istributiin challenge external reward models face when evaluating unfamiliar behavioral patterns/preferenee margin metric, it exhibits notable inconsistency across different datasets, rendering it unreliable for evaluating new datasets where prior preference patterns are unknown. In general, preference learning departs from traditional representation learning, which predominantly leverages contrastive samples to improve discriminative capacity [10 8). Preference learning focuses on capturing
As for the IFD
7

---

Table 3: Performance comparison on AlpacaEval 2.0 using DPO-trained models with different 6,000e subsets (10% of full set). Both SFT and Instruct variants of Llama-3-88 were evaluated. LC and WR denote length-controlled and win rate, respectively. Bold number denotes the best-performing selected subset. Blue numbers denote results that show little advantage over random.
Dataset Model
UltraFeedback
Llama-3-Base (8B)
-lama-3-Instruns (8B)
Llama-3- B-Be (8B)
Llama-3-Instuutt (8B)
Metric
LC (%)
WR (%)
LC (%)
WR (%)
LC (%)
WR (%)
LC (%)
WR (%)
Init
9.61
6.69
22.92
22.57
9.61
6.69
22.92
22.57
Random
12,33
10,96
22.74
24.59
11.58
9.51
31.51
31.92
Low-Gap
13.93
11.40
28.19
27.95
11.12
7.87
34.95
34.25
M-Ex
16.61
14.81
26.28
25.24
21.11
18.63
35.10
34.80
M-Im
19.33
17.80
29.71
29.44
18.88
16.25
33.71
32.92
BeeS
19.53
19.09
30.03
30.46
21.67
20.01
36.36
36.47
Full
17.32
15.30
28.64
26.54
19.92
16.45
32.31
32.44
DPO
DPO Train Margin
Margin
0.0
Stegp
Steps
Figure 3: DPO training loss and margin of Llama-3.2-3B Base (Left) and Llama-3-8B Base (Middle and Right) on UltraFeedback datasets.
semantic relationships, and models benefit when the underlying preferences in the data are explicit and well-defined.
4.3 AlpacaEval 2.0 Win Rate Comparison
In this section, we aim to understand how data filtering influences DPO training efficiency and models' versatile conversational abilities, representing a key application area for preference learning. We use both Llama-3-8B (Base) and (Instruct) models, measuring performance through raw and length-controlled win rates on AlpacaEval 2.0, and results are shown in Table Figures
BeeS consistently outperforms fullest DPO training and other selection strategies. As shown in T BeeS-selected subsets achieve around 4% higher win rates compared full dataset training across all four settings. This distinct advantage highlights BeeS's superior data and training efficiency, and further confirms the significant value of effective data filtering for DPO training. In contrast, all baseline strategies demonstrate inferior performance or limited improvement on some evaluated settings (see blue results). We attribute this instability to samples with ambiguous preferences, and whose margins differ a lot for different reward models.
Different training dynamics of 'P'/N'/'Z subset region. The left panel of Figure shows DPO training loss curves for subsets selected by various strategies. Notably, training on subsets filtered according the 'P' and 'N' criteria results in rapid decrease in loss. In contrast, the loss curves corresponding to the 'Z' criterion tend to stabilize at consistently higher plateaus. Notably, 'N'- selected samples, which are often assumed "difficult-toleemm"" (16), rapidly as pp'sselected samples, suggesting that 'bad preferences are also easy to grasp for L.M. The pattern consistent across different datasets and models (see Appendix C.1 for more results. While this observation might explain proposals that use absolute margin values for selection (35), Table reeveals that 'P' and 'N' samples produce opposing effects despite similar training dynamics.
actually be learned as
The middle and right panels of Figure 3 illustrate that data subsets selected by BeeS exhibit both the most rapid decrease in training loss and the fastest increase in the DPO training margin, i.e., current train-batch average implicit margin. These concurrent observations of accelerated optimization help explain the superior performance achieved by BeeS.
Extension Iterative DPO. We explore the data efficiency of iterative DPO using prompts from UltraFeedback as in [53] In comparison, 20k prompts are used for on-policy preference pair

---

50
Llama-3-Rase (8B)
Ilama-3-Instruet (8B)
45:17
Method
45
LC
WR
Lon
1.C (%)
WR (%)
Len
Win Rate
4..80
DPO (rl)
17.64
13.36
1496
40.51
43,90
2173
37,90
DPO (22)
23,06
22.45
1897
42.51
49.23
2366
35
DPO
29.03
30.86
2736
44.51
53.12
2860
30
DPO LC WR
F-OPO WR
DPO WR
OPO WR
DPO-BeeS (cl)
16.35
13.09
1624
42:20
45.74
2091
SimPO
DPO-ReeS (r2)
23.79
24.17
1901
46.40
50.60
2316
WR
WR
DPO-BecS (13)
32.31
33,91
2565
48.49
54.99
2774
3K
6k
10k 15k All (60k)
Subset Size
Figure 4: AlpacaEval 2.0 results for on-policy datasets: (Left) Iterative DPO results across three DPO training iterations using UltraFeedback prompts. (Right) DPO on Llama- UltraFeedbacc subsets of varying sizes, selected by BeeS. Results of DPO-variants trained on fullset are also compared.
generation per iteration, and our online version uses BeeS to reserve only 5k samples per iteration. The results are in the left panel of Figure 4
There is high redundancy in the on-policy data construction manner. Although iterative DPO shows much higher data efficiency than one-pass DPO training (i.e., better results than those in 3 data selection is still important for quality control. This can be attributed to the presence of numerous ambiguous, wow-margin samples (usually paired with low-quality prompts).
A smaller 3 value in DPO loss correlates with higher data efficiency. While 3 is commonly recognized as factor controlling the strength of the Kullback-Leibler (KI.) divergence, it also significantly influences data efficiency. Specifically, the DPO loss, defined as log a (B x mim), indicates that a reduced 33 allows for effective gradient updates for more preference pairs with wider margins. To investigate this, we conducted DPO training on the Llama-3-Instruct 8B model using its on-policy dataset, Llama-Ultra Feedback, with / set to 0.01. We then evaluated performance using varying numbers of samples selected by our method, BeeS. The results, presented in the right panel of Figure 4 demonstrate that relaxing the margin constraint in the DPO loss substantially improves data efficiency (Refer Appendix (a.diix[C44ffo for results the Base setup). Notably, DPO training with 3k-sample subset selected by BeeS achieved performance comparable to training with the full dataset (which is 20 times larger).
Furthermore, we compared BeeS data selection with several established DPO variants that modify the original loss function, including SimPO [32], 3-DPO [50), and a-DPO 49 Our findings indicate that BeeS is unique in its ability effectively enhance both the win rate and the length-controlled
(LC) win rate. In contrast, these variants primarily improved the LC win rate, and to a lesser extent than BeeS. These results underscore the significant potential data selection and data efficiency considerations to enhance the original DPO training algorithm.
4.4 Ablation Study
critical aspect of dataset filtering methods is their generalization performance when transferred to new models or applied with similar optimization algorithms.
the
Data filtering remains effective for new LLM architecture. we evaluated BeeS on several contemporary model architectures: Mistral--7B-Instruec, 2.wen-255-BB-Insteee Instruct. Consistent with previous experiments, BeeS was used to select 10% data subset, and its performance was compared against DPO training on the full dataset. The AlpacaEval 2.0 evaluation results presented in the left panel of Figure:] We observe that BeeS consistently and significantly outperforms full-dataset DPO training. Notably, even though larger models like Qwen- 14B inherently demonstrate higher data efficiency, our data selection strategy, BeeS, still improved the win rate by approximately 3% while utilizing only 10% of the data.
and Qwen-2.5-
BeeS selected subsets are effective for diverse preference learning algorithms. We examine whether the subset selected by BeeS remains data efficient for DPO-variants like IPO [I. KTO 15). and SLiC (62]. We utilize the BeeS selected 6k-sample subset from Alrma-llraaeedback and the results are presented in the right panel of Figure [5] We observe that the high-margin subset consistently benefit these preference learning algorithms by outperforming full-set training.

---

1R0 LC WA
KTG LC WR
SUCLEWE
MA
KTO WM
SUX. WM
Mistral-7B
Qwen-2.5-7B
Qwen-2.5-144
20
LC (%)
WR (98)
LC (%)
WR (%)
LC (%)
WR (%)
Win Rate (8)
Initial
17.11
14.72
31.27
31.60
37.03
32.64
Full
18.00
18.77
39.67
38.24
49.99
46.81
BeeS
26.04
20.53
46.20
43.78
50.20
49.74
Figure 5: Ablation Study: (Left) different model choices Hiistrall7B--Isttuet--ve Qwen-2.5- Instruct-7B and Qwen-2.5-Instructtle
BeeS selects a 6k-sample subset for training. (Right) variants of DPO: win rate comparison on IPO, KTO, and SLiC algorithms. UltraFeedback is used for the preference learning on Llama-3-8B (Base) model. Rand and BeeS select a 6k-sample subset.
Notably, achieves large improvements in raw/LC win rates over 12% for the IPO algorithm. This advantage is maintained even across these preference learning algorithms with varying data efficiency (as measured by the performance gap between randomly selected 6,000 samples and the full dataset). These findings highlight the significant value of sample filtering for other preference learning. Additional results related to Instruct model training can found Appendix C.3
Conclusion
Our research bridges the gap between algorithmic advances and Language Model (LLM) alignment by systematically examining how preference data quality affects Direct Preference Optimization (DPO). We address the issue of parameter shrinkage caused by noisy data and introduce a BeeS strategy for selecting high-quality training examples. This approach not only improves model performance but also significantly increases computational efficiency. Our extensive experiments show that the method maintains or enhances performance while using just around 10% of the original training data, demonstrated through consistent improvements on the AlpacaEval2 benchmark. Additionally, our framework successfully extends to iterative DPO applications. These results emphasize the importance of careful data curation in developing better alignment techniques and provide practical guidelines for future research and implementation.
approaches in Large
Acknowledgements
This work is supported by the National Natural Science Foundation of China (U24B20180 and the Alibaba Group through Alibaba Research Intern Program. We appreciate the reviewers for their insightful feedback and advice, these constructive criticism and recommendations have been invaluable in helping us improve the quality of this work.
10

---

References
[1] M. G. Azar, Z. D. Guo, B. Piot, R. Munos, M. Rowland, M. Valko, and D. Calandriello. A general theoretical paradigm understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447 4455. PMLR, 2024.
[2] Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:220400580) 2022.
[3] R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(344):324-355, 1952.
[4] Y. Cao, Y. Kang, C. Wang. and L. Sun. Instruction mining: Instruction data selection for tuning large language models. COLM. 2024.
[5] X. Chen, Zhong, Yang, Z. Wang, and L. Wang. Human-in-the-loop: Provably efficient preference-basel reinforcement learning with general function approximation. In International Conference on Machine Learning, pages 3773-3793 PMLR, 2022.
[6] P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[7] G. Cui, L. Yuan, N. Ding, G. Yao, W. Zhu, Y. Ni, G. Xie, Z. Liu, and M. Sun. Ultrafeedback: Boosting language models with high-quality feedback. ICML, 2024.
[8] X. Deng, F. Feng, X. Wang, X. He, H. Zhang, and T.-S. Chua. Learning to double-check model prediction from causal perspective. IEEE Transactions on Neural Networks and Learning Systems, 35(4):5054-5033, 2023.
[9] X. Deng, J. Liu, H. Zhong, F. Feng, C. Shen, X. He, J. Ye, and Z. Wang. A3s: A general active clustering method with pairwise constraints. arXiv preprint UrXiv:2407.101990 2024.
[10] X. Deng, W. Wang, F. Feng, H. Zhang, X. He, and Y. Liao. Counterfactual active learning for out-of-distribution generalization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1 11362-1 11377, 2023.
[11) H. Dong, W. Xiong, D. Goyal, Y. Zhang, W. Chow, R. Pan, S. Diao, J. Zhang, K. Shum, and Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint N3i::2304406667, 2023.
[12] H. Dong, W. Xiong, B. Pang, H. Wang, H. Zhao, Y. Zhou, N. Jiang, D. Sahoo, C. Xiong, and T. Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405.0740 07.666 2024.
[13] Q. Dong, L. Dong, X. Zhang, Z. Sui, and F. Wei. Self-boosting large language models with synthetic preference data. ICLR, 2025.
[14] A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
[15] K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01 3036, 2024.
[16] C. Gao, H. Li, L. Liu, Z. Xie, P. Zhao, and Z. Xu. Principled data selection for alignment: The hidden risks of difficult examples. arXiv preprint 250.00250,.09650, 2025.
[17] C. Gulcehre, T. I.. Paine, S. Srinivasan, K. Konyushkova, L.. Weerts, A. Sharma, A. Siddhant, A. Ahern, M. Wang, C. Gu, et al. Reinforced self-training (rest) for language modeling. arXiv preprint a3Xiv:2308
08998, 0899898,8 2023.
[18] Han, M. Jiang, Y. Song, J. Leskovec, Ermon, and M. Xu. f-po: Generalizing preference optimization with f-divergence minimization. arXiv preprint arXiv:2410.21662, 2024.
11

---

[19] J. Hong, N. Lee, and J. Thome. Orpo: Monolithic preference optimization without reference model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11170-11188 89, 2024.
[20] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. d. I. Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.0682. 2023.
[21] D. Jiang, X. Ren, and B. Y. Lin. Lim-blender: Ensembling large language models with pairwise ranking and generative fusion. arXiv preprint arXiv:2306.022 02561, 2023.
[22] S. Khaki, J. Li, L.. Ma, L. Yang, and P. Ramachandra. Rs-dpo: hybrid rejection sampling and direct preference optimization method for alignment of large language models. arXiv preprint arXiv:2402. 10038, 2024.
[23] D. Kim, K. Lee, J. Shin, and Kim. Spread preference annotation: Direct preference judgment for efficient Ilm alignment. In ICLR, 2025.
[24] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng. C. H. Yu, J. Gonzalez, H. Zhang, and Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611 626, 2023.
[25] M. Li, Y. Zhang, S. He. Z. Li, H. Zhao, J. Wang, N. Cheng. and T. Zhou. Superfiltering: data filtering for fast iustruction-tuning ACL, 2024.
[26] M. Li, Y. Zhang, Z. Li, Chen, L. Chen, N. Cheng, J. Wang. T. Zhou, and J. Xiao. From quantity to quality: Boosting Ilm performance with self- self-guided data selection for instruction tuning. ACL, 2023.
[27] X. Li, T. Zhang, Y. Dubois, R. Taori, I. Gulrajani, C. Guestrin, P. Liang, and T. B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-folloving models, 2023.
[28] Z. Li, T. Xu, Y. Zhang. Y. Yu, R. Sun, and Z.-Q. Luo. Remax: A simple, effective, and efficient method for aligning large language models. arXiv preprint arXiv:2310. 10500,,, 2023.
[29] C. Y. Liu, L. Zeng. J. Liu, R. Yan, J. He, C. Wang, S. Yan, Y. Liu, and Y. Zhou. Skework-reward: Bag of tricks for reward modeling in llms. arXiv preprint arXiv:2410. 18451, 2024.
[30] J. Liu, J. Liu, S. Yan, R. Jiang, X. Tian, B. Gu, Y. Chen, C. Shen, and J. Huang. Mpe: Multi-view probabilistic clustering. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9509-9515 9818 2022.
[31] X. Lu, Welleck, J. Hessel, L. Jiang, L. Qin, P. West, P. Ammanabrolu, and Y. Choi. Quark: Controllable text generation with reinforced unlearning. Advances in neural information processing systems, 35:27591-27609, 2022.
[32] Y. Meng, M. Xia, and D. Chen. Simpo: Simple preference optimization with reference-free reward. NeurIPS, 2024.
[33] MetaAL. Llama 3.2: Revolutionizing edge ai and vision with open, customizable models. Blog article, 2024.
[34] S. Mukherjee, A. Lalitha, K. Kalantari, A. Deshmukh, G. Liu, Y. Ma, and B. Kveton. Optimal design for human preference elicitation. Advances in Neural Information Processing Systems, 37:90132 901599 2024.
[35] W. Muldrew, P. Hayes, M. Zhang, and D. Barber. Active preference learning for large language models. arXiv preprint ar arxiv:200..081 14, 2024.
[36] L. Ouyang. J. Wu, X. Jiang. D. Almeida, C. Wainwright, Mishkin, C. Zhang. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744 2022.
[37] A. Pacchiano, A. Saha, and J. Lee. Dueling rl: reinforcement learning with trajectory preferences. arXiv preprint 2rX.v:2111.04850, 2021.
12

---

[38] R. Park, R. Rafailov, S. Ermon, and C. Finn. Disentangling length from quality in direct preference optimization. arXiv preprint arXiv:2403.1915, 2024.
[39] R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct preference optimization: Your language model is secretly reward model. NeurIPS, 36, 2024.
[40] P. Ren, Y. Xiao, X. Chang, P.-Y. Huang, Z. Li, B. B. Gupta, X. Chen, and X. Wang. A survey of deep active learning. ACM computing surveys (CSUR), 54(9):1-40, 2021.
[41] Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, Proximal policy optimization algorithms. arXiv preprint arXiv: 17::::1777.00 1707.06347 2017.
[42] N. Stiennon, Ouyang, Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008 -001,,001122222 2020.
[43] S. Swayamdipta, R. Schwartz, N. Lourie, Y. Wang, H. Hajishirzi, N. A. Smith, and Y. Choi. Dataset cartography: Mapping and diagnosing datasets with training dynamics. ACL, 2020.
[44] Y. Tang, Z. D. Guo, Z. Zheng, D. Calandriello, R. Munos, M. Rowland, H. Richemond, M. Valko, B. Á. Pires, and B. Piot. Generalized preference optimization: A unified approach to offline alignment. arXiv preprini arXiv:2402.0577 2024.
[45] K. K. Thekumparampil, G. Hiranandani, Kalantari, S. Sabach, and B. Kveton. Comparing few to rank many: Active human preference learning using randomized frank-wolfe, arXiv preprint arXiv:2412.19396 2024.
[46] L. Tunstall, E. Beeching, N. Lambert, N. Rajani, K. Rasul, Y. Belkada, S. Huang, L. von Werra, C. Fourrier, N. Habib, et al. Zephyr: Direct distillation of Im alignment. COLM, 2024.
[47] M. Völske, M. Potthast, S. Syed, and B. Stein. TI: dr. Mining reddit to learn automatic summarization. In Proceedings of the Workshop New Frontiers in Summarization, pages 59-63, 2017.
[48] J. Wang, B. Zhang, Q Du, J. Zhang, and D. Chu. A survey on data selection for Ilm instruction tuning. arXiv preprint 2402.0200.. 2024.
[49] J. Wu, X. Wang, Z. Yang, J. Wu, J. Gao, B. Ding, X. Wang, and X. He. a-dpo: Adaptive reward margin is what direct preference optimization needs. arXiv preprint arXiv:2410. 10148, 2024.
[50] J. Wu, Y. Xie, Z. Yang, J. Wu, J. Gao, B. Ding, X. Wang, and X. He. B-dpo: Direct preference optimization with dynamic B. NeurIPS, 2024.
[51] Y. Wu, Z. Sun, H. Yuan, K. Ji, Y. Yang, and Q. Gu. Self-play preference optimization for language model alignment. arXiv preprint 4xXi:22455.00675 2024.
[52] M. Xia, Malladi, Gururangan, S. Arora, and D. Chen. Less: Selecting influential data for targeted instruction tuning. ICML,, 2024.
[53] W. Xiong, H. Dong, C. Ye. Z. Wang, H. Zhong. H. Ji, N. Jiang, and Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. In Forty-first International Conference on Machine Learning. 2024.
[54] H. Xu, A. Sharaf, Y. Chen, W. Tan, L. Shen, B. Van Durme, K. Murray, and Y. J. Kim. Contrastive preference optimization: Pushing the boundaries of Ilm performance in machine translation. arXiv preprint arXiv:2401.08417 2024.
[55] A. Yang, B. Yang. B. Zhang. B. Hui, B. Zheng, B. Yu, C., D. Liu, F. Huang, H. Wei, et al. Qwen2. 5 technical report. arXiv preprint wxiv:2412.15115. 2024.
[56] R. Yang, X. Pan, F. Luo, S. Qiu, H. Zhong, D. Yu, and J. Chen. Rewards-in-context: Multi- objective alignment of foundation models with dynamic preference adjustment. arXiv preprint 202.107702 2024.
13

---

[57] M. Yasunaga, L. Shamis, C. Zhou, A. Cohen, J. Weston, L. Zettlemoyer, and M. Ghazvininejad. Alma: Alignment with minimal annotation. arXiv preprint arXiv:2412.04305, 2024.
[58] J. Ye, X. Chen, N. Xu, C. Zu, Z. Shao, S. Liu, Y. Cui, Z. Zhou, C. Gong, Y. Shen, et al. compre- hensive capability analysis of gpt-3 and gpt-3.5 series models, arXiv preprini 1ttt:2333..042000 2023.
[59] Yu, W. Yuan, O. Golovneva, T. Wu, S. Sukhbaatar, J. Weston, and J. Xu. Rip: Better models by survival of the fittest prompts. arXiv preprint arXiv:2501.18578, 2025.
[60] Zhang, Z. Liu, B. Liu, Y. Zhang, Y. Yang, Y. Liu, L. Chen, T. Sun, and Z. Wang. Reward- augmented data enhances direct preference alignment of Ilms. arXiv preprint arXiv:2410.08067, 2024.
[61] Zhang. Yu, H. Sharma, H. Zhong, 7. Liu, 7.. Yang. S. Wang, Hassan, and 7. Wang. Self-exploring language models: Active preference elicitation for online alignment. arXiv preprint arXiv:2405. 19332, 2024.
[62] Y. Zhao, R. Joshi, T. Liu, M. Khalman, M. Saleh, and P. Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint 20X45 2023.
[63] H. Zhong, G. Feng. W. Xiong, X. Cheng, L. Zhao, D. He, Bian, and L. Wang. Dpo meets ppo: Reinforced token optimization for rlhf. arXiv preprini arXiv:2404. 14442, 2024.
[64] B. Zhu, M. Jordan, and J. Jiao. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons. International Conference on Machine Learning, pages 43037-43067. PMLR, 2023.
[65] D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiaano, and G. Irv- ing. Fine-tuning language models from human preferences. arXiv preprint 1909:990.005 2019.
14

---

A Datasets and Evaluation Details
Data information. The detailed information about the datasets used the experiments is shown in T The test sets of TL;DR and HH are sampled from their original large testing pool, and we utilize prompts in AlpacaEval variants. In particular, results in Table 2 Qutilize Text-Davinci-O00 generated answers as reference response as it can better reflect how models' ability varies with different training data, and results in other tables all utilize GPT4- 106-preview generated answer as reference response (i.e., AlpacaFval 2.0).
the test sets for all models trained on UltraFeedback and its
Table Statistical information about the training and evaluation datasets used in the experiments.
Praan
Type Summarization Helpful & Harmless
IL.DR UttraFoebacck Llarna- Uleaabedltuck
92858
400
61135
Hybrid
59876
Hybrid
Mistral- L ltrabbebaock AlpacaEval
59720
Hybrid
805
Chal
Evaluation details. Win rate judgments serve as key evaluation metric across our experiments. For the TL;DR and HH, we slightly modify the evaluation prompts suggested by [39] for the win rates judge. We use the original prompt designed for AlpacaEval and AlpacaEval 2.0 benchmark in all testing.
DR win rate prompt**: Which of the following summaries does a better job of summarizing the most important points in the given forum post, without including unimportant irrelevant details? good summary is both precise and concise.
Post: (Post)
Summary A: (Summary_A)
Summary B:
{Summary_BB
FIRST provide one-sentence comparison of the two summaries, explaining which you prefer and why. SECOND, new line, state only "A" or "B" to indicate your choice. Your response should the format: Comparison: <one-sentence comparison and explanation Preferred: <"A" or "B">
win rate prompt**: For following query to chatbot, which response is more helpful or harmless? A helpful response provides useful information, while a harmless response avoids causing/suggesting harm or discomfort.
Query: {query>
Response A: (res_a)
Response
(res_b)
FIRST provide one-sentence comparison of the two responses and explain which you feel helpful harmless. SECOND, indicate which response more helpful or harmless. Your response should use the
line, state only "A" "B" to
format:
Comparison: <one-sentence comparison and explanation> More helpful harmless: <"A" "B">
15

---

A.1 More Implementation Details
SFT. The SFT training of the Base model is carried out for two epochs with a learning rate of 2 x 10-5, Sample packing [46] is employed to accelerate the training, and we use a block size of 4096.
We present the implementation details of our baseline methods: P, 7, and N using margins calculated from three metries, IFD//ondiiiooal Perplexity (CPPL), External (Ex), and Implicit (Im) rewards. Subsequently, we describe the implementation of BeeS in our experimental setup.
The baseline strategies are implemented as follows: first, we eliminate outlier samples with extreme margin values (both positively high and negatively low) for CPPL, Ex, and Im metrics. For the and N strategies, select samples based on their ranking positions the upper and lower ends of the distribution, respectively. The Z strategy involves random sampling from the subset of samples whose margin values fall within the interval [-T, T]. where is set 0.1 for CPPL and 1.0 for Ex/Im metrics.
For BeeS, we set I, = 2 as the lower bound for both external and implicit reward margins. The upper bound U is determined dynamically based on two conditions: (1) The number of samples with margin values in the interval [U, max margin] is less than 30, or (2) The number of samples in [U, max margin] is less than max margin- U.
prompt:
Hi, I'm trying to solve crosward puzzle, but I've never done one of these before. Can you help me out?
response:
<lstrt header dljessiitante/end header id]=-dend header id] d/jed header id|=s/start header idlpassistant<eeed header id]=ddend heade id]=uu20cttal he happy to help you solve your word puzzle:trinTo get started, you please share the following details: What size is the crossword puzzle (e.g. 15x15, 17x27)P/22, W clues are you baving troblle with? Share the cluets) and the lettar count (if given)
prompt:
How do dioe without slicing my finger
response:
Dicing vegetables safely involvesusing sharp knivesinand proper TepperTTechnggee Technique passes* (boldsyybo) yuffddyffftd
jeu /Taceess (boldsymol wttp:lubbbeeee0555000000 4% w001a) jeereey //tt0010 "tral. Choose Sharp Knives: Using dull knives can incruase the risk of slipping and causing injuries. Make sure your knives are sharp before starting dice. If needed, have them sharpened by a professional./rtlte
Secure Your Wo Worksurface: Make sure your cutting board stable and wont slip during use. You can secure by placing damp towel underneath orusing rubber mot Irinitn3. Proper Holding Tochnique: Hold the vegetable
stabiliznCREEP Vegetable j.m (trin vegidta pess (rraa "r (nn//dd (1ruldc3:) fimmly with your con-dominant hand. This will give you better control and preveat vegetable from moving excessively while you're dicing Miuuind. SmallChurks: v' 2 WwW. W Cutlaryer vegetablestntoquat
HHEIRsizsAA ADD.RssssAANDDD make? HE Meastattounndlee.. trinbrinS.
Weethe ometTechnique: i Arin- Placeethevegeeta hww.b/owwwooehtttoogooooooom
irtn
Applygentedd wow www.ooorrwwwwwwwweeeineeeo.eeeed wes.deiddedaannnnnniioee tbyourfinger: utlidaa? CleanUp: Vil
HEknif ethrought HEvegetable, vtirin. KeepYoufFocus: (Y:
Payfullatt Iallattettionttee wewerrionttteeviiee www.eeeeeeeeeeeeeee
hhefimisheddiicie wwtdhnnnnniiieetooonnooo www.enugggggygyyyhin..... beforestoringthemeee wtw.mmaaaayyaaaaaaa.. Thiswi hwppmmtttiiiiimmnniiieeettrriiin targee
paration sessions.
Figure 6: The model breaking pattern when conducting DPO training with small 3 (3 0.01) for 1amaa-3-8B-Instruu and Mistral--B-IIntmuce- V0.2. We select two examples of abnormal responses given by each model.
B
Visualization of Margin Distributions
B.1 Singular Margin Distribution
The margin distributions calculated using CPPL. margin, External and Implicit DPO reward margins, as illustrated in Figures 9 8 reveal a notable concentration of sample margins around zero. This clustering around the zero indicates ambiguous preference labels. It leads to the challenge in preference learning, as evidenced by the substantially slower decrease in training loss (and slower increase in training margin) compared samples with larger margins, as shown in Section 3.2
B.2 Joint Margin Distribution
To complement the left and middle subfigures Figure 2 we present additional results showing the joint margin distributions of samples on the other datasets in Figure I0) Our analysis reveals
16

---

Dateibation Eegileit Revard Margls
Dair Buirobllenco of Tuglich Beward Narghs
Distrbaakks of implich Reward Nargio
0.20
5.0
KDI
3.00
KXC
KDC
6.36
3.00
3.60
6:23
3.04
0:22
9.00
0:21
3.00
Ingiir Poowert Mtrett
Ingket Bewerd Marze
Lmplict Reward Marpit
(c) UlrraFeddback Figure 7: Distribution of implicit reward margins on TL:DR, HH, and UltraFeedback datasets. The reward is calculated using the Llama-3.2-3B SFT model, and its weakly aligned DPO model that is fine-tuned on 2,000 randomly selected samples from the full set.
(a) TL:DR
(b) HH
Distrthalion el Extornal Rewerd Marptn
Rowart Margin
Distetbation d..atermal Bewerd Marpte
KDZ
3.00
3.00
0.230
0.225
0.14
2-220
0-215
0.60
0.210
0.00
0.300
(a) TL:DR
(b) HH
(c) UltraFeedback
Figure 8: Distribution of external rewards TL;DR, HH, and UltraFeedback datasets. The reward
is calculated using Skywork- Reward-Llama, -.1-ama-..----0.222
that external and implicit margins exhibit minimal correlation across all four datasets, while implicit margins calculated by different models maintain a high correlation. These further enhance the rationality of our design detail of BeeS: fusion of both margins and disentangling implicit margin from the target model (if the target model is bit large and we want to accelerate the enumeration process of the full-set.)
C More Experimental Results
C.1 Train Loss and Margin Curves -3B
To complement the right subfigure in Figure 2, we present additional results showing the progression of training loss and margins throughout the DPO training process. The results are shown in Figure II] All strategies demonstrated consistent patterns in loss reduction: both P margin-oriented and N strategies achieved rapid decreases in training loss, while the Z strategy exhibited slower convergence and remained at significantly higher final loss values. Regarding training margins, P strategies achieved higher levels compared to N and 7. approaches. Notably, our proposed BeeS strategy demonstrated even larger margins than the Implicit Margin-P strategy.
C.2 Resources and computation cost
For all experiments, we utilized 8 A100 GPUs. We conduct SFT/DPO training with 4 A100 GPUs for all runs in our experiments. For both Supervised Tine-Tuning (SFT) and Direct Preference Optimization (DPO) training, we allocated 4 A100 GPUs per run. Training 8B parameter models on the UltraFeedback dataset for two epochs required approximately 9 hours of computation time. In each round of iterative DPO implementation, we performed generation and annotation processes on 4 A 1100 GPUs, with each GPU processing 5,000 prompts with 5 distinct generations per prompt. The overall generation that utilizes vL..M (24) for acceleration takes about 1.5 hours, and the corresponding reward annotation takes about 2 hours.
17

---

Rebribariaa of Canditional PVS Margie
Dutribation Conditional FPLL Narple
DidrOsiion of Conditional Nargin
KDIG
KDB
K3
2.3
220 -15
2.0
Contikooa ITL Margs
Chedltood PRC Marz
Ocecitionl MrLMargh
(a) TL:DR
(b) HH
(c) UltraFeedback
Figure 9: Distribution conditional perplexity (also named instruction following difficulty) margins
on TL;DR, HH, and UltraFeedback datasets. The perplexity is calculated with the Llama-3.2-3B SFT model.
Implicit Reward Margin
Trend line
Implicit Reward Margin
Trend line
30
40
40
25
-40
-20
20
60
60
-40
-20
20
40
60
External Reward Margin
External Reward Margin
(a) TL;DR
(b) HH
Implicit Reward Margin
Trend Ince
80
Implicit Reward Margin
Trend line
Abslune Diflewess
20
Abottre Offferee
-20
-20
External Reward Margin
External Reward Margin
(c) Ultma-Ulrraeeddbckk
(d)
3B Implicit Reward Margin
60
38 Implicit Reward Margin
60
12
Abrolute Ditterence
20
20
20
-20
-20
-10
1B Implicit Reward Margin
Implicit Reward Margin
(e) TL:DR
(f) HH
Figure 10: Subfigure (a)-(d): scatter plot showing the joint distribution of samples across external and implicit reward margin values on four datasets. Subfigure (e)-(f); joint distribution of implicit reward margins computed using models of IB and 3B scales on two datasets.
C.3 More Results for Ablation Study on the DPO Variants
As a complementary study to the results shown in Figure 5 we conducted experiments using the 1amaa---BB-Instreee model while maintaining all other experimental parameters. The results, presented in Figure [2] demonstrate that models trained on subsets selected by BeeS achieved significantly higher win rates across most evaluation scenarios.
18

---

0.0
Bee
m-
Bees
mmz
Margin
0.4
0.2
Steps
Steps
(a) TE;DR
(b) TI.DD
DPO Train Loss
DPO Train Margin
0.0
Bees Im-P
Im
In-
Im-N
0.0
Margin
0.2
0.0
Steps
Steps
HH
(d) HH
DPO Train Loss
DPO Train Margin
Im-N
Im
Im-N
0.6
Loss
Margin
40
Steps
Steps
(c) UltraFeedback
(f) UltraFeedback
Figure 11: DPO train loss and margin on TL;DR, HH, and UltraFeedback datasets. The training
was implemented with Llama-3.2-3B SFT version on different subsets selected by five strategies.
C.4 Hyperparameters Risks
Although smaller 3 can lead to higher data efficiency, observe that it can bring potential issues for DPO training. Specifically, a small |3 corresponds to a relaxed Kullback-Leibler (KL) divergence constraint in the policy optimization process. This relaxation can permit excessive deviation from the initial policy, potentially compromising the model's learned behaviors and stability during training. For instance, when conduct DPO training with selected 6,000 subsets from UlrraFeedback, with 3 = 0.01 and two epochs update, we find that although the model could respond normally most questions, it sometimes outputs repeated or chaotic tokens, as shown in Figure 6 And their win rates on AlpacaEval 2.0 dropped by more than 10 points as a consequence.
Int-----strre//Mistaaaal-aa 1-nd----ttttttt--..... on
the seleted
Further analysis of the training details revealed a significant degradation log probabilities for both chosen and rejected samples, coinciding with the model's performance decline. Specifically, during the lbove-mentioned Mistral model DPO training, the log probability values for chosen samples decreased from -400 to 1400. while rejected samples showed
dramatic reduction from -600 to -4600. Overall, Mistral suffers more from this log probability drop compared to Llama.
Such phenomenon can be avoided by using smaller learning rate: from 5 x 10-7 to x 10-7 or early stop at the end of epoch These operations can lead to relatively smaller drop in chosen/rejected log probability.
19

---

POLC MR
Win (%6
10
Rand
Bed5
FJll
Figure 12: Ablation study variants of DPO: win rate comparison IPO, KTO, and SLiC algorithms. The experiments utilize the UltraFeedback dataset for preference optimization, with the Llama-3-8B- I-strruuct model the initial model. Random and BeeS select 6,000 samples (10% of the full set) for subset training.
33,95
35
31.76
30.65
30
51
28:20
Win Rate
25
27.55
20
15
9.67
10
DPO LC
DPO
5
3k
6k
10k
15k All (60k)
Subset Size
Figure 13: DPO on Llama- (aama-UltaFeedbaae subsets of varying sizes, selected by BeeS, The training is conducted on Llama-3-8B-BBl model.
D Limitations.
The empirical evaluations in this study primarily focused models up the 14B parameter scale, where BeeS demonstrated notable efficacy. Extending these investigations to significantly larger foundation models, such those in the 70B parameter range or beyond, was constrained by the computational resources available for the current work. Future research could build upon our findings by exploring the scalability and performance of BeeS in these larger-scale settings.
20

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiiaton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our abstract and introduction accurately reflect the paper's contributions on data selection for direct preference learning.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
21

---

Justification:
See Appendix D
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification:
We provide the derivation in Section 3.1
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide all implementation details the experimental part and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
22

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All the datasets and models used in this work are publicly available. We utilize the open-source TRL repo for all our DPO experiments, which is easy implement. All the details required to reproduce the main experimental results can be found in the experiments and appendix, and the code is available at https//github..om/kiaggggaassnit Data-Selection.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
gocicc/gguieseeee Cwt.//odeeeeeeeeeeeeeeeo httpa///ggi.ieee///deeuubiibaae wwwttbbmsssiooo ssi assionooiicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
23

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide all training and details in experimental section and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: LLM experiments are typically costly and relatively robust, we do not conduct the experiments repeatedly or report statistical significance metrics.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Refer Appendix C.2
Guidelines:
The answer NA means that the paper does not include experiments.

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Yes, the research in this paper fully conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
[NA]
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
25

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, the paper properly credits the original creators of all assets used, explicitly mentions licenses and terms of use, and respects these conditions throughout the research.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pawpsssipppppitttaaee thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
26

---

Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non sta-srdd components.
Please refer to LLM policy (https://neur))) http://ppp...oooooeeeeeee/////// for what should or should not be described.
27


Reviews:
[{"review_id": "y53FB4NfFr", "reviewer": "Reviewer_18xZ", "summary": "This paper presents a novel preference data selection method designed to enhance DPO training effectiveness and efficiency. Specifically, the authors address noise reduction through a margin-maximization principle and propose a Bayesian Aggregation method to unify multiple margin sources (both external and implicit). Using only 10% of the original data, their approach achieves a 3-8% performance improvement.", "questions": "See above", "limitations": "See above", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper introduces an innovative preference data selection method, the Bayesian Aggregation for Preference data Selection (BeeS) strategy, which incorporates signals from both external rewards and DPO implicit rewards. It deprioritizes preference pairs exhibiting low reward margins from any single reward source to mitigate potential noise.\n2. The authors provide a detailed analysis of Parameter Shrinkage and Inflation Analysis, highlighting how inaccuracies in the reward model can cause LLM parameters to shrink toward zero.\n3. The paper includes comprehensive comparative experiments and analyses that demonstrate the method's effectiveness. Using only 10% of the data, the approach achieves a 3-8% performance improvement, with significant performance gains even in online DPO scenarios.\n\n\n**Weaknesses**\n\n1. The proposed method is somewhat limited in scope, as the analysis and experiments focus exclusively on the DPO method. It remains unclear whether the approach can be extended to other, potentially superior RL methods such as PPO, GRPO, etc.\n2. Minor concern: The authors do not address whether their method is applicable to thinking models or whether it would be equally effective for larger models.", "was_revised": true, "final_justification": "This paper is solid with novel method and superior performance. I appreciate the authors' effort to address my concerns.", "rebuttal": "We thank the reviewer for the encouraging feedback and insightful comments.\n\n**Weakness 1:** The proposed method is somewhat limited in scope, as the analysis and experiments focus exclusively on the DPO method. It remains unclear whether the approach can be extended to other, potentially superior RL methods such as PPO, GRPO, etc.\n\n**Answer:** Thank you for your question. We answer your question from two perspectives: (1) Data selection remains important for efficient PPO training, and margin-based guidance can extend to RL scenarios. To illustrate, we compared two prompt sets: BeeS-selected 2k samples from UltraFeedback versus randomly selected 2k samples, performing RLHF on our SFT version of Llama-3.2-3B (using PairRM as the reward model). The AlpacaEval1.0 win rates were 88.84% for BeeS versus 78.12% for random selection (initial win rate is 61.49%). (2) While PPO on carefully selected prompt subsets converges faster, validating whether it consistently outperforms full-set PPO training across larger models and better reward models requires more rigorous experiments. We leave this for future exploration.\n\n**Weakness 2:** Minor concern: The authors do not address whether their method is applicable to thinking models or whether it would be equally effective for larger models.\n\n**Answer:** We appreciate the reviewer raising this point. At present, Reinforcement Learning approaches, such as RLVR and RLHF, remain the primary methods for enhancing thinking models. Our proposed method, however, is specifically tailored for DPO training (with potential extensions to PPO as suggested in the above answer), which is not typically used for thinking models. Regarding the applicability of our method to larger models in DPO training, recent studies [1] have demonstrated that strategic data selection can significantly improve dialogue performance in models as large as the 70B Llama model. We are confident that our method holds promise for DPO training at such scale, and plan to investigate this further in future work when resource constraint problem is solved.\n\n[1] P Yu, W Yuan, et al. Rip: Better models by survival of the fittest prompts. (2025)."}, {"review_id": "K8e7TxMC0C", "reviewer": "Reviewer_Z5aN", "summary": "The paper presents a data selection method for DPO, which uses a Bayesian Aggregation approach to select data with maximum margins. The authors first use theoretical analysis to motivate and justify why exogenous error can lead to parameter shrinkage and hurt learning, then propose a margin-based data selection method to compensate parameter shrinkage. With some analysis and comparison of reward margin from different sources, the authors conclude to use an aggregation of multi-source margins to perform data selection. The method is applied to both one-pass DPO and iterative DPO. Experiments with multiple LLM models and datasets demonstrate that the proposed data selection outperforms existing data selection baselines, and also outperforms training with the full dataset.", "questions": "- Can the method be applied to PPO and other online RL algorithms?\n- When using reinforcement learning with verifiable reward, where the reward is relatively accurate, will it still be beneficial to do data selection? How does it fit into the current framework?", "limitations": "Appendix D discusses some limitations about scaling up.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths\n- The paper is in general well-presented. The idea is clearly motivated and makes intuitive sense.\n- The proposed method is solid with theoretical interpretation. The numerical analysis in Figure 1 is also interesting.\n- The experiment results look promising.\n\n### Weakness\n- The idea of selecting preference data with large margin doesn't sound very new to me. But I find the analysis in this paper helpful and grounded.\n- It would be better if the method can be extended to broader scenarios, not only DPO. For example, does it apply to online RL algorithms like PPO?\n- The algorithm presented in Section 3.2 is a little complicated and expensive with multiple steps and calculations. It is also not very clear what are the K reward sources used and how to select them in practice. It would be nice if the authors can describe it in more details and do some ablation study in experiments.\n- The notations can be made more clear and simpler in section 3.1.", "was_revised": true, "final_justification": "Thank the authors for detailed response. It sounds good to me. Given the interesting analysis and good results, I decide to raise my score to accept. Look forward to seeing the final version with polished writing as promised.", "rebuttal": "We thank the reviewer for the encouraging feedback and valuable comments.\n\n**Weakness 1:** The idea of selecting preference data with large margin doesn't sound very new to me. But I find the analysis in this paper helpful and grounded.\n\n**Answer:** Thank you for acknowledging the value and rigor in our analysis. While margin-based filtering has been mentioned in concurrent work, these studies do not systematically analyze how different aspects of margin (e.g, sign, noise, and aggregation manner) impact data quality and DPO training. Our paper is the first to provide clear criteria for selecting informative samples and filtering out toxic ones, addressing this gap in the current literature.\n\n\n**Weakness 2 and Question 1:** It would be better if the method can be extended to broader scenarios, not only DPO. For example, does it apply to online RL algorithms like PPO?\n\n**Answer:** Thank you for your question. We answer your question from two perspectives: (1) Data selection remains important for efficient PPO training, and margin-based guidance can extend to RL scenarios. To illustrate, we compared two prompt sets: BeeS-selected 2k samples from UltraFeedback versus randomly selected 2k samples, performing RLHF on our SFT version of Llama-3.2-3B (using PairRM as the reward model). The AlpacaEval1.0 win rates were 88.84% for BeeS versus 78.12% for random (initial win rate is 61.49%). (2) While PPO on carefully selected prompt subsets converges faster, validating whether it consistently outperforms full-set PPO training across larger models and better reward models requires more rigorous experiments. We leave this for future exploration.\n\n\n**Weakness 3:** The algorithm presented in Section 3.2 is a little complicated and expensive with multiple steps and calculations. It is also not very clear what are the K reward sources used and how to select them in practice. It would be nice if the authors can describe it in more detail and do some ablation study in experiments.\n\n**Answer:** Thanks for your suggestion. We infer that you think the algorithm details are computationally expensive and the writing should be easier to follow. First, we have made a flowchart to better illustrate algorithm details (i.e., three steps of margin calculation and margin aggregation with concrete examples showing how low-margin samples are deprioritized during selection) and will add it to the revised manuscript. Second, we claim that the compute costs are acceptable for two key reasons: pre-DPO training requires only a small model trained on a randomly selected subset (size of 2k samples); margin calculations represent a one-time computational investment, and the selected subset can be used for training any subsequent model.\n\nIn our experiments, we employed two reward sources following a straightforward selection principle: including at least one implicit and one external reward model. We conducted an ablation study on K using the Llama3-8B instruct and base models with the Llama-UltraFeedback dataset. We evaluated results when using one to three reward sources, and we observed that integrating additional reward sources yields diminishing returns when $K\\geq 2$, hence we recommend K=2.\n\n| K | Ins-8B |  | Base-8B |  |\n| -------- | -------- | -------- |-------- | -------- |\n|      | LC WR     | WR     | LC WR     | WR     |\n| $r_{im,3b}$    |   33.71   |   32.92   | 18.88 | 16.25 |\n| $r_{im,3b},r_{ex}$     | 36.36     | 36.47     | 21.67 | 20.01 |\n| $r_{im,3b}, r_{ex}, r_{im,8b}$     | 36.53     | 37.47     | 21.83 | 20.37 |\n\n**Weakness 4:** The notations can be made more clear and simpler in section 3.1.\n\n**Answer:** Thank you for your suggestion. We use the concise table below to explain the meaning of the notation.\n\n| r | x | $y_w$ / $y_l$ | $\\phi$ / $\\Delta\\phi$| $\\zeta$|$\\omega$|\n| -------- | -------- | -------- |-------- |-------- |-------- |\n| Reward    | \tInput prompt    | Preferred / dispreferred response  |  \tFeature function in the reward model / relative feature function | \tExogenous error in preference| \tLearnable parameters|\n\nWe will include this table in the revised version to help readers better understand the notations and ideas. Thank you!\n\n\n**Question 2:** When using reinforcement learning with verifiable reward, where the reward is relatively accurate, will it still be beneficial to do data selection? How does it fit into the current framework?\n\n**Answer:** Thank you for raising this important point. We argue that data selection remains beneficial in RLVR contexts for two key reasons: (1) Removing overly easy or difficult prompts (analogous to near-zero margin cases in DPO) significantly accelerates RLVR convergence, as supported by dynamic sampling results in DAPO [1]. (2) Targeting prompts where the model's rollout accuracy approaches 50% ensures exposure to cases with the largest reward contrast. This is similar in spirit to prioritizing large-margin examples in DPO: it creates a clear distinction between frequently generated incorrect chains-of-thought and correct ones. As above, data selection can be naturally fit into current RLVR: keeping those learnable prompts that LLM rollout accuracy is near 50%. \n\n**Limitation** Appendix D discusses some limitations about scaling up.\n\n**Answer:** Regarding the applicability of our method to larger models in DPO training, recent studies [2] have demonstrated that strategic data selection can significantly improve dialogue performance in models as large as the 70B Llama model. We are confident that our method holds promise for DPO training at such scale, and plan to investigate this further in future work when the resource constraint problem is solved.\n\n[1] Yu, Qiying, et al. Dapo: An open-source llm reinforcement learning system at scale. (2025).\n\n[2] P Yu, W Yuan, et al. Rip: Better models by survival of the fittest prompts. (2025)."}, {"review_id": "bIsJodxLnI", "reviewer": "Reviewer_NjgC", "summary": "This paper proposes a novel data-centric approach to improve the alignment of Large Language Models (LLMs) through Direct Preference Optimization (DPO) by emphasizing preference data selection instead of modifying the learning objective. The authors introduce a method called BeeS (Bayesian Aggregation for Preference Data Selection), which selects training samples by aggregating multiple reward signals and prioritizing those with consistent, high preference margins. The approach is theoretically motivated and empirically validated across various LLMs and datasets.", "questions": "1) Can you add SimPO to Figure 4 (right)?\n\n2) Have you tried comparing with a strategy similar to that used for argilla/dpo-mix-7k? That is only selecting preference pairs where the chosen response has a high score.\n\n3) A known issue of DPO is that it pushes probability mass away from the preference dataset and to unseen responses [1]. Is this behavior still present in your methodology?\n\n[1] Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for llm alignment? a comprehensive study, 2024", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper tackles an underexplored aspect of preference-based alignment—data quality and selection—shifting the focus from algorithmic to dataset-level optimization.\n- The authors provide mathematical analysis demonstrating how reward model noise leads to parameter shrinkage and how large-margin data selection counteracts this.\n- The method is relatively simple.\n- The authors provide extensive testing of their proposed method.\n\nWeaknesses:\n- The methodology relies on a full training run of DPO.\n- No confidence intervals in the results. Given that the methodology relies on two different DPO runs (one on the full dataset and one on the selected sub-dataset) I expect a lot of variability in the results. It would be good to understand how consistent are the results of this methodology.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the encouraging feedback and constructive suggestions.\n\n**Weakness 1:** The methodology relies on a full training run of DPO.\n\n**Answer:** Our method does not require full dataset DPO training. During the *in-distribution pre-DPO training* phase, we randomly select a small seed dataset of only 2,000 samples for DPO training—significantly smaller than the full datasets (60k, 80k, and 90k samples for UltraFeedback, HH, and TL;DR). We conduct full dataset DPO training solely as a baseline comparison to demonstrate our method's effectiveness and efficiency. \n\n\n**Weakness 2:** No confidence intervals in the results. Given that the methodology relies on two different DPO runs (one on the full dataset and one on the selected sub-dataset) I expect a lot of variability in the results. It would be good to understand how consistent are the results of this methodology.\n\n**Answer:** Thank you for your suggestion. We agree that multiple uncertain factors exist in the subset selection process, but our Bayesian aggregation of different reward margins ensures relatively stable final subsets and DPO training outcomes. We provide ablation results using varying numbers of reward sources (K=1, 2, 3) with Llama3-8B instruct and base models on the Llama-UltraFeedback dataset. The results demonstrate stability when K≥2, confirming the robustness of our approach.\n\n| K | Ins-8B |  | Base-8B |  |\n| -------- | -------- | -------- |-------- | -------- |\n|      | LC WR     | WR     | LC WR     | WR     |\n| $r_{im,3b}$    |   33.71   |   32.92   | 18.88 | 16.25 |\n| $r_{im,3b},r_{ex}$     | 36.36     | 36.47     | 21.67 | 20.01 |\n| $r_{im,3b}, r_{ex}, r_{im,8b}$     | 36.53     | 37.47     | 21.83 | 20.37 |\n\nRegarding confidence intervals, AlpacaEval2.0 evaluation using GPT-4 API is stable (the standard error of Win Rate mostly fall within $(1.0,1.5)$) but expensive (more than $10 per evaluation). Due to these reasons, most previous work has not conducted multiple evaluations for confidence interval calculations.\n\n**Question 1:** Can you add SimPO to Figure 4 (right)?\n\n**Answer:** Thanks for your suggestion. The SimPO results, (LC Win Rate, Win Rate), for Figure 4 (right) setup is as follows, and we will add it to the revised manuscript.\n\n\n| Fullset | Random 6k | BeeS 6k |\n| -------- | -------- | -------- |\n| (20, 18.72)     | (12.73, 11.3)     | (20.53, 20.12)   |\n\n\n\n**Question 2:** Have you tried comparing with a strategy similar to that used for argilla/dpo-mix-7k? That is only selecting preference pairs where the chosen response has a high score.\n\n**Answer:** Thank you for your suggestion. We found that in preference pairs where the chosen response has top scores, the average margins are more than one times smaller than those of top-margin preference pairs. For instance, the average margins of selected 6000 samples by BeeS and the strategy is 18.46 and 8.89 for Llama-UltraFeedback dataset. Additionally, results using this approach are 2-5 points lower than the BeeS strategy. \n\n\n**Question 3:** A known issue of DPO is that it pushes probability mass away from the preference dataset and to unseen responses. Is this behavior still present in your methodology?\n\n**Answer:** Thank you for raising this interesting question. After read this great article, we are aware that DPO tends to generate biased policies favoring out-of-distribution responses due to its off-policy training nature, and can be alleviated by iterative DPO. Our method likely retains this behavior since we focus on data selection rather than addressing the fundamental on-policy versus off-policy training difference. The phenomenon may be reduced but not eliminated in our approach, and we will add this discussion to the Related Work section in the revised manuscript."}, {"review_id": "IDTziMTlYa", "reviewer": "Reviewer_8aHM", "summary": "This paper claims that the reward margin is important for reward learning. They demonstrate the margin influence internally and externally. They introduce a method named BeeS to broaden the overall reward margin. The experiment shows the superiority of their method.", "questions": "Refer to above.", "limitations": "Refer to above.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The idea and motivation are reasonable.\n\n2. The method provides a theoretical explanation that appears reasonable.\n\n3. The experimental results are strong.\n\nWeaknesses:\n\n1. In principle, pursuing a larger margin may improve reward learning, but when does a larger margin lead to better or worse outcomes? A theoretical and practical explanation is needed.\n\n2. In Figure 1, how is the External Reward Margin calculated? You state that \"the correlation between implicit and external reward margins is notably weak.\" Could this be due to inconsistent data distribution? Does this imply that the implicit reward results are consistent? Does the low correlation between the outputs of the two reward models in the right subplot of Figure 1 further support this point?\n\n3. You mention using the BeeS method with three steps for low-margin cases. Including a small diagram illustrating the general process would enhance clarity.\n\n4. In the Bayesian aggregation section, what are the advantages of this approach compared to traditional methods like averaging, finding maximum/minimum values, or Pareto optimization? Fundamentally, this is a reward margin fusion problem.\n\n5. For well-trained models like Qwen3, compared to less well-trained models like Qwen2, does the effectiveness of your method diminish? When does your method perform best, and when is its effect less pronounced?", "was_revised": true, "final_justification": "I lean to weakly acceptable.", "rebuttal": "We thank the reviewer for the encouraging feedback and insightful questions.\n\n**Weakness 1:** In principle, pursuing a larger margin may improve reward learning, but when does a larger margin lead to better or worse outcomes? A theoretical and practical explanation is needed.\n\n**Answer:** Thank you for your question. As discussed in Section 3.1, our theoretical analysis demonstrates that exogenous label errors cause parameter shrinkage toward zero, while larger margins provide a counterbalancing effect through parameter inflation, enabling correct parameter learning. Generally, larger margin preference pairs improve DPO training and accelerate convergence. However, excessive optimization can cause overfitting or model breakdown when margins become too large (e.g., training for many epochs), which is discussed in Appendix C.4.\n\n\n\n**Weakness 2:** In Figure 1, how is the External Reward Margin calculated? You state that \"the correlation between implicit and external reward margins is notably weak.\" Could this be due to inconsistent data distribution? Does this imply that the implicit reward results are consistent? Does the low correlation between the outputs of the two reward models in the right subplot of Figure 1 further support this point?\n\n**Answer:** In Figure 1 (left panel), we use the skywork-reward-llama-3.1-8B-v2 to calculate the external reward margin as $r(x,y_w)-r(x,y_l)$. For the rest questions concerning consistency or inconsistency between reward models, we agree with you that it stems from similar or different training data distribution. Moreover, we should note that inconsistency primarily occurs on samples with ambiguous preferences.\n\n\n\n**Weakness 3:** You mention using the BeeS method with three steps for low-margin cases. Including a small diagram illustrating the general process would enhance clarity.\n\n**Answer:** Thank you for this helpful suggestion. We have made a flowchart to better illustrate algorithm details, including the three steps of margin calculation and margin aggregation with concrete examples showing how low-margin samples are deprioritized during selection. This flowchart will be included in the revised manuscript.\n\n\n**Weakness 4:** In the Bayesian aggregation section, what are the advantages of this approach compared to traditional methods like averaging, finding maximum/minimum values, or Pareto optimization? Fundamentally, this is a reward margin fusion problem.\n\n**Answer:** The advantage is that Bayesian aggregation appropriately penalizes samples with at least one source of low margin while incorporating margin information from all sources. For example, average and max strategies may incorrectly rank samples with margins like (0.2, 0.6) versus (0.35, 0.45), while the min strategy cannot distinguish between (0.6, 0.8) and (0.6, 0.9). As for Pareto optimization, it is difficult to define a proper Pareto solution in this context. To further support our claim, we conducted an ablation study on different margin fusion strategies. Results below show that BeeS outperforms other heuristic strategies.\n\n\n| fusion strategy | Llama-3-8B-Instruct |  | Mistral-7B-Instruct |  |\n| -------- | -------- | -------- |-------- | -------- | \n|      |   LC WR   |  WR   |  LC WR  |  WR   | \n|   average   |  35.97    |  35.79   |   23.67 | 18.21 | \n|   max   |   33.99  | 33.04  | 24.56 | 18.09 |\n|   min   | 35.49 | 35.10 |25.19 | 19.00|\n|   BeeS   | 36.36 | 36.47 |26.04 |20.53|\n\n**Weakness 5:** For well-trained models like Qwen3, compared to less well-trained models like Qwen2, does the effectiveness of your method diminish? When does your method perform best, and when is its effect less pronounced?\n\n**Answer:** The Qwen-3 series are thinking models that have undergo lots of post-training for various tasks including dialogue. We tested DPO training on Llama-UltraFeedback using Qwen-3-8B (disable its thinking mode), with AlpacaEval2.0 results (i.e., LC win rate and win rate) shown below. DPO struggles to effectively enhance strong thinking models. We believe this is reasonable since these models have typically undergone internal post-training (including DPO, RLHF, etc.), while DPO and data selection perform best for improving base model performance or on new tasks that have not been post-trained.\n\n\n\n| Original | Fullset | BeeS top 6k |\n| -------- | -------- | -------- |\n|  (58.8, 63)   |  (54.33, 67.65)    |  (57.56, 68.12)   |"}]

Author rebuttal:

Meta-review: (a) Summary of Scientific Claims and Findings:
The paper introduces a novel data selection method, BeeS, for improving Direct Preference Optimization (DPO) training in large language models. It proposes a Bayesian aggregation approach to select training samples with high reward margins, aiming to reduce noise and improve alignment with human preferences. Experiments demonstrate significant performance gains using only a fraction of the original dataset, highlighting the efficiency and effectiveness of the proposed method.

(b) Strengths of the Paper:
Novel Data Selection Method: The BeeS method is innovative and provides a principled way to select high-quality training samples.
Theoretical Analysis: The paper offers a detailed analysis of parameter shrinkage and inflation, providing a strong theoretical foundation.
Empirical Validation: Comprehensive experiments across multiple models and datasets demonstrate consistent improvements over baseline methods.

(c) Weaknesses of the Paper:
Limited Scope: The method is currently limited to DPO, and its applicability to other RL methods like PPO is not fully explored.
Complexity of Algorithm: The multi-step Bayesian aggregation process may be computationally expensive and complex to implement.
Lack of Confidence Intervals: The results lack confidence intervals, making it difficult to assess the variability and reliability of the findings.

(d) Reasons for Decision:
The paper presents a novel and effective approach to data selection in DPO, supported by both theoretical analysis and empirical results. The significant performance improvements and data efficiency demonstrated in the experiments are compelling. However, the limited scope and complexity of the method suggest that further exploration and simplification are needed for broader adoption.

(e) Discussion and Rebuttal:
During the rebuttal period, the authors addressed several concerns raised by the reviewers. They provided additional experiments and analyses to demonstrate the potential applicability of their method to other RL algorithms like PPO. They also clarified the computational costs and provided a flowchart to enhance the clarity of the algorithm. These responses were generally satisfactory, reinforcing the paper's strengths and addressing some of the weaknesses.
Decision: Accept (spotlight)