Title: Less is More: Improving LLM Alignment via Preference Data Selection
Venue: NeurIPS 2025
Accepted: yes
Keywords: data selection

DPO

alignment
Abstract: Direct Preference Optimization (DPO) has emerged as a promising approach for aligning large language models with human preferences. While prior work mainly extends DPO from the aspect of the objective function, we instead improve DPO from the largely overlooked but critical aspect of data selection. Specifically, we address the issue of parameter shrinkage caused by noisy data by proposing a novel margin-maximization principle for dataset curation in DPO training. To further mitigate the noise in different reward models, we propose a Bayesian Aggregation approach that unifies multiple margin sources (external and implicit) into a single preference probability.  Extensive experiments in diverse settings demonstrate the consistently high data efficiency of our approach. Remarkably, by using just 10\% of the Ultrafeedback dataset, our approach achieves 3\% to 8\% improvements across various Llama, Mistral, and Qwen models on the AlpacaEval2 benchmark.  Furthermore, our approach seamlessly extends to iterative DPO, yielding a roughly 3\% improvement with 25\% online data, revealing the high redundancy in this presumed high-quality data construction manner. These results highlight the potential of data selection strategies for advancing preference optimization.


Peer reviews:
comments: SUMMARY:
This paper claims that the reward margin is important for reward learning. They demonstrate the margin influence internally and externally. They introduce a method named BeeS to broaden the overall reward margin. The experiment shows the superiority of their method.

STRENGTHS AND WEAKNESSES:
Strengths:

1. The idea and motivation are reasonable.

2. The method provides a theoretical explanation that appears reasonable.

3. The experimental results are strong.

Weaknesses:

1. In principle, pursuing a larger margin may improve reward learning, but when does a larger margin lead to better or worse outcomes? A theoretical and practical explanation is needed.

2. In Figure 1, how is the External Reward Margin calculated? You state that "the correlation between implicit and external reward margins is notably weak." Could this be due to inconsistent data distribution? Does this imply that the implicit reward results are consistent? Does the low correlation between the outputs of the two reward models in the right subplot of Figure 1 further support this point?

3. You mention using the BeeS method with three steps for low-margin cases. Including a small diagram illustrating the general process would enhance clarity.

4. In the Bayesian aggregation section, what are the advantages of this approach compared to traditional methods like averaging, finding maximum/minimum values, or Pareto optimization? Fundamentally, this is a reward margin fusion problem.

5. For well-trained models like Qwen3, compared to less well-trained models like Qwen2, does the effectiveness of your method diminish? When does your method perform best, and when is its effect less pronounced?

QUESTIONS:
Refer to above.

LIMITATIONS:
Refer to above.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-26

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel data-centric approach to improve the alignment of Large Language Models (LLMs) through Direct Preference Optimization (DPO) by emphasizing preference data selection instead of modifying the learning objective. The authors introduce a method called BeeS (Bayesian Aggregation for Preference Data Selection), which selects training samples by aggregating multiple reward signals and prioritizing those with consistent, high preference margins. The approach is theoretically motivated and empirically validated across various LLMs and datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper tackles an underexplored aspect of preference-based alignment—data quality and selection—shifting the focus from algorithmic to dataset-level optimization.\n- The authors provide mathematical analysis demonstrating how reward model noise leads to parameter shrinkage and how large-margin data selection counteracts this.\n- The method is relatively simple.\n- The authors provide extensive testing of their proposed method.\n\nWeaknesses:\n- The methodology relies on a full training run of DPO.\n- No confidence intervals in the results. Given that the methodology relies on two different DPO runs (one on the full dataset and one on the selected sub-dataset) I expect a lot of variability in the results. It would be good to understand how consistent are the results of this methodology.\n\nQUESTIONS:\n1) Can you add SimPO to Figure 4 (right)?\n\n2) Have you tried comparing with a strategy similar to that used for argilla/dpo-mix-7k? That is only selecting preference pairs where the chosen response has a high score.\n\n3) A known issue of DPO is that it pushes probability mass away from the preference dataset and to unseen responses [1]. Is this behavior still present in your methodology?\n\n[1] Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for llm alignment? a comprehensive study, 2024\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a data selection method for DPO, which uses a Bayesian Aggregation approach to select data with maximum margins. The authors first use theoretical analysis to motivate and justify why exogenous error can lead to parameter shrinkage and hurt learning, then propose a margin-based data selection method to compensate parameter shrinkage. With some analysis and comparison of reward margin from different sources, the authors conclude to use an aggregation of multi-source margins to perform data selection. The method is applied to both one-pass DPO and iterative DPO. Experiments with multiple LLM models and datasets demonstrate that the proposed data selection outperforms existing data selection baselines, and also outperforms training with the full dataset.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- The paper is in general well-presented. The idea is clearly motivated and makes intuitive sense.\n- The proposed method is solid with theoretical interpretation. The numerical analysis in Figure 1 is also interesting.\n- The experiment results look promising.\n\n### Weakness\n- The idea of selecting preference data with large margin doesn't sound very new to me. But I find the analysis in this paper helpful and grounded.\n- It would be better if the method can be extended to broader scenarios, not only DPO. For example, does it apply to online RL algorithms like PPO?\n- The algorithm presented in Section 3.2 is a little complicated and expensive with multiple steps and calculations. It is also not very clear what are the K reward sources used and how to select them in practice. It would be nice if the authors can describe it in more details and do some ablation study in experiments.\n- The notations can be made more clear and simpler in section 3.1.\n\nQUESTIONS:\n- Can the method be applied to PPO and other online RL algorithms?\n- When using reinforcement learning with verifiable reward, where the reward is relatively accurate, will it still be beneficial to do data selection? How does it fit into the current framework?\n\nLIMITATIONS:\nAppendix D discusses some limitations about scaling up.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper presents a novel preference data selection method designed to enhance DPO training effectiveness and efficiency. Specifically, the authors address noise reduction through a margin-maximization principle and propose a Bayesian Aggregation method to unify multiple margin sources (both external and implicit). Using only 10% of the original data, their approach achieves a 3-8% performance improvement.

STRENGTHS AND WEAKNESSES:
**Strengths**

1. The paper introduces an innovative preference data selection method, the Bayesian Aggregation for Preference data Selection (BeeS) strategy, which incorporates signals from both external rewards and DPO implicit rewards. It deprioritizes preference pairs exhibiting low reward margins from any single reward source to mitigate potential noise.
2. The authors provide a detailed analysis of Parameter Shrinkage and Inflation Analysis, highlighting how inaccuracies in the reward model can cause LLM parameters to shrink toward zero.
3. The paper includes comprehensive comparative experiments and analyses that demonstrate the method's effectiveness. Using only 10% of the data, the approach achieves a 3-8% performance improvement, with significant performance gains even in online DPO scenarios.


**Weaknesses**

1. The proposed method is somewhat limited in scope, as the analysis and experiments focus exclusively on the DPO method. It remains unclear whether the approach can be extended to other, potentially superior RL methods such as PPO, GRPO, etc.
2. Minor concern: The authors do not address whether their method is applicable to thinking models or whether it would be equally effective for larger models.

QUESTIONS:
See above

LIMITATIONS:
See above
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03