Principled Data Selection for Alignment: The Hidden Risks of Difficult Examples
Authors: Chengqian Gao
Haonan Li
Liu Liu
Zeke Xie
Peilin Zhao
zhiqiang xu
Venue: ICML 2025 poster (2025)
Keywords: alignment
data difficulty
large language models
Primary area: deep_learning->large_language_models
TL;DR: LLM alignment, particularly DPO, suffers from examples that are too difficult for the model to handle.

Abstract: The alignment of large language models (LLMs) often assumes that using more clean data yields better outcomes, overlooking the match between model capacity and example difficulty. Challenging this, we propose a new principle: *Preference data vary in difficulty, and overly difficult examples hinder alignment, by exceeding the model's capacity*. Through systematic experimentation, we validate this principle with three key findings: (1) preference examples vary in difficulty, as evidenced by consistent learning orders across alignment runs; (2) overly difficult examples significantly degrade performance across four LLMs and two datasets; and (3) the capacity of a model dictates its threshold for handling difficult examples, underscoring a critical relationship between data selection and model capacity. Building on this principle, we introduce *Selective DPO*, which filters out overly difficult examples. This simple adjustment improves alignment performance by 9-16\% in win rates on the AlpacaEval 2 benchmark compared to the DPO baseline, surpassing a series of DPO variants with different algorithmic adjustments. These results together illuminate the importance of aligning data difficulty with model capacity, offering a transformative perspective for improving alignment strategies in LLMs. Code is available at https://github.com/glorgao/SelectiveDPO

Principled Data Selection for Alignment: The Hidden Risks of Difficult Examples
Chengqian Gao 1
Haonan Li' Liu Liu
Zeke Xie
Peilin Zhao 24 Zhiqiang Xu
Abstract
0.30
The alignment of large language models (LLMss) often assumes that using more clean data yields better outcomes, overlooking the match between model capacity and example difficulty. Challeng- ing this, we propose new principle: "Preference data vary in difficulty, and overly difficult exam- ples hinder alignment, by exceeding the model's s capacity." Through systematic experimentation, we validate this principle with three key findings: (1) preference examples vary difficulty, as evi- denced by consistent learning orders across align- ment runs; (2) overly difficult examples signif- icantly degrade performance across four LLMs and two datasets; and (3) the capacity of a model dictates its threshold for handling difficult exam- ples, underscoring a critical relationship between data selection and model capacity. Building on this principle, we introduce Selective DPO, which filters out overly difficult examples. This simple adjustment improves alignment performance by 9-16% in win rates on the A AlpacaEval bench- mark compared to the DPO baseline, surpassing a series of DPO variants with different algorithmic adjustments. These results together illuminate the importance aligning data difficulty with model capacity, offering transformative perspective for improving alignment strategies in LLMs. Code is available at https: / /ttp//giittt github.. com/ gloorg ao/SelectiveDD
WR (win mite a/st gpt-4-turbo)
9.4% reduction due to difficult examples
0.25
0.20
0.15
Random Ordering
by Difficulty
80
Data Percentage (%)
Figure 1:
Overly difficult examples hinder the alignment. Training on difficult examples, identified by high valida- tion loss, adversely affects alignment and decreases overall performance by 9,4% in win rate. The results are from experiments with four SFT models on the UltraFeedback- binarized dataset, i.e., Figure 3.
stages of training large language models (LLMs): pre- training (Lee et al., 2021; Penedo et al., 2023; Tang et al., 2024) and supervised fine-tuning (SFT) (Cao al., 2023; Qin et al., 2025; Zhou et al., 2023), by adhering to well- established principles. However, in the third stage, i.e. preference alignment (Askell al., 2021; Weidinger et al., 2021), data selection principles are often implicit and su- perficial, potentially limiting the alignment between LLM outputs and human preferences.
Prior studies in alignment underscore the importance of se- lecting error-free data by demonstrating the presence and negative impacts of mislabeled data (Wang al., 2024a; Gao et al., 2024), noisy feedback (Mitchell, 2023; Chowd- hury et al., 2024), and data with low agreement (Argilla, 2024), implicitly assuming that all error-free data are bene- ficial regardless of the model's capacity. However, we argue this assumption overlooks the relationship between data difficulty and model capacity. Our experiments show that overly difficult examples not only fail improve alignment but can actually hinder the performance (see Figure 1). This observation motivates our systematic investigation into how example difficulty affects alignment performance.
1. Introduction
Data selection focuses on identifying the most valuable sub- set from a dataset while excluding ineffective samples (Al- balak et al., 2024). It significantly improves the first two
'This work is done when Chenggian Gao works as an in- tern in Tencent AI Lab. 'MBZUAI "Tencent Inc 'HKUST (Guangzhou) "SJTU. Correspondence uliu aliu@tencent.com>> Peilin Zhao <masonzhao@e wataaaaottennettcmm..... Zhiqiang Xu <zhhqiiang.x xu ac.a..
Liu Liu <leonli-
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Our main contribution is a new principle for preference data selection, which emphasizes the match between model

---

Principled Data Selection for Alignment
capacity and example difficulty:
we focus on direct preference optimization (DPO) (Rafailov et al., 2024), a method known for its simplicity and robust performance in alignment tasks (Dubey et al., 2024). DPO trains policy model, no. on a dataset 'D containing prompt T; preferred response yw, and rejected response y1- The training objective incorporates a reference SFT model, Tra and a hyperparameter, 3, to control the divergence between Th and Tee
Preference data vary in difficulty, and overly difficult exam- ples hinder aligument. by exceeding the model's capacity.
This principle has three key claims: (1) preference data can be categorized by difficulty levels, (2) overly difficult ex- amples can harm alignment performance, and (3) difficulty is relative to the model's greater capacity, can benefit from more difficult examples. We validate this principle through systematic experiments. Specifically;
Sraacity-largee models, with
(DPO D) -I(..yu.
(I)
To To (2)
Blog To (30 lax) Tret (30 (z)
)
log a (8 log T (3ho
Preference examples vary in difficulty level (Section 3). We show that, in DPO (Rafailov et al., 2024), the order in which testing examples are correctly classified by the im- plicit reward model is consistent across different runs and training data. This robust ordering reflects the existence of inherent example difficulties. Based on this observation, we use validation loss as a computational proxy to systemati- cally identify and rank example difficulty.
2.2. Quantifying the Example Difficulty
Learned step measure of difficulty. An example's learned step is defined as the earliest training step after which the model reliably distinguishes preferred responses from rejected answers. This is formalized as:
Difficult examples hinder alignment (Section 4). We reveal that difficult examples-identified by high validation loss loss-significantly hinder alignment. Our experiments across two datasets and four pre-trained models show consistent performance drops when including these difficult examples. These challenging examples emerge naturally during data collection, rather than through artificial construction. This highlights the imperfections of the previous principle and calls for new data selection principle for alignment tasks.
LS(I, ywu 31) = min the to, (y /xr) TTer(y (z))
(2)
9 log To, (21(z) Tieer (z)
tim 3 log
8, Vt > tin
A similar metric has been explored by Wu et al. (2021).
The difference is that we calculate Eq. (2) exclusively on held-out examples, ensuring it reflects intrinsic difficulty rather than the order of data presentation (Zhu et al., 2024a), Larger learned steps indicate more difficult examples. For all experiments, we set d = 0.4.
Difficult examples exceed the model's capacity (Sec- tion 4). We demonstrate that example difficulty interacts directly with model capacity. Experiments with models of 3B, 8B, and 14B parameters show that larger models benefit from higher proportions of difficult examples, confirming that difficulty must be calibrated to the model's capacity.
Validation loss as an alternative difficulty proxy. We borrow validation loss (Wu et al., 2021; Rampp et al., 2024) as a computationally cheaper alternative to the learned step. Specifically, for specific example (I, yu 31) from D D.D. validation loss is defined as:
Filtering out overly difficult examples yields remarkable gains (Section 5 and 6). Finally, we validate principle with new method, Selective DPO, which filters out overly difficult examples. This approach achieves 9-16% higher win rate on AlpacaEval 2 (Dubois et al., 2024) compared to standard DPO (Rafailov et al., 2024), outperforming state- of-the-art art methods such as SimPO (Meng et al., 2024) and R- DPO (Park al., 2024) while maintaining better perplexity and implicit reward margins.
VL(x, Mwrs M1)
(3)
- -log logo (glo To To(yulz))
To(Un(r)
3 log
where Tg = arg min., (To, D) is a reference model trained on the subset D c D. Low validation losses indi- cate easier examples. To compute the validation loss, we partition D equally into D and D \ DD, train on one partition, evaluate on the other, and finally output average results over three runs.
2. Preliminaries
2.1. Preference Alignment with DPO
Preference alignment (Ouyang et al., 2022) aims to align the outputs of L.MSs with human ethics and styles, ensuring that these models are safe, reliable, and effective for real- world applications (Christiano al., 2017). In this study,
Throughout this work, the term reference model (Tz) refers to the aligned model trained on subsets of the training examples, while reference SFT model (тTrT) specifically denotes the model defined in Eq. (1).
2

---

Principled Data Selection for Alignment
Learned Step
Spearman's Rank Correlation
Velddriion
Jaccard Similarity
Promorts sorted by aog, leamned step
Gemma
Gemma
it
-98-i
0.9
LLama-3
0.90
LLama-3 -88-instuctt
600
0.8
0..55
400
Mistro
0.50
Mistral 78 SPT
0.7
7B SFT
0.75
Gemma-2
0.70
Gemma- 2
-2Hz
28-4
Gemma-2
Liama-3
Mistral
Gemma-2
Leamed Gemma-2
LLama-3
Mistral 78--FT
Gemman-
Leamed Step Gammaa-2
LLama-3
Nistral
Gemma-2
96-1
-SS-nitruct 7B--FT -284t
-06-1
-S8-intruct
-28-1
-9-4t
3B-Instract
-78-SFT
-26-7
Figure 2: Examples are learned in consistent orders across different runs of the same LLM, despite variations in the training data and random seeds. Left: The learned step (ranging from 1 to 948) represents the step at which the implicit reward model distinguishes between preferred and rejected responses (see Eq. (2), threshold 5 = 0.4). X-axis: 40 unique combinations of model size (4 total) and training data subset (10 per model). Y-axis: 300 test examples, sorted by average learned step across 40 runs. Color gradients encodes difficulty, Middle: Two Spearman's 's rank correlation matrices. Lower triangle: correlations of learned step across runs; upper triangle: validation loss correlations. Right: Two Jaccard similarity matrices for difficult examples (top 50%) defined by learned step and validation loss across runs.
3. Preference Examples Vary in Difficulty
at scale is computationally expensive, as the computing of learned step requires evaluating the model after each gra- dient update. To address this, we adopt the validation loss from the curriculum learning literature (Wu et al., 2021; Rampp et al., 2024) (see Eq(3)). Specifically, we train six reference models using the DPO objective on the randomly sampled half training set and evaluate the validation loss for examples on the other half. We refer the difficult examples to examples with large validation loss.
Examples are learned remarkably consistent order, re- vealing the inherent example difficulty. We then validate the validation loss an effective measure of this difficulty for alignment tasks.
3.1. The Underlying Example Difficulty
While various metrics such as length (Spitkovsky et al., 2010; Tay et al., 2019; Nagatsuka et al., 2023) and per- plexity (Wu et al., 2024) have been proposed to measure difficulty of text samples, their ability to reliably capture example difficulty remains controversial (Campos, 2021). We address this concern by demonstrating: (1) examples have distinct learned steps (see Eq. indicating different difficulty levels, and (2) these learned steps are consistent across runs with different training data and random seeds.
Definition 3.1 (Difficult example). A preference example (x, Uue 30) is considered a difficult example if its validation loss is no less than a specified value:
VL Beur 91) 2 Q(T).
Remark 3.2. We introduce flexible threshold Q(T) which is the r-quantile of the validation loss. This accounts for the lack of formal definition of sample difficulty (Zhu et al., 2024b) and the variation in loss distributions across models.
In Figure 2 (left), we visualize the learned steps of 300 test examples from U'traffeedback-biiee binarrieedaaiaeeeeeee where darker colors indicate more training steps needed for model com- prehension. Results from 10 runs show consistent learning order across different models (Jiang et al., 2023; AI@ Meta, 2024; Team et al., 2024) varying in size (2B-9B), training stage, and data sampling. This consistency confirms that examples vary in difficulty, allowing us to discuss difficult examples without debating various definitions of difficulty.
To assess whether the validation loss effectively approxi- mates the learned step, examine the correlation between difficulty rankings produced by these two measures. Using Spearman's 's rank correlation, we compared rankings across different runs and models. As shown in the middle panel of Figure 2, the validation loss exhibits patterns remarkably similar to the learned step. Furthermore, the high correlation coefficients between average learned step and average vali- dation loss across the four models (0.9258, 0.9227, 0.9336. and 0.9283) validate the effectiveness of validation loss as computationally efficient proxy for learned step. Addition- ally, the Jaccard similarity between difficult example sets (defined as top 50% by either metric) remains consistently high for each model (Figure 2, right). confirming that both measures identify similar sets of difficult examples.
3.2. Validation Loss as Proxy for Learned Step
The robust learning order suggests the existence of diffi- cult examples-some examples are consistently harder for LLMs to understand. However, identifying these examples
co/ datasataasesss.e patasets/Huuggingh
3

---

Principled Data Selection for Alignment
Random Ordering (DPO)
Sorted by VL (Fram Easy to Difficult)
Selected by VL (Shuffied)
Mistral:B:SSF (Argilla-7k)
Qwen-2.5.72 2.T lakk)
Llama.3 8B.SFT (Argilla 7k]
Gemma-22 (aaimaa---9------ (Argilla 7k)
WR (win rate alst got-4-turoo)
0.25
2.0
2.0
2.0
2.0
0.25
0.25
0.30
exammpp
0.20
1.5
1.5
1.5
0.25
-1.5
0.20
0.20
0.15
1.0
1.0
1.0
0.20
1.0
0.15
0.15
0.15
0.5
0.5
0.10
0.5
0.5
0.10
0.10
0.05
0.0
60
100
40
80
20
60
100
40
100
WR (win rate eSt gut-4-turbe)
Mistral-7B-Sr (UF-binerized)
Owen-225-7B-SFF (UF-binerized)
Llama-3-68-20 3-B-SFF (UF-binarized)
Gemma-2-9B-SF7 (UF-binarized)
0.35
2.0 0.25
2.0
0.30
2.0
2.0
of somed examples
0.30
1.5
0.20
1.5
0.25
1.5
0.25
0.20
0.25
0.20
0.15
0.15
0.20
1.0
0.15
0.5
0.10
0.10
0.15
0.5
0.10
0.05
0.0
0.0
0.05
0.0
0.0
0
20
40
60
80
100
20
40
60
80
100
0
20
40
60
80
100
0
20
40
60
80
100
Data Percentage (%)
Data Percentage (%)
Data Percentage (%)
Data Percentage (%)
Figure 3: Direct Preference Optimization (DPO) struggles with difficult examples, broadly and significantly. We present the defined WR' evolution for four models trained on the argilla-mix-dpo-7 and ultrafeeddack-
datasets. The results are based on checkpoints from three I-eopch runs with different seeds. Random Ordering (DPO): Training data are presented in randomized sequence. Sorted by VI. (From Easy to Difficult): Training examples are ranked by their validation loss (VL) and presented from easy to difficult, following a curriculum learning approach. Selected by VL (Shuffled): The easiest 60% (for Argilla-7K) or 50% (for UF-binarized) of the data is selected based on VL, and examples are sampled in a random order for training. The VL measurements are displayed as bar plots. We include evaluation results (dashed lines) from the two corresponding DPO models released by Meng al. (2024) for reference.
4. Difficult Examples Hinder Alignment
following Meng et al. (2024).
In this section, we first demonstrate that difficult examples significantly degrade alignment performance across various datasets and model scales. We then investigate the factors that contribute to their difficulty through a series of system- atically designed empirical studies.
Evaluation. We employ WR', the win rate against gpt-4- turbo on 805 testing examples from AlpacaEivll (Dubois et al., 2024) with ArmoRM (Wang et al., 2024c), re- ward model with impressive performance on the Reward- Bench (Lambert et al., 2025), as the evaluator. This evalua- tion setup allows us to evaluate thousands of checkpoints.
4.1. Investigation Setup
4.2. Difficult Examples Hinder Preference Alignment
Models. We start the alignment from SFT models trained on the UltrChht-200 dataset: Mistral-7B-SFT (Jiang al., 2023), Qwen-2.5-7B-FFT (Yang et al., 2024), Llama3- 8B-SFT (AI @ Meta, 2024), and Gemma-2-9B-SFV (Team et al., 2024). This setting better demonstrates the effects of different alignment procedures (Meng et al., 2024).
As shown in Figure 3, training on difficult examples leads to significant performance declines. We compare three example-ordering strategies: (1) random ordering (standard DPO), (2) easy-to-difficult sorting by validation loss, and (3) random ordering with only easy examples. Despite us- ing the same training recipes, models consistently perform better when trained on easier examples across all four archi- tectures and both datasets. Notably, the benefits are mainly unlocked by excluding difficult examples rather than the or- dering itself, as shown by the similar performance of sorted and shuffled easy examples (Strategies 2 and 3).
hetprreeedbucc--bnizieee a widely adopted alignment dataset (Tunstall et al., 2023; Meng et al., 2024; Zhou al., 2024; Pattnaik et al., 2024), and Argilla- dpo-mix-7k3, a small but high-quality dataset.
Datasets. We use
Hyper-parameters. Following prior work, we set B = 0.01 (Zhou et al., 2024). The learning rate is swept for DPO with random ordering and directly applied to DPO with other settings. We conduct the alignment with one epoch
The performance drop due difficult examples is more pro- nounced in Mllrafeedback-bnnrriee This is aligned with the observation that Ultrafeedback bhnnfrrebbac-bbbariieedddd contains misla- beled examples (Argilla, 2024; Bartolome et al., 2023) and Argilla-dpo-mix-7 is characterized by high-quality data.
hhttps:/ httes///hhgggnggc/eeeee htcp//aaaaett///agggii dpo-mix-77 7
4

---

Principled Data Selection for Alignment
got-tttt- (pt-4-turbo))
0.20
0.20
0.15
0.15
0.10
(1-4) - 1.0, Groedy
LR=1e44, Best
0.16
Label Flipping
0.05
LR
WR
Label Flipping
(1:4)
LR=20-4
0.05
60
80
100
20
40
60
80
100
40 60
80
100
Data Parcentage (%)
Data Percentage
Data Percentage
(a) Label flipping.
(b) Distribution shift.
(c) Improper learning rate.
Figure 4: Difficulty examples are not necessarily data errors. (a): flipping the last 40% examples with higher validation
loss. (b): sorting the examples with the e-greedy sorting algorithm. In this case, each mini-batch data contains (I-e) part of casy-to-difficul examples and (c) part of randomly sampled examples. (c): increasing and decreasing the learning rate. All experiments are conducted on the Mistral-7B-SF7 model with Argilla-dpo-mix-7 dataset.
Qwen-2.5-388 TWT (Argilla-7k)
Qwen-2.5-788 (Argilla-7k)
Qwen-2. 1aiee2.55118.1 5-1 (Argilla-7k)
0.20
0.18
WR (oin rate gpe-t-uubb))
0.16
0.26
0.14
0.24
0.22
0.10
0.20
0.32
0.08
0.10
Polynomial Sweet
0.30
Spot
0.16
Spot
60
75
90
55
70
85
80
Data Percentage 196, Sorted by Dificulty)
Data Percentage (%) Sorted by Difficulty)
Data Percentage (%, Sorted by Difficuity
Figure 5: Difficult examples benefit larger models with greater capacities. Examples are sorted by their validation loss, ranging from easy to difficult. We fit the measured WR' (scatter points) using a second-degree polynomial (dashed line), identifying the peak of each parabola as the sweel spot (marker). Notably, larger models reach sweet spots at higher data percentages, indicating that model with greater capacity can manage more challenging examples. The results are from ten runs per model type, evaluated using ArmoRM (Wang et al., 2024c).
4.3. Difficult Examples Are Not Necessarily Data Errors
Learning rate sensitivity (Figure 4 (c)). We argue that the performance drop is not simply caused by the improper learning rate. We investigate this with a varying learning rate. However, adjusting the learning rate neither alleviates performance drops nor delays the decline, demonstrating that the issue is unrelated to improper optimization settings.
Before proposing our solution to filtering out difficult and harmful examples, we shed light on their traits justify their removal here. For statistics and case study difficult examples, please refer to Appendix E and F.
Mislabeled data (Figure 4 (a)). Prior work suggests that difficult examples might be mislabeled (Argilla, 2024; Bar- tolome et al., 2023). To test this hypothesis, we sort the examples by their validation loss and flip the labels of last 40% (the most difficult) examples. However, this modifi- cation does not alleviate the performance drop, suggesting that label noise is not the primary cause.
4.4. Difficult Example Exceeds Model's Capacity
We hypothesize that difficult examples bring about training tasks beyond the model's 's current capabilities, thus requiring larger models to properly understand the nuanced preference differences. To validate this hypothesis, we conduct experi- ments using Qwen-2.5 models (Yang et al., 2024) of three sizes: 3B, 7B, and 14B. The dataset is Argilla-dpo-mic Figure 5 shows clear relationship between model size and manageable example difficulty: the optimal percentage of training data (the sweet spot) increases from 64% for the 3B model to 81 for the 14B model. This scaling pattern demonstrates that larger models can effectively learn from more difficult examples, confirming the direct relationship between model capacity and example difficulty threshold.
Distribution shift (Figure (b)). Another possibility is that difficult examples represent a distinct distribution, causing catastrophic forgetting when models transition from easy to difficult examples. We test this using e-greedy sorting: each mini-bated contains portion of randomly sampled exam- ples and (1 e) portion of examples sorted by validation loss. This ensures continuous exposure to both distributions, yet shows no improvement over the greedy sorting.
5

---

Principled Data Selection for Alignment
Step 1
Step 2
Step
Train reference models on different subsets the dataset.
Compute validation loss for held-out examples and rank them by this difficulty measure.
Align the LLM using the easlest 1% of training examplas.
The sligned LLM
Preference examples
Sorted
Reference models
preference examples
Figure 6: The pipeline of Selective DPO. It extends DPO (Rafailov et al., 2024) with a principled data selection process: selecting preference examples within the model's capacity. Specifically, Selective DPO comprises three steps: (1) Train set of reference models using the DPO loss on different subsets of the training data. (2) Evaluate the reference models to compute the validation loss, which serves as proxy for example difficulty. (3) Selectively align LLMs on examples with low validation loss from easy to difficult examples.
5. Selective DPO
6. Experiments
Having verified the three key claims underpinning our data selection principle, we are now well-positioned to propose an instantiated algorithm, Selective DPO. It extends the standard DPO (Rafailov et al., 2024) by selectively training on examples within the model's capacity. The algorithm consists of three main steps, as illustrated in Figure
We evaluate the proposed preference data selection prin- ciple by benchmarking the Selective DPO algorithm on formal benchmarks: AlpacaEval 2 (Dubois et al., 2024), Arena-Hard vO. (Li et al., 2024b), and M1-Bench (Zheng et al., 2023). We report scores following each benchmark's evaluation protocol.
Train reference models. The training dataset is ran- domly split into two partitions. Using the standard DPO loss (Eq. 1), SFT models are trained separately on each partition, resulting in two reference models per split. This process is repeated three times, yielding six reference models. Unlike the reference SFT model used in the DPO objective to control KI. divergence, these reference models are specifically employed for computing validation loss.
6.1. Performance Comparison
Baselines. Data selection for alignment remains a rela- tively underexplored yet promising direction. To provide a comprehensive evaluation, we consider three categories of relevant bascline algorithms: (1) Data correction methods, including label flipping and label smoothing, aim to mitigate annotation errors; (2) DPO (Rafailov et al., 2024) and its variants such as IPO (Azar et al., 2024), KTO (Ethayarajh et al., 2024), ORPO (Hong et al., 2024), SimPO (Meng et al., 2024), and WPO (Zhou al., 2024); (3) Potential so- lutions for preference data selection, such CHES (Razin et al., (0255)designme prompts- with RM (filtering out samples with low reward margins (Gao et al., 2024)) and PPL (selecting SFT samples with moderate perplexity (Wu et al., 2024; Ji12 et al., 2024)). All baseline algorithms undergo hyperparam- eter tuning on the learning rate. Implementation details are provided Appendix C.
Rank examples by their validation loss. The trained reference models evaluate held-out examples from their respective complementary partitions (D) D). Each ex- ample assessed three times using different reference models, and the mean validation loss is computed to rank the examples in ascending order.
for refusal alignment
unsafe
Align with the selected data. comprising the lowest t percent of validation losses, are selected for alignment training. The alignment al- gorithm, such as DPO, is applied exclusively to these examples. To fully utilize the difficulty ranking, exam- ples are processed sequentially from easy to difficult.
The easiest examples,
Results (Table 1 and Figure 7). Table compares re- sults the Mistral-7B (Jiang et al., 2023) and Ilama-3- 8B (AI@Mete 2024) models. Label flipping yields only marginal gains, supporting our insight that difficult exam- ples are not necessarily data errors. In contrast, Selective DPO, which carefully selects 50% of the training data, sig- nificantly outperforms all baselines across all three bench- marks, demonstrating the strength of our data selection prin- ciple for alignment tasks. Figure extends the comparison to Gemma-2-98 (Team et al., 2024) and Qwen-2.5-7B (Yang et al., 2024), showing exceptional performance in win rate (WR) on AlpacaEval 2 and comparable performance on
Remark 5.1 (Flexible hyper-parameter r). The optimal T, which determines the percentage of selected data, depends on the data difficulty distribution and the model's capacity. In practice, 7 can be tuned using third-party evaluator such as AlpacaEval 2 (Dubois et al., 2024). For the evaluation in the next section, we set T = 50 for the UlraFeedback- binarized dataset, based on insights from Figure 3. For clarity and reproducibility, pseudocode for Selective DPO is provided in Appendix A.
6

---

Principled Data Selection for Alignment
Table I: Benchmarking results from AlpacaEval (Dubois et al., 2024), Arena-Hard (Li el al., 2024b), and MT-Bench (Zheng et al., 2023). In AlpacaEval 2, WR and LC indicate the win rate and length- length-controlled win rate against GPT-4-Turbo. We report the mean and standard variance across three runs, Arena-Hard. WR represents the win rate against GPT-4-0314 with GPT-4-Tuubb serving as the evaluator. MT-Bench scores the quality of generated responses on a scale from 10, using either GPT-4 or GPT-4-Turbo the evaluator. All results are based full parameter fine-tuning (FPFT), except for the row labeled with LoRA (Hu et al., 2022). We run this comparison the PrraFeeddbcck-bnnariie dataset.
Mistral-7B-SFT
Llama-3-8B-SF7
Method
AlpacaEval 2
Arena-Hard
MT-Bench
AlpacaEval 2
Arena-Hard
MTT-enchh
(%)
WK (%)
WR
GPT-4 Turbo GPT-4 (C(()
WR (%)
(%)
GPT- Turbo GPT44
SFT
8.4
6.2
1.3
4.8
6.3
6.2
4.6
3.3
5.2
6.6
DPO (Rafailov al., 2024)
15.1
12.5
10.4
5.9
7.3
18.2
15.5
15.9
6.5
7.7
+ Label Flipping (Wang et al., 2024a)
15.4
13.1
10.9
7.3
19.1
15.9
16.2
7.7
+ Label Smoothing (Mitchell, 2023)
15.2
12.7
10.2
7.3
17.7
14.8
15.7
7.6
RRHF (Yuan al., 2023)
11.6
10.2
5.8
12.1
10.1
6.3
5.8
7.0
SLiC-HF (Zhao
2023b)
10.9
8.9
7.3
7.4
12.3
13.7
6.0
7.6
IPO (Azar al., 2024)
11.8
9.4
7,5
5.5
7.2
14.4
14.2
17.8
6.5
7.4
CPO (Xu et al., 2024)
9.8
8.9
6.9
5.4
6.8
10.8
8.1
5.8
6.0
7.4
(Ethayarajh et al., 2024)
13.1
9.1
5.6
5,4
7.0
14.2
12.4
12.5
7.8
ORPO (Hong et al., 2024)
14.7
12.2
7.0
5.8
7.3
12.2
10.6
10.8
6.1
7.6
R-DPO (Park et al., 2024)
17.4
12.8
8.0
5.9
7.4
17.6
14.4
17.2
6.6
7.5
SimPO (Meng et al., 2024)
21.5
16.6
6.0
22.0
20.3
23.4
7.7
WPO (Zhou et al. 2024)
24.4
23.7
16.7
7.4
23.1
22.2
23.1
7.3
CHES(lowest 50%) (Razin al., 2025)
18.90.74
16.61.18
17.10.69
15.91.11
RM(highest 50%) (Gao et al., 2024)
16.2o.a6
13.11.21
19.70.61
16.11.24
PPL(middle 50%) (Wu et al., 2024)
17.30.a
15.41.10
15.3p.50
15.71.10
Selective DPO (Ours w/ LoRA)
25.4n.80
27.41.2
16.2
21.1o.73
18.31.14
22.7
Selective DPO (Ours)
27.10.63
28.91.31
17.0
7.4
24.9a.77
25.3
24.1
8.0
Qwen-2.5-7B-SFT
Gemma-2-9B-SFT
35
35
Metrics (36)
30
WR
27.2
28.2
28.0
26.4
25.7
30.1
29.1
26.7
25
23.4
20
20
15.9
16.4
15
DPO
SinPO
WPo
DPO
Simnr
WPD
S'O
DPO
SimPO
WPO
S'DPO
DPO
SimP
wro
S10FO
Figure 7: Comparison results against SimPO and WPO, with all methods tuned for their learning rates. Selective DPO
DPODPO) demonstrates superior performance in win rate (WR) and comparable results in length-controlle win rate (LC).
cogth-conttolled win rate (LC). The slightly lower perfor- mance on LC consistent with results in Table 1, where Selective DPO demonstrates better performance under WR.
the number of reference models used to compute the vali- dation loss improves performance on AlpucaEvall 2 (LCC. However, considering computational costs, training six ref- erence models strikes balance between performance and efficiency. Percentage of selected easy examples (Fig- ure 8 (b)): Increasing incorporates examples exceeding the model's capacity, leading to performance degradation, while excessively low values limit training to the simplest examples, also resulting in suboptimal performance.
We emphasize that our goal is not propose the best alignment algorithm, but to verify the proposed data selec- tion principle for alignment: selecting examples that match the model's capacity. The length exploitation issue, while beyond the scope of this paper, could potentially be ad- dressed using techniques from SimPO (Meng et al., 2024) WPO (Zhou al., 2024), which leave future work.
6.3. In-Depth Analysis of DPO vs. Selective DPO
6.2. Hyper-Parameter Study
Selective DPO outperforms DPO in terms of likelihood distribution and reward margin distribution. As shown in Figure 8(c), Selective DPO achieves distribution of nega- tive log-likelihoods (NLLs) closer zero on test prompts
Selective DPO introduces two implicit hyperparameters. Number of reference models (Figure (a)): Increasing

---

Principled Data Selection for Alignment
Reference Models
(b)
(c) Dist. of NLLs
(d) Dist. of Reward Margins
28.5
1.5
1.0
DPD
DPO
28.0
Selective DPO
0.8
Selective DFO
LC1
27
Density
1.0
Density.
0.6
27.5
0.5
0.4
27.0
25
0.2
0.0
0.0
0
-10
10
Figure 8: Hyperparameter study and in-depth analysis of Selective DPO. (a): Relationship between the number of reference models and performance. (b): Performance with different percentages of selected easy examples. (c): Negative log-likelihoods distributions on the generated responses. (d); Reward margin distributions of the implicit reward models.
WR (win rate a/st gpt-4-turbe)
clude techniques such as label flipping (Wang et al., 2024a) and confidence-basse data filtering (Gao et al., 2024). Ap- proaches like cDPO (Mitchell, 2023) and rDPO (Chowdhury et al., 2024) aim to mitigate the impact of mislabeling with- out explicitly removing mislabeled examples. In our study, we incorporate label flipping and label smoothing experi- ments to support our claim that difficult examples are not necessarily mislabeled examples, but rather those exceeding the model's capacity. Seemingly relevant work. Our study differs from general data selection research, such as Liu et al. (2024); Xia et al. (2024), which uses the term alignment but actually focuses on the SFT stage. For a comprehensive review of data selection for LLMs and curriculum learning, we refer readers to Appendix B.
Weak-to-Stree to-Strong Curriculum
0.25
0.20
0.15
7B Curfcuuum Curricluum
0.10
7B Random Ordaring
20
40
60
80
100
Data Percentage (%))
Figure 9: Weak-to-strong curriculum under-perfin under-performs.
Aligning a 7B model with examples ordered by 3B reference models yields compromised results.
from AlpacaEval 2, indicating higher confidence in gen- crated responses. Additionally, the implicit reward model learned by Selective DPO exhibits better accuracy and larger reward margins on testing examples (Figure 8(d)..
8. Conclusion and Future Work
In this work, reveal and address a critical gap in LLM alignment: the mismatch between data difficulty and model capacity. Challenging the assumption that more clean data uniformly improves alignment, we propose a novel principle for alignment tasks:
6.4. Weak-to-Strong Curriculum
To investigate whether difficult examples can be identified using smaller reference models, we compare alignment ex- periments where a 7B SFT model is trained with its own curriculum versus curriculum derived from a smaller 3B model. Results in Figure 9 show moderate benefits from the smaller model's curriculum, though slightly inferior to the model's own curriculum. This suggests that while smaller models can provide insights, data selection remains more effective when tailored to the target model's capacity.
Preference data vary in difficulty, and overly difficult exam- ples hinder alignment, by exceeding the model's capacity.
Comprehensive experiments validate the three key claims underlying this principle. Building on this data selection principle, we introduce Selective DPO, an alignment algo- rithm that selectively trains on examples within the model's 's capacity. Selective DPO achieves state-of-the-an results on benchmarks including AlpacaEval 2, Arena-Hard, and MT- Bench, with up to 16% gains in win rates over DPO, Our work advocates a paradigm shift in alignment: alignment should prioritize data difficulty relative to model capacity rather than treating all preference data equally.
7. Related Work
Response selection. The importance of selecting high- quality responses as preferred choices has been highlighted in several studies (Bai et al., 2022; Ethayarajh et al., 2022: Tunstall et al., 2023). These works focus on ensuring that preferred responses are aligned with human values. Our work builds upon these efforts in two key ways: (1) the datasets we consider already incorporate these response selection techniques, and (2) we prioritize whether prefer- ence examples fall within the capabilities of the target LLM,, rather than solely emphasizing their alignment with human values. Data correction, Efforts to address noisy labels in-
However, limitations remain: (1) Selective DPO tends to favor longer responses due to potential data bias; and (2) the proposed principle is designed and validated specifically for the DPO setting, limiting its direct applicability to RLHE, These gaps highlight opportunities for future work.
8

---

Principled Data Selection for Alignment
Acknowledgement
In International Conference on Artificial Intelligence and Statistics, pp. 4447-4455. PMLR, 2024.
The authors thank Qichao Wang, Qingyang Zhang, and Ziqiao Meng for their valuable feedback during the initial phase of this work. The authors also thank Guangyi Chen, Yongqiang Chen, Guozheng Ma, Cong Zeng, Gongxu Luo, and Loka Li for their helpful discussions and suggestions during the writing phase. The authors are grateful to the open-source communities, especially the developers and researchers of Hlignment-handbon SimPO and WPO, for generously sharing their data, results, and code.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training a helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022.
Bartolome, A., Martin, G., and Vila, D. Notus. https: inttiiusnntuu..ooo notus, 2023.
//github.
Beeching, E., Fourrier, C., Habib, N., Han, S., Lambert,
Impact Statement
N., Rajani, N., Sanseviero, O., Tunstall, L., and Wolf, T.
Open LLM leaderboard. hatps::/hugginggaeee co / spaces Hugg httpaaaa//aaaennnnnaaa lea derboard, 2023.
This paper presents work whose goal is to advance the align- ment between large language model behaviors and human values. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Bengio, Y., Louradour, J., Collobert, R., and Weston, J. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41-48, 2009.
References
AI@ Meta. Llama 3 model card. web page, 2024.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
Albalak, A., Elazar, Y., Xie, S. M., Longpre, S., Lambert, N., Wang, X., Muennighoff, N., Hou, B., Pan, L.,, Jeong, H., Raffel, C.,, Chang. S., Hashimoto, T., and Wang, W. Y. A survey on data selection for language models. Trans- actions on Machine Learning Research, 2024. ISSN 2835-8856. URL https::tt//.cc...t.. /wwprevvee.eeeeeeeeeeeeeeeee.e.e net/E Trum?id=XfHwaNTeE
Campos, D. Curriculum learning for language modeling. arXiv preprint arXiv:2108.022 027000 2021.
Survey Certification.
Cao, Y., Kang, Y., Wang. C., and Sun, L. Instruction mining:
When data mining meets large language model finetuning, arXiv preprint M.Xiv:2307006290 2023.
Ankner, Z., Blakeney, C., Sreenivasan, K., Marion, M., Leavitt, M. L.,, and Paul, M. Perplexed by perplexity: Perplexity-based data pruning with small reference mod- els. arXiv preprint ar 2055v:2405.20541, 2024.
Chang. E.,, Ych, H.-S., and Demberg, V. Does the order of training samples matter? improving neural data-to- text generation with curriculum learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics:: Main Volume, pp. 727-733, 2021.
Argilla. Ultrafeedback binarized preeerencss----eeeeed ht tps: sps:///ute /tt:://hugggnne face.. http://cccccccctttttttee/errrrli
-cleaned, 2024.
Chen, L., Li, S., Yan, J., Wang, H., Gunaratna, K., Yadav, V., Tang, Z., Srinivasan, V., Zhou, T., Huang, H., and Jin, H. Alpagasus: Training a better alpaca with fewer data. In The Twelfth International Conference on Learning Representations, 2024a. URL stttps://openeee ew . www.ee/tooorn id=
Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2 /2.00061,, 2021.
Avrahami, J., Kareev, Y., Bogot, Y., Caspi, R., Dunaevsky, S., and Lemner, S. Teaching by examples: Implications for the process of category acquisition. The Quarterly Journal of Experimental Psychology Section A, 50(3): 606, 1997.
Chen, M., Roberts, N., Bhatia, K., Wang, J., Zhang, C., Sala, F., and Ré, C. Skill-it! a data-driven skills framework for understanding and training language models. Advances in Neural Information P Processing Systems, 36, 2024b.
Azar, M. G., Guo, Z. D., Piot, B., Munos, R., Rowland, M., Valko, M., and Calandriello, D. A general theoretical paradigm to understand learning from human preferences.
Chen, Z., Feng. S., Zhang, Z., Xiao, X., Gao, X., and Zhao, P. Sdformer: Similarity driven discrete transformer for time series generation. In The Thirty-eighth Annual

---

Principled Data Selection for Alignment
Conference on Neural Information Processing Systems, 2024c.
Gao, Y., Alon, D., and Metzler, D. Impact of preference noise the alignment performance of generative lan- guage models. In First Conference Language Model- ing, 2024. URL WtLLhhtss///peeeeee http://ppeeeeiiieeeeeeeeeee net for um? d=MMAaCsCTC.....
Chowdhury, S. R., Kini, A., and Natarajan, N. Provably robust dpo: Aligning language models with noisy feed- back. In Forty-first International Conference on Machine Learning, 2024.
Ghosal, G., Hashimoto, T., and Raghunathan, A. Under- standing finetuning for factual knowledge extraction. In Proceedings of the 4 International Conference on Ma- chine Learning, pp. 15540-15558, 2024.
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information pro- cessing systems, 30, 2017.
Hong, J., Lee, N., and Thorne, ORPO: Monolithic pref- erence optimization without reference model. ArXiv, abs/2403.07691, 2024.
Cobbe, K. Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L.,, Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers solve math word problems. arXiv preprint (tt::2110.14166 2021.
Hu, E. J., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al. Lora: Low-rank adaptation of large language models. In ICLR, 2022.
Ji12, S., Sinulingga, H. R., and Kwon, D. Robust guidance for unsupervised data selection: Capturing perplexing named entities for domain-specifer machine translation.
Croitoru, F.A.,, Hondru, V., lonescu, R. T., and Shah, M. Re- verse stable diffusion: What prompt was used to generate this image? arXiv preprint arXiv:2308.01 2023.
LREC-COLING 2024, pp. 307, 2024.
Croitoru, F.A.,, Hondru, V., Ionescu, R. T., Sebe, N., and Shah, M. Curriculum direct preference optimization for diffusion and consistency models. arXiv preprint arXiv:2405. 13637, 2024.
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. I., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b, arXiv preprint arXiv:2310.06825, 2023.
Du, Q., Zong. C., and Zhang, J. Mods: Model-oriented data selection for instruction tuning. arXiv preprint arXiv:2311.1565: 2023.
Jiang, L., Meng, D., Mitamura, T., and Hauptmann, Easy samples first: Self-paced reranking for zero- example multimedia search. In Proceedings of the 22nd ACM international conference Multimedia, pp. 547- 556, 2014.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama herd of models. arXiv preprint arXIv:2407.21783, 2024.
Joulin, A. Fasttext. zip: Compressing text classification models. arXiv preprint 161ivv1612.036511 2016.
Dubois, Y., Liang, P., and Hashimoto, T. Length-controlled alpacaeval: A simple debiasing of automatic evaluators. In First Conference on Language Modeling, 2024. URL https: www.review.e.......... net wttt://ormmiiddcypp m2WBXO.
Kim, J. and Lee, J. Strategic data ordering: Enhancing large language model performance through curriculum learning. arXiv preprint urXiv:2405.074900 2024.
Lambert, N., Pyatkin, V., Morrison, J., Miranda, L. J. V.,
Lin, B. Y., Chandu, K., Dziri, N., Kumar, S., Zick, T., Choi, Y., Smith, N. A., and Hajishirzi, H. RewardBench: Evaluating reward models for language modeling. In Chiruzzo, L.,, Ritter, A., and Wang, L. (eds.), Findings of the Association for Computational Linguistics: NAACL 2025, pp. 1755-1797, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-89 19--898877--99--77 URL https: / htc//aalanntt aclaathologg .rgg f winniiiinnnnnaaaacc 96/-
Engstrom, L., Feldmann, A., and Madry, A. Dsdm: Model- aware dataset selection with datamodels. In Forty-first International Conference on Machine Learning. 2024. URL wwt.t:ttp/...........tt htkk.ttt///:/pereelie oppprevieeeeeeeeeeeeeeeeee net. forum? GCBHKKeH8s.
Ethayarajh, K., Choi, Y., and Swayamdipta, S. Understand- ing dataset difficulty with usS-le information. In International Conference on Machine Learning, pp. 5988- 6008. PMLR, 2022.
Laurençon, H., Saulnier, L., Wang, T., Akiki, C., Vil- lanova del Moral, A., Le Scao, T., Von Werra, I.., Mou, C., González Ponferrada, E., Nguyen, H., et al. The bigscience roots corpus: 1.6 tb composite multilin- gual dataset. Advances in Neural Information Processing
Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. Model alignment as prospect theoretic optimiza- tion. In Forty-first International Conference on Machine Learning, 2024.
Systems, 35:31809-31824 2022.
10

---

Principled Data Selection for Alignment
Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burehh C., and Carlini, N. Deduplicating train- ing data makes language models better. arXiv preprint arXiv:2107.06499 2021.
Nagatsuka, K., Broni-Bediako, C., and Atsumi, M. Length- based curriculum learning for efficient pre-training of language models. New Generation Computing, 41(1): 134, 2023.
Ouyang, L.,, Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 01127730-2774444 2022.
Li, M.. Zhang, Y., Li, Z., Chen, J., Chen, L., Cheng, N., Wang, J., Zhou, T., and Xiao, J. From quantity to quality: Boosting IIm performance with self relf-guided data selec- tion for instruction tuning. In Proceedings of the 2024 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies (Volume Long Papers), pp. 7595-7628, 2024a.
Park, R., Rafailov, R., Ermon, S., and Finn, C. Disentangling length from quality in direct preference optimization. In Findings of the Association for Computational Linguistics ACL 2024, pp. 4998-5017, 2024.
Li, T., Chiang, W.-L., Frick, E., Dunlap, L., Wu, T., Zhu, B., Gonzalez, J. E., and Stoica, I. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406.1 1939, 2024b.
Pattnaik, P., Maheshwary, R., Ogueji, K., Yadav, V., and Madhusudhan, S. T. Curry-dpo: Enhancing alignment using curriculum learning & ranked preferences. arXiv preprint arXiv:2403.072.30 2024.
Lin, Z., Gou, Z., Gong, Y., Liu, X., yelong shen, Xu, R., Lin, C., Yang, Y., Jiao, J., Duan, N., and Chen, W. Not The
Penedo, G. Malartic, Q., Hesslow, D., Cojocaru, R., Alobei- dli, H., Cappelli, A., Pannier, B., Almazrouei, E., and Launay, J. The refinedweb dataset for falcon Ilm: Outper- forming curated corpora with web data only. Advances in Neural Information Processing Systems, 36:79155- 79172, 2023.
all tokens are what you need for pretraining. Tiihty- Annual Conference on Neural Information Processing Systems, 2024. URL https: ptctp:://ppeee openrevi ew net fortt/form?aaa id= 3NM22MMMMMMA
Liu, L. High dimensional robust m-estimation: Arbitrary corruption and heavy tails, The University of Texas at Austin, 2021.
Platanios. E. A., Stretcu, O., Neubig, G., Poezos, B., and Mitchell, T. M. Competence-based curriculum learning for neural machine translation. arXiv preprint 2019.
Liu, W. Zeng, W., He, K., Jiang, Y., and He, J. What makes good data for alignment? a comprehensive study of auto- matic data selection in instruction wning. In The Twelfih International Conference on Learning Representations, 2024. URL https:/ opwnreviewwweeeee///eeeee net/ forum ? Pid-BTKAeIQIMM
Qin, Y., Yang, Y., Guo, P., Li, G.. Shao, H., Shi, Y., Xu, Z.. Gu, Y., Li, K., and Sun, X. Unleashing the power of data tsunami: A comprehensive survey on data assessment and selection for instruction tuning of language models. Transactions on Machine Learning Research, 2025.
Meng, Y., Xia, M., and Chen, D. Simpo: Simple preference optimization with reference- free reward. Advances in Neural Information Processing Systems. 37:124198- 124235, 2024.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Er- mon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Ad- vances in Neural Information Processing Systems, 36, 2024.
Mishra, S. and Sachdeva, B. S. Do we need to create big datasets to learn a task? Proceedings of SustaiNL.P: Workshop on Simple and Efficient Natural Language Pro- cessing, pp. 169-173, 2020.
Rampp. S., Milling, M., Triantafyllopoulos, A., and Schuller, B. W. Does the definition of difficulty matter? scoring functions and their role for curriculum learning. arXiv preprint arXiv:241 1.00973, 2024.
Mitchell, E. A note on dpo with noisy preferences and relationship to ipo, 2023, URL https: /ttes:///rrmmiiiii hell. adp.cpp...//oooooooooo/ooooo..oa pdf.
Razin, N., Malladi, S., Bhaskar, A., Chen, D., Arora, S., and Hanin, B. Unintentional unalignment: Likelihood displacement in direct preference optimization. Inter- national Conference on Learning Representations, 2025.
Mukherjee, S., Mitra, A., Jawahar, G., Agarwal, S., Palangi, H., and Awadallah, A. Orca: Progressive learning from complex explanation traces of gpt-4. arXiv preprint 02707 2023.
Sachan, M. and Xing, E. Easy questions first? a case study on curriculum learning for question answering. In Pro- ceedings of the 54th Annual Meeting of the Association

---

Principled Data Selection for Alignment
for Computational Linguistics (Volume 1: Long Papers), pp. 453-463, 2016.
via bootstrapping. In ICML 2024 Workshop on Models of Human Feedback for AI Alignment, 2024b. URL https:
/ / //ppppeeeeeww.ccc net
Shen, Y. and Sanghavi, Learning with bad training data via iterative trimmed loss minimization. In International conference on machine learning. pp. 5739-5748 PMLR, 2019.
Wang, H., Xiong, W., Xie, T., Zhao, H., and Zhang. T. Inter- pretable preferences via multi-objective reward modeling and mixture-of experrss.. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 10582- 10592, 2024c.
Spitkovsky, I., Alshawi, H., and Jurafsky, D. From baby steps to leapfrog: How "less is more" in unsupervised de- pendency parsing. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association.fr Computational Linguistics, pp. 751-759, 2010.
Wang, Y., Ivison, H.,, Dasigi, P., Hessel, J., Khot, T., Chandu, K., Wadden, D., MacMillan, K., Smith, N. A., Beltagy, I., et al. How far can camels go? exploring the state of instruction tuning on open resources. Advances in Neural Information Processing Systems, 36:74764-747885 2023.
Swayamdipta, S., Schwartz, R., Lourie, N., Wang, Y., Ha- jishirzi, H., Smith, N. A., and Choi, Y. Dataset cartog- raphy: Mapping and diagnosing datasets with training dynamics. arXiv preprint (rXiv:2009.10795 2020.
Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., et al. Ethical and social risks of harm from language models. arXiv preprint 1:Xiv:2112.443:0 2021.
Tang, L., Ranjan, N., Pangarkar, O., Liang, X., Wang, Z., An, L.. Rao, B., Jin, L., Wang, H., Cheng, Z., Sun, S., Mu, C., Miller, V., Ma, X., Peng, Y., Liu, Z., and Xing, E. Txt360: A top-quality I1m pre-training dataset requires the perfect blend, 2024.
Wenzek, G., Lachaux, M.-A., Conneau, A., Chaudhary, V., Guzmán, F., Joulin, A., and Grave, E. Cenet: Extracting high quality monolingual datasets from web crawl data. arXiv preprint arXiv: 19::::91110.0000 1.00359, 2019.
Tay, Y., Wang, S., Luu, A. T., Fu, J., Phan, M. C., Yuan, X., Rao, J., Hui, S. C., and Zhang. A. Simple and effec- tive curriculum pointer-generane comprehension over long narratives. In Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics, pp. 4922-4931 2019.
Wettig, A., Gupta, A., Malik, S., and Chen, D. QuRating: Selecting high-quality data for training language mod- els. In International Conference on Machine Learning (ICML), 2024.
networks for reading
Team, G., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L.., Mesnard, T., Shahri- ari, B., Ramé, A., et al. Gemma 2: Improving open language models at practical size. arXiv preprint arXiv:2408.0011 00118, 10001011100001 2024.
Wu, B., Meng, F, and Chen, L. Curriculum learn- ing with quality-driven data selection. arXiv preprint arxiv:2407.00102. 2024.
Wu, S., Lu, K., Xu, B., Lin, J., Su, Q., and Zhou, C. Self- evolved diverse data sampling for efficient instruction tuning. arXiv preprint arXiv:2311.08182. 2023.
Tudor lonescu, R., Alexe, B., Leordeanu, M., Popescu, M., Papadopoulos, D. P., and Ferrari, V. How hard can it be? estimating the difficulty of visual search in an image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2157-2166, 2016.
Wu, X., Dyer, E., and Neyshabur, B. When do curricula work? In ICIRR, 2021.
Xia, T., Yu, B., Dang, K., Yang, A., Wu, Y., Tian, Y., Chang, Y., and Lin, J. Rethinking data selection at scale: Ran- dom selection is almost all you need. arXiv preprint 2rXiv:2410.09333 2024.
Tunstall, L.. Beeching. E., Lambert, N., Rajani, N., Ra- sul, K., Belkada, Y., Huang, S., von Werra, L., Fourrier, C., Habib, N., et al. Zephyr: Direct distillation of Im alignment. arXiv preprint arXiv:2310. 16944, 2023.
Xu, H., Sharaf, A., Chen, Y., Tan, W., Shen, L., Van Durme, B., Murray, K., and Kim, Y. J. Contrastive preference optimization: Pushing the boundaries of Ilm performance in machine translation. In International Conference on Machine Learning, pp.
Wang, B., Zheng, R., Chen, L., Liu, Y., Dou, S., Huang, C.,, Shen, W., Jin, S., Zhou, E., Shi, C., et al. Secrets of rihf in large language models part ii: Reward modeling. arXiv preprint arXiv:2401.0m 06080, 2024a.
PMLR, 2024.
Wang, H., Ma, G., Meng, Z., Qin, Z., Shen, L.,, Zhang, Z., Wu, B., Liu, L., Bian, Y., Xu, T., Wang, X., and Zhao, P. Step-on-fee tuning: Scaling self-alignment of L.LMS
Xu, Y., Yao, Y., Huang, Y., Qi, M., Wang, M., Gu, B., and Sundaresan, N. Rethinking the instruction quality: Lift is what you need. CoRR, 2023.
12

---

Principled Data Selection for Alignment
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu,
B., Li, C., Liu, D., Huang, F., Wei, H., al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, 2024.
Yu, Z.., Das, S., and Xiong, C. Mates: Model-aware data selection for efficient pretraining with data influence mod- els. Advances in Neural Information Processing Systems, 37:108735-10873 2024.
Yuan, H., Yuan, 7., Tan, C.,, Wang, W., Huang, S., and Huang, Rrhf: Rank responses to align language models with human feedback. Advances in Neural Information Processing Systems, 36: 10995-19950, 2023.
Zaremba, and Sutskever, Learning to execute. arXiv preprint arXiv:1410 4615, 2014.
Zhao, W. X., Zhou, K., L., Tang, T., Wang, X., Hou, Y.,
Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. survey of large language models. arXiv preprint arXiv:2303. 18223, 2023a.
Zhao, Y., Joshi, R., Liu, T., Khalman, M., Saleh, M., and Liu, J. SLiC-HF: Sequence likelihood calibration with human feedback. ArXiv, 12 2023b.
Zheng, L.,, Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging alm-as-a-judge with mt-bench and chatbot arena. Ad- vances in Neural Information Processing Systems, 36: 46595 46623, 2023.
Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., et al. Lima: Less is more for alignment, Advances in Neural Information Processing Systems, 36, 2023.
Zhou, W. Agrawal, R., Zhang, S., Indurthi, S. R., Zhao, S., Song, K., Xu, S., and Zhu, C. Wpo: Enhancing rlhf with weighted preference optimization. Proceedings of the 2024 Conference on Empirical Methods Natural Language Processing, pp. 8328-8340, 2024.
Zhu, B., Jordan, M., and Jiao, J. Iterative data smoothing: Mitigating reward overfitting and overoptimization in rlhf. In International Conference on Machine Learning, pp. PMLR, 2024a.
Zhu, W. Wu, O., Su, F., and Deng, Y. Exploring the learning difficulty of data: Theory and measure. ACM Transac- tions on Knowledge Discovery from Data, 18(4):1-37, 2024b.
13

---

Principled Data Selection for Alignment
A. Pseudocode for the Instantiated Algorithm: Selective DPO
Algorithm Selective DPO
Input:
TTFT: An SFT model that serves the starting point for preference alignment. D: A dataset consisting of preference examples.
RandomSumpler. A utility for sampling elements randomly without replacement.
SequentialSampler: A utility for sampling elements sequentially.
Cpo: DPO loss function with the form: CDro( Ywu 91) = log o log Turlu =arlu(
molye 2) log s0(in
Step 1: Train six reference alignment models: T0 Tou T011 T0a2 T021 T022
for t 0, 2, do
Randomly split the dataset D into two subsets, D1 and D2- Initialize Tet +- and <-
while RandomSampler has not finished do Sample a mini-batci of examples from D1 using RandomSampler.
Update no by minimizing the DPO loss function: wo f- arg min B( [CDpo Uw yr] end while
Save the model: 70 <- no-
Reinitialize:
Th TTFT.
while RandomSampler has not finished do
Sample a mini-batch of examples from D2 using RandomSampler.
Update To by minimizing the DPO loss function: To f- arg minz, B B(e.ee [DPO Uw mr3
end while
Save the model: 70.2 <- To. Reinitialize: no MSFT-
end for
Step 2: Rank examples by their validation loss. for each example (xr, Yw, y1) in dataset do Compute validation loss using the three held-out reference alignment models:
VL Mwr vt) E Tega Fe
1822 x (I, Uw mi
end for
Select the 50% examples with the lowest validation losses form D selected
# Step 3: Conduct alignment on the selected data Dselected while SequentialSampler has not finished do Sample a mini-batch of examples from Deeeectd using SequentialSampler.
Update ne by minimizing the DPO loss function: Tg +- arg mill,g Bi 31) Datisua [ID yoos 91) end while
Output:
No: The aligned model obtained by Selective DPO.
14

---

Principled Data Selection for Alignment
B. Related Work
B.1. Data Selection for Pre-Treee Training
Selecting training corpus brings significant performance gains in the pre-training stage (Wenzek et al., 2019; Brown et al., 2020; Zhao et al., 2023a: Penedo et al., 2023; Tang et al., 2024). Existing approaches can be broadly categorized into two categories: Sample-level selection focuses on filtering out undesired content such non-target languages, duplicated data, toxic materials, and low-quality information (Albalak al., 2024). This is often achieved through model-based filters (Joulin, 2016; Engstrom et al., 2024; Wettig et al., 2024) or heuristic filters (Wenzek et al., 2019; Lee al., 2021; Laurençon et al., 2022), each applying specialized filters for specific objectives. Token-level selection, an emerging strategy, down-weights low-quality tokens to enhance data quality (Lin et al., 2024), complementing sample-level filtering.
B.2. Data Selection for Supervised Fine-Tuning
Recent study suggests that SFT changes only the format of generation (Zhou et al., 2023). In light of this, various methods are proposed for finding the most informative subset for SFT, mainly following three principles: data quality, diversity, and importance (Qin et al., 2025). The mcasurement of data quality can be manual indicators such as the linguistic DQI (Mishra & Sachdeva, 2020), human scores (Zhou et al., 2023). Model-based quality measurement includes predictions from ChatGPT (Chen et al., 2024a), reward models (Cao et al., 2023), small reference models (Ankner et al., 2024) and the LLM itself (Li al., 2024a). Measurements of data diversity are mainly manually defined, such as the source diversity (Mukherjee et al., 2023; Wang et al., 2023) and distance in the embedding space (Wu et al., 2023; Xu et al., 2023; Du et al., 2023; Chen et al., 2024c; Liu et al., 2024). Data importance, which evaluates an example's 's contribution to a specific task, measured using performance scores (Engstrom et al., 2024), data influence models (Yu et al., 2024), or relevance to desired skills (Chen et al., 2024b).
B.3. Scoring the Example Difficulty
Scoring data difficulty is central to curriculum learning, which prioritizes training on simpler examples before progressing to more complex ones (Bengio et al., 2009). Heuristic scoring functions mirror human priors of difficulty understanding, such as sentence length (Spitkovsky et al., 2010; Tay et al., 2019; Nagatsuka et al., 2023), word rarity (Chang et al., 202 I), and linguistic perplexity (Campos, 2021). In contrast, principled scoring functions leverage model behavior to indicate example difficulty, including reward margins from third-party reward models (Croitoru et al., 2024), model perplexity on responses (Wu et al., 2024), attention patterns (Ghosal et al., 2024) or attention scores from transformer models (Kim & Lee, 2024). In addition, we refer readers interested in training dynamics-the motivation behind our work-to Shen & Sanghavi (2019), Liu (2021), and Swayamdipta et al. (2020) for further insights into learning dynamics and sample difficulty. In this work, we employ two principled scoring measures, demonstrating their robustness and consistency in ranking examples. This allows us to analyze difficult examples objectively, avoiding ambiguities inherent in heuristic definitions.
B.4. Curriculum Learning for Alignment
Curriculum learning (CL. mimics human cognition by structuring learning from simpler to more complex concepts (Avrahami et al., 1997; Bengio et al., 2009). However, CL remains a highly debated technique. While some studies show that it accelerates convergence, enhances generalization, and/or improves robustness in models like convolutional neural networks (Jiang et al., 2014; Tudor Ionescu et al., 2016), recurrent neural networks (Zaremba & Sutskever, 2014; Sachan Xing. 2016), transformers (Platanios et al., 2019), and diffusion models (Croitoru al., 2023). other research finds little no benefit (Platanios et al., 2019; Campos, 2021; Wu et al., 2021). In preference alignment for LLMs (Rafailov et al., 2024; Wang et al., 2024b), the results are similarly mixed. Kim & Lee (2024) explored CL for preference alignment and concluded that sorting examples according to prompt length and attention score offered no clear benefits. On the other hand, Pattnaik et al. (2024) reported positive results, albeit with other tricks such as multiple candidate pairs data and iterative reference model. Our study suggests that CL, when paired with robust difficulty scoring, can positively impact LLM alignment by aligning data difficulty with model capacity.
15

---

Principled Data Selection for Alignment
C. Experiment Details
C.1. Computational Environment
All training experiments in this paper were conducted on compute nodes equipped with 8 x H100 GPUs. To facilitate reproduction with limited computational resources, we also provide key benchmarking results for selected models trained using 4 x A100 40G GPUs with LoRA. Reproducing SelectiveDPO on 7B models takes about 8 GPU hours (H100).
C.2. SFT Hyper-Paraametess
In this work, we limited our alignment experiments to SFT models, which expected better demonstrate the effects of different preference alignment procedures. We prepared these SFT models using the the UltraChat-200k dataset. We try our best the SFT models from community facilitate the reproduction. However, there checkpoints for some pre-trained models (e.g., Qwen-2.5 models). We in this part list the hyper-parameters for training these community-releasee SFT models as well as the SFT models trained by ourselves in Table 2.
available SFT
Table 2: Training recipes for SFT models used in our experiments.
SFT Model Name
Base Model Name
Batch Size Learning Rate Epoch Optimizer LoRA?
Qwen-2.5-3B-SFT
Qwen/Qwen2.5-3B
128
2e-5
Adam
No
Qwen-2.5-7B-SFFT
Qwen/Qwen2.5444
le-5
Adam
No
Qwen-2.5-14B-FFT
Qwen/QWee2..5114B
128
5c-6
Adam
No
Mistral-7B-SFT
mistralai/Mistral-7BB
128
2e-5
Adam
No
Llama-3-8B-SPT Prinector-ttyLLLam---BBe----
meta-llaama/Met--llamaa
128
20-5
Adam
No
Gemma-2-9B-SFT (ualiiho//zphhy--gmmms222---
googla/gemma-2-9h
128
3c-6
Adam
No
C.3. Key Hyper-Parameters for Alignment
Figure We conducted a series of alignment experiments with LoRA on two datasets for generating Figure Key hyperparameters used the Argilla-dpo-mix-2 experiments are listed in Table where we report the sweep range and the selected best learning rate for DPO in bold font. These parameters are then directly applied to other settings (sorted and selected by VL) for generating Figure 3. The key parameters used for the UltraFeedback-binariee dataset are list in Table 4.
Table 3: Key hyperparameters used for aligning models on the argilla-7k dataset: Figure 3, top.
Model for Alignment Learning Rate
Batch Size
Epoch Optimizer
LoRA?
Mistral-7B-SFT
2e-5, 3e-5, 5e-5, le-4, 2c-4 64
0.01
paged aaged. btbb Yes
Qwen-2.5-7B-SFT
2e-5, 3e-5, 5e-5, le-4, 2e-4 64
0.01
paged_adamw, adamww 32bit Yes
Llama-3-8B-SFT
5e-5, le-4, 2e-4, 3e-4, 5e-4
0.01
paged 33bbi Yes
Gemma-2-9B-SFT
le-5, 2e-5, 3e-5, 5e-5, le-4
0.01
paged_adame adamw 33bit Yes
Table 4: Key hyperparameters used for aligning models on the (itrafeebback-biannize dataset: Figure 3, bottom.
Model for Alignment Learning Rate
Batch Size
Epoch Optimizer
LoRA?
Mistral-7B-SFF
le-6, 3e-6, 5e-6, 8e-6, 10e-6 64 le-6, 3e-6, 5e-6, 8e-6, 10e-6 64
0.01 1
had_dadamaa32000 Yes
Qwen-2.5-7B-SFT
0.01 1
paged adamw.33 -3bit Yes
Llama-3-8B-SFT
le-6, 3e-6, 5e-6, 8e-6, 10e-6 64
0.01 1
@agedaddamw_322 Yes
Gemma-2-9B-SFT
1e-6, 3e-6, 5e-6, 8e-6, 10e-6 64
0.01 1
2eceddddaw33322 Yes
16

---

Principled Data Selection for Alignment
Table Comparison results of this table mainly borrowed from the SimPO paper (Meng al., 2024). All results obtained by full parameter fine-tuning (FPFT) except for the row labeled with LoRA. We added the results of our Selective DPO pipeline using the configurations detailed in the following table. The inclusion of LoRA results is to facilitate the reproduction for practices with limited resources.
Table 5: Key hyperparameters used for aligning models on the ultrafeedback-bianrii dataset: Figure 3, bottom.
Experiment Name
Learning Rate
Batch Size
Epoch Optimizer
LoRA?
SelectiveDPO & Mistral-7B-SFT & LoRA
8c-6
64
0.01
paged _32bit Yes
SelectiveDPO & Mistral-7B-SFT
2e-7,5c- le-6, 2c-6, 3c-6
128
0.01
paged adamw 32bit No
Sc & Llama-3-8B-SFT & LoRA
10c-6
64
0.01
paged_adamw _32bit Yes
SelectiveDPO Llama-3-8B-SFT
2e-7, 5e-7, 1e-6, 2e-6, 3e-6
128
0.01
paged adamw .32bit
WPO & Llama-3-8B-SFT
5c-7, le-6, 2c-6
128
0.01
paged _32bit No
Figure 7 Comparison results of this figure are from runs with full parameter fine-tuning, We rerun two state-of-the-art alignment algorithms, SimPO (Meng et al., 2024) and WPO (Zhou et al., 2024) with hyperparameter sweeping on the learning rate. Other hyperparameter configurations follow the suggestion from their papers. Specifically:
Table 6: Key hyperparameters used for generating comparison in Figure
Experiment Name
Learning Rate Batch Size Epoch Optimizer
Other Hyper-Parameters LoRA?
WPO & Qwen-7B-SFF
5e-7, le-6, 2e-6, 128
paged_adamn .3bitt B 0.01
No
WPO & Gemma-9B-SH
2e-7, 5e-7, le-6 128
|
paged adamw 32hit 0.01
No
SimPO Qwen-7B-SFT
6c-5, 8e-6, le-5 128
paged adamw _3biit 3 0.8
No
SimPO & Gemma-9B-SFT
5e-7, lc-6, 2e-6 128
I
paged aeeeda _32bit 2, 0.8
No
SelectiveDPO Qwen-7B-SHT
5e-7, 8e-7, 1e-6 128
paged adamw 32bit 0.01
No
SelectiveDPO Gemma-9B-SFT 2e-7, 3e-7, 5e-7 128
1
paged adamw .3bit = 0.01
No
C.4. LoRA Configuration for Alignment
We conduct all our analytics experiments using LoRA. Its detailed configurations are described in Table 7.
Table 7: LoRA configuration for all analytics experiments.
Parameter
Value
load-in_4bl
false
lora_r
16
lora alpha
16
lora_dropout
0.05
Iora target modules q proj.k proj.v proj.o proj.gate proj.up proj proj
C.5. Decoding Configuration
AlpacaEval 2. For this benchmark, we employ sampling-bass decoding strategies, configuring the temperature as follows: 0.7 for Mistral models, 0.9 for Llama-3 models, 0.5 for Gemma-2 models, and 0.7 for Qwen-2.5 models. These configurations are aligned with standard practices in the community.
Arena-Hard. For this benchmark, we utilize default greedy decoding across all settings, as outlined in Meng et al. (2024).
MT-Bench. We adapt the official decoding configuration, which varies in sampling temperatures for different models.
17

---

S--eleci---c- Principled Data Selection for Alignment
D. Downstream Task Evaluation
To examine how the proposed selective preference optimization pipeline affects downstream task performance, we evaluate the instantiated algorithm, Selective DPO, alongside other baseline algorithms on various tasks listed in the Hugg Open Leaderboard (Beeching al., 2023). Results, following established evaluation protocols, are presented in Table 8.
Table 8: Downstream task evaluation results. The dataset is UltraFFeddacck-bnariiii
MMLU(5) Winograd(5)
GSM8K(5)
HellaSwag(10)
ARC(25)
TruthfulQA(0))
Average
Mistral-7B-Base
Base
62.46
78.93
38.29
83.38
61.6
42.64
61.22
SFT
59.77
77.58
40.71
82.28
58.19
43.05
60.26
DPO
57.38
77.35
30.4
83,58
61.18
53.11
60.50
SimPO
58.43
77.35
32.3
83.54
61.95
50.82
60.73
WPO
59.54
78.69
32.07
85,23
64.08
51.04
61.78
SelectiveDPO
59.34
76.16
14.48
83.25
65.27
51.95
58.41
SelectiveDPO((05))
59.54
76.87
28.58
84.25
65.96
57.21
62.07
Qwen-2.5-7B-Base
Base
74.16
76.72
82.18
80.03
63.23
56.38
72.12
SFT
73.86
75.77
81.43
80.71
62.71
55.67
71.69
DPO
74.06
75.61
82.79
81.73
65.70
60.92
73.47
SimPO
74.33
77.11
85.22
82.48
68.09
65.51
75.45
WPO
74.29
75.85
83.55
68.52
65.09
75.08
SelectiveDPO
74.05
75.85
80.44
82.82
67.32
63.80
74.04
Llama-3-8B-Base
Base
65.14
76.64
48.45
81.88
58.87
43.93
62.49
SFT
63.79
76.64
50.57
81.40
60.84
45.33
63.10
DPO
63.47
76.95
54.81
83.71
64.51
53.45
66.15
SimPO
63.18
77.58
47.76
82.93
65.44
59.44
66.06
WPO
63.46
76.72
44.58
84.14
65.27
53.84
64.67
SelectiveDPO
63.99
76.48
48.75
83.51
64.93
51.34
64.83
Gemma-2-9B-Baxy
Base
70.29
80.03
40.41
82.66
67.83
45.56
64.46
SFT
70.82
78.77
41.93
83.53
68.77
48.04
65.31
DPO
71.17
80.11
44.43
85.42
71.33
56.96
68.24
SimPO
72.
80.43
42.53
86.06
73.12
65.34
69.94
WPO
70.88
79.40
43.14
85.64
70.99
53.44
67.25
SelectiveDPO 70.88
79.56
43.67
85.30
70.82
54.67
67.48
Overall, Selective DPO performs comparably other alignment algorithms, such DPO and SimPO. However, observe notable performance drop in the Mistral-7B model when evaluated using the GSM8K (Cobbe al., 2021) protocol. Two primary factors contribute this performance decrease: Exclusion of difficult examples. GSM8K predominantly evaluates mathematical skills, which often correspond to difficult examples (as detailed in Appendix F). Since Selective DPO excludes such difficult examples to get better aligned with human preferences, the model's mathematical performance diminishes. Formatting requirements. GSM8K requires numerical answers in specific format: ### <THE ANSWER>. We find DPO model often generates correct answers but presents them in a human dialogue style,
that the M breaking the required format and reducing evaluation scores.
We propose three potential solutions. First, suggested by SimPO (Meng al., 2024), incorporating auxiliary SFT loss to regularize model behavior could help regularize the model's behavior, ensuring compatibility with downstream tasks. Second, using larger models with greater capacity mitigates this issue. For instance, demonstrates better performance and unaffected by this issue. Finally, including more examples that cover mathematical questions could prevent the model from forgetting its mathematical capabilities while aligned with human preferences and dialogue styles. For example Selective (P0(),%), which incorporates 10% more difficult data, alleviates this issue.
18

---

Principled Data Selection for Alignment
E. What Defines Difficult Examples: Insights from Feature Analysis
E.1. Can Length and Reward Margin Prediet Example Difficulty?
We include alternative measures that could potentially indicate example difficulty and evaluate their behavior across varying levels of difficulty.
Response length. Response length may implicitly signal the complexity of generated answers, longer responses often carry more information, potentially making them more challenging for the model. Two measures are defined: (1) Chosen Length: The length of the chosen answer, lyn(yw); (2) Chosen Length Rejected Length: The difference in lengths between the chosen and rejected answers: len(yw) (y)
Reward margin by reward models. Reward models, such as ArmoRM (Wang et al., 2024c), provide score margins that can indicate response difficulty. A large positive margin suggests an easy example, while a large negative margin may signal noisy or mislabeled data. Two measures are defined: (1) Chosen Score: The reward score assigned to the chosen answer, rm(x, ). and (2) Chosen Score Rejected Score: The difference in scores between the chosen and rejected answers: rm(x, tha) rm(x, yt).
Reward margin by GPT-4. GPT-4 can also as an evaluator, assigning scores responses. Similar measures are defined: (1) Chosen Rating: The rating assigned to the chosen answer, GPT-4((r, Uw) and (2) Chosen Rating Rejected Rating: The difference in ratings between the chosen and rejected answers: GPT-4(x, yw) GPT-4(4; y1).
6000
5000
Chosen Length
4000
1.25
3000
0.2
1.00
Valdation Loss
0.0
Easy
Medium
Difficult
Easy
Medium
Difficult:
Easy
Medium
Difficult
4000
Chosan Score Rejacted Score
1.0
0.50
Chozen Length Rejected
0.15
Rejacted
2000
0.10
0.00
-0.05
2-
44000
-6000
Easy
Medium Difficulty
Difficutt
Easy
Medium
Diffiuut
Easy
Medium
Difficul:
Easy
Medium
Difficult
Difficulty Level
Difficulty
Difficulty
Figure 10: Comparison of response length and reward margin measures with validation loss across three difficulty levels: Easy, Medium, and Difficult. The dataset examples are partitioned into these levels based on increasing validation loss. While validation loss increases consistently with difficulty, alternative measures such as response length and reward margin (from reward models and GPT-4) exhibit significant variation across these levels, indicating their limited effectiveness as proxies for difficulty.
Comparison. The distributions of these measures are shown in Figure 10. The data are partitioned into three levels -asy, Medium, and Difficult-basee increasing validation loss. Validation loss serves the ground truth for difficulty due its strong correlation with the learned step measure. Difficult examples tend to have longer responses and larger negative reward margins. However, these trends not statistically significant, as evidenced by substantial overlaps in the distributions across difficulty levels. This suggests that while these measures provide some insight, they may not serve as robust standalone indicators of example difficulty.
19

---

Principled Data Selection for Alignment
E.2. Are Easy Examples for Small Models Still Easy for Larger Models?
2.0%
48.1%
23.2%
93.3%
1448 Model
38 Model
60
40
Easy
98.0%
51.9%
76.8%
6.7%
20
Easy
Difficult
Easy
Difficult
38 Model
Gracle
Medel
Oracie
80
Difficult
0.6%
148 Model
7B Model
60
8
40
Easy
99.4%
36.7%
87.0%
2.4%
20
Easy
Difficult
Easy
Difficult
Model
Gracle
Modal
Oracio
80
Difficult
4.1%
72.1%
90.9%
7B Model
38 Model
60
8
400
Easy- 95.9%
27.9%
85.8%
9.1%
20
Easy
Difficult
Easy
Difficult
Gracle
Modeel the Dracle
Figure 11: Easy examples identified by smaller models are likely also recognized as easy by larger models, and difficult examples identified by larger models are likely challenging for smaller models as well. Each heatmap compares a model's classifications(y-nns (-aiiis against those of an oracle (another model, set on x-axis). Easy examples are defined as the first 63.7%, 71. 11.2%, and 81.3 11,3% of examples with the lowest validation loss for the 3B, 7B, and 14B models, respectively. Difficult examples comprise the remaining examples, with these thresholds drawn from the mean sweet spot in Figure 5. The dataset Argilla-dpo-mix-7k.
20

---

Principled Data Selection for Alignment
F. What Defines Difficult Examples: Insights from Case Studies
In this section, we examine examples with varying levels of difficulty, characterized by their validation losses. We select three representative examples around validation loss quantile: 0.0 (easiest), 0.6 (medium), and 1.0 (most difficult) from the Argilla-dpo-mix-77 7k dataset. We highlight examples near the 0.6 quantile as this range marks critical transition point where the model's performance starts to decline. Our key observations are as follows:
Easy examples are well-labeled and involve straightforward concepts (Table 9). Easy examples show that preferred responses are concise, clear, and of significantly higher quality than their rejected counterparts. which are often overly verbose and filled with excessive emojis. Unlike more difficult examples, these easy examples lack complex mathematical caleulations or the use of languages other than English.
Table 9: Three examples with the lowest validation loss (Model: Mistral-7B-SFT & Dataset: Argilla-dpo-7k).
Prompt
Chosen
Rejected
How can you determine the pH of rainwater?
yes.
"excitedly* Hi there! *bouney bouncy I definitely help you with that! "giggle* So, you wanna know how to determine the PH of rainwater, huh? "curious* Well, pH is like a special number that tells us how yucky yummy something is. "giggle* And rainwater can be little bit yucky sometimes, so we need check its pH make sure it's s's safe to drink. *concerned*
Ooh ooh!
Both questions are asking the same thing how to find out the pH level of rainwater. They have simply been worded differently. Paraphrasing means to say something in different way, and that's exactly what these questions do, so they paraphrases of cach other
How can you determine the pH level of rainwater?
Choose your answer. Are these two questions paraphrases of each other?
OPTIONS:
1). no.
To check the pH of rainwater, we can use special tool called a pH meter
2). yes..
Please add spaces between words: Fixedroofrallngsserrssen-- tmeeaandincreasnnlyymmmoote productarca, andoneinwwhichthecomm
Fixed roof railings represent a new and increasingly important product area, and one in which the company's 's customers include Opel, BMW, Volkswagen, Nissan and Ford
Ooh ah! *giggle* Hi there! *bounce bounce* Do you want to know about something cool? *twirl twirl* It's called "Fixed Roof Railings"! *excited squeal They 're like special rails that go on cars! *bounce bounce" So, you know how cars have roofs, right? *look around* Like this one! "point" And sometimes, the roof is made of metal something hard, right? "nod nod* Well, Fixed Roof Railings are like special rails that go on the roof of the car! *giggle* Theyfe fixed, so they don't move around! *bounce bounce#
pany' httpscustoomersiccuueeeppplp
BMW,
Based on the premise "Peace, amigo. Don Cazarš raised finger silenced his man.", can conclude the hypothesis "Peace, friend, said Don Cazar, silencing the man with a finger." is true (see options)?
Yes, we can conclude the hypothesis "Peace friend, said Don Cazar, silencing the man with a finger." true.
Ooh ooh! *giggle* * Hi there, little buddy! *smile So, you wanna know the hypothesis "Peace, friend, said Don Cazar, silencing the man with finger" is true? *tongue out* Ooh, thats good one! *nod*
Well, let me think... *thinking face Ooh ooh, I know! "excited* The sentence "Peace, amigo" is like magic word that makes the man stop talking, like when you say "Shhh" to baby. "giggle* So, if Don Cazar said "Peace, amigo," means he wants the man to be quiet, and thats why the man stopped talking! *smile*
Available choices:
-yes;
-it is not possible to tell;
no;
So, the hypothesis is... *drumroll* ...true! "confetti* Yay! "high five* Youfe so smart for asking such great question! *hug* Now, do you want to ask another question? "wink
21

---

Principled Data Selection for Alignment
Examples near the model's capacity have correct labels but require complex skills (Table 10). The preferred examples this level of validation loss consistently have correct labels. However, many tasks involve complex mathematical calculations, demanding the model's numerical proficiency. Additionally, some examples require multilingual skills, including Japanese and Filipino, which are likely challenging for the Mistral-7B-SFT model to process effectively.
Table 10: Three examples near the models' capacity (Model: Mistral-B--SFT & Dataset: Argilla-dpo-7k).
Prompt
Chosen
Rejected
6994775544388976633902222
62944725 43866776662222 438869799503737
The sum of the two numbers is: 629,447,225 250 + 44.516.431.106.432
cquations/math only, no commentary.
"A team of cheerleaders cheering sporting event."
1. The premise states that the cheerleaders
No, cannot conclude that the hypothesis "Cheerlcaders laying down in the locker room" true based on the premise team cheerleaders cheering at sporting event." Hereś why:
cheering at sporting event.
Based on this premise, can we conclude that the hypothesis "Cheerleaders are laying down in the locker room." is true?
2. This implies participating and performing at the event.
they are actively
3. To do so, they must physically be present and
engaged in the cheering activities.
1. The premise only tells that the cheerleaders are cheering sporting event, but it doesnf provide any information about their location or activity when they are not cheering.
Options:
4. Hence, the hypothesis that the cheerleaders
yes
are laying down the locker room cannot be true, since they are actively participating in the event.
it is not possible to tell
no Step by step answer:
2. The hypothesis "Cheerleaders are laying down in the locker room" not logical conclusion based on the premise. There is no evidence or reason to believe that the cheerleaders are laying down in the locker room, this activity not related to cheering sporting event.
Answer: no
3. Therefore. the correct answer is "it is not possible tell" whether the hypothesis is true false based on the premise
You will be given definition of first, then some input of task.
Ang mga piskal mahirap na posisyon ay kasalukuyang sinag-immbsstigahan ng Kagawaran Hustisya tungkol sa kanilang mga ginawa sa problemang iyon.
Ensayado muna ang miyembro ng paglagdaan isang bagong problema darating kanilang pamamahala.
Given sentence in the Japanese, provide an equivalent translation Filipino that retains the same meaning through the translation. In translation, keep numbers as it is.
La T
35 b.
Output:
22

---

Principled Data Selection for Alignment
The most difficult examples are likely mislabeled, exhibit distribution shifts, or involve complex concepts (Table 11). First, some preferred examples in this category are likely mislabeled, as illustrated by the first and second examples in the table. Second, there is noticeable distribution shift in this subset of the data. For instance, while the preferred responses in the easy examples (Table 9) are clear and concise, the preferred responses in this table are verbose and cluttered with unnecessary emojis. Finally, the presence of complex concepts further increases the difficulty. Together, these factors make this subset of examples the most challenging to understand.
Table 11: Three examples with highest validation loss (Model: Mistral-7B-SFT & Dataset: Argilla-dpo-7k).
Prompt
Chosen
Rejected
Please add punctuation to this: Stunning coastal world Ferry trip along the Inside Passage
Sure! Here's 's the punctuated version of your task: "Stunning coastal world! Ferry trip along the Inside Passage."
"Stunning Coastal World: Ferry Trip Along the Inside Passage."
Punctuation version:
Now, let me provide you with detailed and long answer to this task.
The Inside Passage is breathtakingly beautiful coastal route that stretches along the Pacific coast of North America, spanning from northern Washington state to southern Alaska. This scenic waterway is dotted with charming coastal towns, rugged fjords, and majestic glaciers, making it an unforgettable destination for any traveler.
How to apply for tickets for the fifa world cup? Apply for your passport. Encourage anyone who wants to attend the world cup with you to do so as well. You will need valid passport number in order to apply for tickets.
Oh my gosh! You wanna go to the FIFA World Cup? That sounds like so much fun! First, you need to get passport, "kay? That's like a special book that proves you're you and you'te 're allowed to go to other countries.
Yes.
Then, you can your passport number apply for tickets to the World Cup! But, oh no! You can't 't get tickets if you don't 't have a passport, so make sure you get one first!
Does this next sentence follow, given the preceding text?
You need to pay for world cup tickets
And, oh my gosh! You'll 'Il also need to pay for the tickets!
Select from: (a). Yes; (b). It's
So, get your passport and your money ready, and you'll be ready to go to the World Cup!
impossible to say; (c). No.
Does that make sense?
Here's brain teaser for you: Can you describe process in which machine can Iearn without being explicitly programmed? Think hard, because that's 's exactly what machine learning is all about. Give a try and let's see if you can crack this puzzle!
Machine learning a field of computer science that gives computers the ability to learn without being explicitly programmed. It is branch of artificial intelligence based on the idea that systems can learn from data, identify patterns and make decisions with minimal human intervention.
Machine learning can be achieved through the process of observation and unsupervised learning, which does not require explicit programming. In this process, a machine can observe its environment and learn from data patterns, without the need be explicitly programmed. Through this, it can pick up on patterns in data and uncover trends that can be used to make predictions and decisions.
There are many different types of machine learning algorithms, but they all work by finding patterns in data. Once machine learning algorithm has learned pattern, it can be used make predictions decisions, For example, a machine learning algorithm could be used to predict whether a customer is likely to churn, or to decide which products recommend to customen, powerful tool that can be used to solve a wide variety of problems. However, is important to note that machine learning is not a silver bullet. Machine learning algorithms can make mistakes, and they can be biased. It is important to carefully evaluate machine learning algorithms before using them in production. One way to describe... ...
Machine learning
23

---

Principled Data Selection for Alignment
G. Additional Comparison and Visualization
Rendom Ordering
Validation Loss
Perplexity (Descending)
Perplexity Gep (Accending)
Perplexity Chosen (Ascending)
7B-SFT (Argilla-7k)
0..en2.5-7B-50 (Argila-7k)
(Argilla-7k)
(Argilla-7))
0.25
0.20
0.25
0.29
0.10
0.30
0.05
0.10
0.05
0.10
40
00
80
100
20
80
80
100
20
40
00
100
20
40
80
100
Data Percent 196)
Data Percent (%)
Duta Parcent (54)
Data Porcent (%)
Rendom Ordering
Velidation Luss
Length of Chosen (Ascending))
Reward Mergin (Descending, GPT-4)
Label Flipping (Skywork)
WWY ool rate alot 022 0u-bb9
0.25
Mistral-78-SFT (Argilla-7k)
Qwen-22 5-78-SFT (Argila-77k
Lamaa3-88-SFT [Argilla774]
Gemma-2-9B-SF (Argilla-7k)
0.25
0.25
0.20
0.20
0.25
0.15
0.10
0.05
0.05
0.10
0.00
6-
20
40
60
80
100
0
20
40
60
80
100
20
40
60
80
100
20
40
60
100
Data Percent 1961
Data Percent
Data Percent 1963
Data Percent Cisa
Figure 12: Comparison of our difficulty metric validation loss against alternative sorting criteria: perplexity gap, completion length, and reward margin. Perplexity Gap is defined as the difference in perplexity between the chosen and rejected responses given the same prompt. Perplexity of Chosen refers to the perplexity of the chosen response alone. Reward Margin denotes the difference reward scores between the chosen and rejected responses. Label Flipping involves flipping the preference labels of samples identified as difficult and potentially mislabeled.
1.0
10
20
Gradient
30
0.25
40
Prompt (200 samples, sorted by the averaged loss)
Figure 13: Evolution of preference probabilities lities during 2-epoch training. We track the trajectory of p(y. > 31 (z) for 200 (e(w)())
held-out test examples for better intuition. The probability is defined as: p(y > yn (x) = a (3 log Tar(vul - B log ( (m) following the derivation of DPO paper (Appendix A.2). In general, the evolution of the validation loss (which is - log (( .z)) is quite stable and gradual. Only few "ambiguous instances" flip their preference probability (from greater than 0.5 to less than 0.5) during the 2-epoch training.
u ar)
Weak Strong Curriculum
WR (win rate aist got turbol
0.25
0.20
78 Curricutum
0.15
38 Curicclums
7B Random Ordering
328 Currculuu
Mistral Camiuulm
20
40
60
80
100
Data Percentage (%)
Figure 14: Weak-to-strong curriculum under-performs. Aligning 7B model with examples ordered by 3B reference models yields compromised results.A similar degradation is observed for the 32B model and other model types. Notably, using curriculum derived from the 32B model performs better, consistent with our observation in Appendix E.2.
24


Reviews:
[{"review_id": "SaEmmBIxGs", "reviewer": "Reviewer_TgH4", "summary": "This paper focuses on the alignment performance w.r.t. data difficulties. The central claim is that the difficult data points exceeds model capabilities, and therefore harm the alignment results.\n\nTo start with, it is crucial to define the difficulty measure, authors use \"learned step\" as a metric to quantify data difficulty, and find that this variable keeps to be similar across different runs on different data split and shuffle. However, this metric requires evaluation every gradient update, which is time consuming. Authors propose to use validation loss as a proxy metric and show its strong correlation with learned step.\n\nBy using proxy metric, data is ordered by its difficulty, which is given by trained reference models, and authors keep $\\tau$% top easiest data to sequentially train DPO models.\n\nResults  show that SelectDPO is superior to DPO and other variants on chat tasks. Abalation studies on reference models , $\\tau$% and weak-to-strong curriculum fultill the analysis of SelectDPO", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful and constructive feedback, particularly regarding the label flipping experiment and the perplexity gap baseline. These comments help clarify and reinforce our central contribution: that alignment performance is critically influenced by the mismatch between model capacity and example difficulty. Please check our new results at [this link](https://selective-dpo.pages.dev/)  (https://selective-dpo.pages.dev/) and the following response:\n\n**Q1) Label flipping experiment (Figure 4a)**\n\nFigure 4a tests whether label noise is the primary cause of performance degradation on difficult examples. Flipping all difficult samples did not improve performance, suggesting that label noise alone is unlikely to explain the difficulty. While partial noise (e.g., 30%) may exist, we do not claim the data is noise-free—only that noise is not the dominant factor.\n\nTo address your concern, we flipped only those examples identified as both difficult and potentially mislabeled by a reward model (`Skywork/Skywork-Reward-Gemma-2-27B-v0.2`, 1,414 examples in Qwen2.5). This targeted flipping (`Label Flipping (Skywork)`) also showed no consistent benefit across four models, reinforcing our conclusion. Notably, the original labels are from GPT-4.\n\n**Q2) Reporting variance**\n\nAll figures already include standard error bars across runs. In addition, we report standard error (over 3 runs) in the result tables for completeness. Here are the results on Alpaca Eval 2: \n\n| Mistral-7B-SFT | Length-Controlled Win Rate, | Win Rate |\n| --- | --- | --- |\n| SimPO | 21.5 | 20.8 |\n| WPO | 24.4 | 23.7 |\n| Selective DPO (LoRA) | $\\textbf{25.4} \\pm0.80$ | $\\textbf{27.4} \\pm1.26$ |\n| Selective DPO | $\\textbf{27.1} \\pm 0.63$ | $\\textbf{28.9} \\pm1.31$ |\n\n| Llama-3-8B-SFT | Length-Controlled Win Rate, | Win Rate |\n| --- | --- | --- |\n| SimPO | 22.0 | 20.3 |\n| WPO | 23.1 | 22.2 |\n| Selective DPO (LoRA) | $ 21.1 \\pm 0.73$ | $18.3 \\pm 1.14$ |\n| Selective DPO | $\\textbf{24.9} \\pm 0.77$ | $\\textbf{25.3} \\pm 1.36$ |\n\n**Q3) Curriculum transfer**\n\nFigure 9 compares a model’s own curriculum with that of a smaller model trained on the same (pre-training and SFT) data. Results show that a model benefits more from its own curriculum. To test cross-model transfer, we trained Qwen2.5-7B using curricula from Mistral-7B and Qwen2.5-32B. The results confirm our conclusion: **a model’s own curriculum is most effective**.\n\n**Q4) Comparison with perplexity gap.**\n\nWhile various data selection methods exist for SFT, such as perplexity-based filtering, they are not directly applicable to DPO-style preference data due to fundamental differences in format: SFT data consists of (prompt, completion) pairs, while DPO uses (prompt, preferred, rejected) triplets. This mismatch makes direct application of SFT techniques unsuitable for alignment. To address this gap, we implemented *perplexity of chosen,* *perplexity gap, etc.*\n\nTo assess their effectiveness without relying on arbitrary thresholding, we adopted a consistent evaluation protocol: (1) sort examples by each metric, (2) train DPO using the hyper-parameter set in Table 3 (page 15), and (3) evaluate whether performance drops after removing a portion of the data.\n\nNone of these alternatives yielded consistent improvements over our validation-loss-based selection strategy.\n\n**Q5) Length bias in the selected data.** \n\nAs shown in Figure 10 (page 18), selected (easier) examples tend to have slightly shorter responses and smaller length gaps (preferred minus rejected). This pattern arises because DPO loss often assigns higher loss to longer examples. Since our strategy filters out high-loss (difficult) samples, it indirectly favors examples with smaller length gaps. While this introduces a mild length bias, we agree that it warrants future investigation.\n\n**We sincerely appreciate your insightful suggestions.** These have helped clarify the scope and robustness of our findings. We will incorporate the new experiments and discussions into the revised manuscript. Please let us know if this response sufficiently addresses your concerns.", "claims_and_evidence": "### Main Claims: \n* Learned step is a proper metric for difficulty evaluation and shows consistency across runs. Supported by Figure 2.\n* Validation loss is a good proxy metric for difficulty evaluation. Supported by Figure 2: strong correlationwith learned step.\n* Data difficulty orderding (SelectDPO) outperforms random shuffling. Supported by Figure 3.\n* Difficult data are not all noisy data. Supported by Figure 4.\n* Larger model can benefit from more difficult problem. Supported by Figure 5.\n* SelectDPO outperforms other DPO variants (Table 1).\n\n### Weakness & Question\n* Label flipping (Figure 4) cannot fully support the claim on data error. It is possible that the \"difficult\" problem contains more mislabeled data, let's say 30%, and flipping causes the mislabeled data to be 70%. Therefore, the performance drop after flipping will be expected.", "methods_and_evaluation_criteria": "Methods and eval make sense.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experiment looks good overall.\n\nSome questions and weaknesses:\n* Considering the variance of the chat benchmarks, authors should report the variance in the result table for better understanding of the performance.\n* Is the reference model crucial to be the same as the policy model? Figure 9 shows 7B curriculum is better than 3B's one, what about other 7B models curriculum? Is the reference model type important here?\n* Is the validation loss suprior to other data selection strategies?E.g., perplexity gap.\n* The DPO validation loss may have length bias. Does the data selection strategy based on validation loss has a length bias too?", "relation_to_broader_scientific_literature": "This work aligns the line of work in data selection, such as perplexity-based selection.\n\nThis work distinct from other works from:\n(1) selection strategy\n(2) detailed ablation on different hypothesis, design choices and hyperparameters.", "essential_references_not_discussed": "The paper does not miss important paper to the best of my knowledge.", "other_strengths_and_weaknesses": "Strenths:\n\n* The paper is very clear\n* The experiment is overall solid.\n\nWeakness:\n* See above", "other_comments_or_suggestions": "## update after rebuttal\n\nI keep my score.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4c7IWfcv96", "reviewer": "Reviewer_La1n", "summary": "This paper investigates the impact of difficult samples in DPO settings and finds that overly difficult examples can be detrimental to LLM alignment. Following the curriculum learning (CL) pattern, which organizes examples from easy to difficult, the authors propose Selective DPO. This method utilizes the original DPO loss (referred to as validation loss) as an alternative to the typical metric in CL, the learned step, as a measure of difficulty. The empirical results highlight the effectiveness of Selective DPO, revealing that using only half of the Ultrafeedback examples can achieve better performance compared to using all the data.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful suggestions regarding comparisons and related work. We address each comment in detail below:\n\n**Q1) Comparison with other difficulty metrics.**\n\nPrior work [0] has examined *prompt length* and *attention scores*, finding limited benefits for alignment. Building on this, we conducted experiments with: *completion length*, *perplexity*, *perplexity gap*, and *reward margin*. Full results are available in [this link](https://selective-dpo.pages.dev/) (https://selective-dpo.pages.dev/).\n\nNone of these metrics consistently outperformed our validation-loss-based approach. Notably, sorting by *completion length* (ascending) led to model collapse: the model overfit to short completions and failed to recover, highlighting the potential risks of overly simplistic heuristics.\n\n**Q2) Comparison with additional data selection baselines.** \n\nThe suggested baseline [1] targets multimodal LLMs and introduces *perplexity* to select high quality samples for **SFT**. Following their idea, we implemented two variants in the **DPO** setting:\n- *Perplexity of chosen*\n- *Perplexity gap*\n\nTo avoid arbitrary thresholding, we followed a consistent protocol: (1) sort examples by the metric, (2) train with fixed hyperparameters (Table 3, page 15), and (3) evaluate performance across data percentages. Result is available at the shared link. Key findings:\n\n- *Perplexity of chosen* improves over random sampling, suggesting it is a viable scoring function for curriculum learning.\n- However, when used as a selection filter, it does not clearly distinguish “useful” from “harmful” examples—all data partitions appear beneficial.\n\n---\n**(Q3) Missing related work.** \n\nThank you for pointing out relevant papers. We have reviewed and will include them in our revised Related Work section:\n\n- [1] emphasizes diversity and complexity for SFT data selection.\n- [2] introduces a tree-based measure of data difficulty and finds complex SFT data contributes the most.\n- [3] proposes LLM-rated quality scores and emphasizes the role of data quality in SFT.\n- [4] explores a general data selection framework for pre-training and fine-tuning.\n\n---\n\n**(Q4) Can curriculum learning benefits SFT?** \n\nOur work centers on **data selection for alignment**, not SFT. We discuss curriculum learning primarily as a tool to investigate example difficulty for alignment tasks. While we are encouraged by positive signs (e.g., Figure 3), we acknowledge that the following discussion is preliminary and may lack the nuance of dedicated studies.\n\n- **CL in alignment**: We observe modest gains in our ablation (Figure 3) and expect greater benefits with refined pacing functions. However, designing these strategies is outside the scope of this work.\n- **CL in SFT**: This remains a promising area. Prior studies ([5], [6]) show benefits for learning robustness and reasoning tasks.\n- **Difficulty-aware selection in SFT**: Although we focus on alignment, the core idea—that overly difficult examples may hurt small models—may extend to SFT. In particular, [7] reports similar challenges in distillation, where small models underperform when exposed to overly complex teacher outputs. However, we don’t think borrowing our finding to SFT is an easy thing since it needs to redesign the *learned step and validation loss* (SFT data has different format).\n\n[5] YODA: Teacher-Student Progressive Learning for Language Models, Arxiv 2024\n\n[6] Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyond, Arxiv 2025\n\n[7] Small Models Struggle to Learn from Strong Reasoners, Arxiv 2025\n\n---\nOnce again, we thank for the insightful comments and references. We appreciate the positive comments regarding our experiment design and we respectfully hope that the reviewer could reevaluate our work given the responses to your concerns.", "claims_and_evidence": "The claims within this paper are well supported by empirical evidences.", "methods_and_evaluation_criteria": "The proposed method, Selective DPO, appears meaningful for LLM preference alignment and could enhance the efficiency of the process.", "theoretical_claims": "This paper does not contain theoretical claims.", "experimental_designs_or_analyses": "The experimental design is valid and well-organized, encompassing a variety of ablation studies. These include different base models, learning rates, and hybrid DPO schemes, such as other DPO-series algorithms that use selectively easy examples.", "relation_to_broader_scientific_literature": "This paper confirms the practicality of using curriculum learning in LLM preference alignment.", "essential_references_not_discussed": "There are no essential references missing.", "other_strengths_and_weaknesses": "**Strengths**\n- The proposed method, Selective DPO, could largely improve data efficiency in LLM alignment.\n- This paper discusses what are difficult examples for LLMs, clarifying that these are not simply data errors in practice.\n\n**Weaknesses**\n\n- The essence of curriculum learning lies in the difficulty metric. Although the authors have discussed simpler metrics in Section 3.1 through the learned step, from a data selection standpoint, it would be more intuitive to conduct experiments that explore and compare these with more simple or relevant metrics, such as completion length or attention scores.\n\n\n- Compared to DPO-series baselines, this work does not focus on constructing a DPO-based loss function, but instead on data selection for LLM preference alignment. Therefore, it would be more relevant to compare it with data selection baselines, such as [1], which is already mentioned in this work.\n\n[1] Curriculum Learning with Quality-Driven Data Selection, NeurIPS 2024.", "other_comments_or_suggestions": "The related work could be improved. More recent data selection work are missing, for example, [1-4].\n\n[1] \"Instag: instruction tagging for analyzing supervised fine-tuning of large language models\", ICLR 2024.\n\n[2] \"A preliminary study of the intrinsic relationship between complexity and alignment\", ACL 2024.\n\n[3] \"Improving Data Efficiency via Curating LLM-Driven Rating Systems\", ICLR 2025.\n\n[4] \"Rule-Based Rating and Selection of LLM Training Data\", arxiv 2024.", "questions_for_authors": "Does curriculum learning also apply to the supervised fine-tuning phase beyond preference alignment? Are there any existing studies that support this?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mJwabgkU0N", "reviewer": "Reviewer_4NoX", "summary": "The paper starts w/ an observation that preference samples have different difficulty level (i.e., how easy/hard it is to learn for abgiven model w/ a different capacity). The paper posits that harder examples deteriorates preference alignment due to the examples being too hard for a model to learn. The way to quantify this is using the earliest training time that the sample is learned correctly.\n\nOn section 3, the paper shows experimental results showing that this is indeed the case-- the number of samples classified as easy/hard correlates w/ the model's sizes. Next, the paper propose to use validation loss as proxy. The paper then shows that including hard samples deteriorates model's performance. And then proposes Selective-DPO: DPO w/ discarding hard samples", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thoughtful feedback and for pointing us to highly relevant related works. Your kind comments have helped us better position our contribution. Below we address the concerns in detail:\n\n**Q1) Missing related work.** \n\nThank you for highlighting `arxiv 2410.08847.` This work focuses on identifying and filtering out training examples that cause *likelihood displacement* in DPO. We believe our work carries different motivations: we study how example difficulty impacts alignment while they analyze which sample cause *likelihood displacement* in DPO. We now include this paper in the Related Work section and highlight the distinction.\n\nRegarding `arxiv 2009.10795`, we appreciate your mention of this important work on training dynamics. Their findings—easy samples have little value, ambiguous ones aid generalization, and hard samples is helpful despite may contain noise—are insightful. However, our conclusions differ in several ways:\n\n- In alignment, hard samples consistently degrade performance.\n- Simple data errors cannot fully explain this degradation—we introduced a series of experiments to support this.\n- Larger models benefit from difficult examples, as shown in Figure 5.\n\nWe believe this difference stems from the models under study: our work focuses on LLM alignment, while theirs analyzes small models and classification tasks. \n\n---\n\n**Q2) Comparison with complicated data selection**  \n\nWe agree that comparing with more advanced data selection methods (e.g., CHES scores from Arxiv 2410.08847) would strengthen our work. However, conducting this comparison is non-trivial and requires:\n\n- Calculating model-specific CHES scores on our datasets.\n- Establishing principled thresholding strategies for selection.\n\nWe are actively working on this and will update our results once we complete a fair and thorough comparison. \n\nAlternatively, we add comparisons against other intuitive data selection metrics including *perplexity gap*, *reward margin*, and *completion length* as suggested by other reviewers. The results are available at [this link](https://selective-dpo.pages.dev/) (https://selective-dpo.pages.dev/). We hope these comparisons would alleviate your concern.\n\n---\n\n**Q3) Evolution of validation loss**\n\nTo better illustrate our intuition, we visualize the *preference probability* metric: $P(y_w >y_l |x)=\\sigma(r(x,y_w) - r(x,y_l))$, which closely aligns with validation loss: $VL = -\\log P(y_w > y_l|x)$. \n\nAs shown in the new results (Figure 13), many easy samples are learned early. Roughly 40% of the samples remain difficult throughout training, indicated by consistently low preference probabilities.\n\nIn response to your question: yes, we do observe a small subset of samples that become “correct” first and “incorrect” after. These samples tend to lie between the easiest and hardest, suggesting the model intermittently understands them—consistent with the “ambiguous instances” concept in the training dynamics work you cited. \n\n---\n\nWe appreciate your encouragement to consider easy and ambiguous examples more deeply. While our current work emphasizes the negative impact of overly difficult samples, we agree that a fuller picture—including the role of ambiguous instances—would benefit the field. \n\nOnce again, we thank the reviewer for the valuable suggestions and for pointing us to relevant research. Your comments have significantly strengthened the scope and clarity of our revision.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "The paper is related to the DPO and data selection for DPO literature.", "essential_references_not_discussed": "Yes, I think there is 1 paper that is very related, but not cited: https://arxiv.org/abs/2410.08847\nAnother thing that i think is somewhat related (but the authors should correct me if I'm wrong), is training dynamics: https://arxiv.org/abs/2009.10795", "other_strengths_and_weaknesses": "1. I feel like it will be beneficial to compare or study the relation w/ more complicated data selection techniques that uses other criteria such as: https://arxiv.org/abs/2410.08847\n2. Another thing I'm wondering is: Does the validation loss/lerning time changes for each sample as we increase the number of epochs? like for example, if it becomes correct at epoch 2, but then becomes incorrect again in epoch 3, what does this mean?", "other_comments_or_suggestions": "See Strength/Weakness", "questions_for_authors": "See Strength/Weakness", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qhdmIYOlgc", "reviewer": "Reviewer_Utw6", "summary": "This paper investigates the effect of “difficult” examples in preference optimization (particularly, in the context of DPO).\nIt finds that these examples harm performance, and propose a data selection algorithm to filter these examples to be applied before DPO.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their careful reading and insightful comments. We address the concerns below:\n\n**Q1) Claims about hallucinations**\n\nWe revised the statement “reduces undesired hallucinations” to “generates policies that have lower NLLs”.\n\n---\n\n**Q2) Application to other DPO variants**\n\nOur study evaluates data selection on two datasets and four LLMs, showing significant gains. We agree that testing additional DPO variants is valuable. However, extending our method is non-trivial, as the curriculum relies on model, dataset, and loss function. To address your concerns, we include comparisons with other data selection baselines (see Q3).\n\n**Q3) Comparison with more data selection baselines.** \n\nThe mentioned confidence-based selection method selects data using reward margins from golden reward functions. We reproduce their idea using GPT-4 ratings: $r^*(x,y_w) - r^*(x,y_l)$ and sort samples by this reward margin. Results are reported in [this link](https://selective-dpo.pages.dev/) (https://selective-dpo.pages.dev/), labeled with `Reward Margin (Descending, GPT-4)` .\n\nWe observe no consistent performance benefit, suggesting that reward margin is not a reliable selection metric in our setting. This may stem from our focus on real datasets, in contrast to the mentioned work using manually corrupted labels.\n\n---\n\n**Q4) Evaluation on MMLU**\n\nWe report MMLU and related results in Table 8 (page 17). Performance is broadly consistent across DPO variants. One exception is GSM8K, where SelectiveDPO-Mistral-7B often produces correct answers in dialogue form rather than the strict format `### <The Answer>`. This issue is resolved by including ~10% more math-style examples during training.\n\n---\n\n**Q5) Definition of the learned step**\n\nWe defined the *learned step* as the step after which the *implicit reward model* can distinguish the preferred from rejected answers with a large probability: $P(y_w > y_l | x) = \\sigma(\\beta\\frac{\\pi_{\\theta}(y_w |x)}{\\pi_{\\text{ref}}(y_w|x)} - \\beta\\frac{\\pi_{\\theta}(y_l |x)}{\\pi_{\\text{ref}}(y_l |x)}) > \\sigma(\\delta)$. This follows the beautiful DPO formulation and reflects the intuition that the LLM is secretly a reward model. Defining the *learned step* using only $\\pi_\\theta$ is indeed cleaner and worth further exploration.\n\n---\n\n**Q6) Missing related work**\n\nWe appreciate the reviewer’s reminder. The mentioned work observes similar trends in **SFT** training. However, its definition of “unfamiliar knowledge” does not directly apply to **alignment** tasks.  We agree the work is very relevant and have now cited it in our Related Work section.\n\n---\n\n**Q7) Scope of the data selection strategy**\n\nWe thank the reviewer for recognizing our contribution. The proposed selecting strategy is tailored for alignment data (prompt–preferred–rejected). It does not transfer directly to pre-training or SFT tasks, which use corpus or prompt–completion pairs and would require adapted difficulty metrics. However, we note that many SFT data selection papers (including the one mentioned) are not tested on alignment or pre-training tasks. This is not typically viewed as a limitation. \n\n---\n\nWe sincerely thank the reviewer once again. We have revised the paper to include additional baselines and discussions addressing the raised concerns. With these updates, we hope the reviewer may find the work strengthened and reconsider their evaluation.", "claims_and_evidence": "The claims are mostly convincing. Certain claims, e.g., “Selective DPO … reduces undesired hallucinations” are not directly supported by evidence. The authors should remove this claim if they do not specifically investigate hallucinations.", "methods_and_evaluation_criteria": "There are a few areas in which evaluations could be improved (in decreasing order of importance):\n\n1. It feels strange that the authors compare Selective DPO to DPO variants like SimPO which involve an algorithmic change rather than data selection. Rather than comparing Selective DPO to other preference optimization algorithms, to illustrate the value of data selection the authors should be applying data selection to each of these algorithms and showing improvement over the base algorithm (without data selection). For example, they should compare SimPO to Selective SimPO.\n\n2. The authors mention other data selection methods in the related works (e.g., confidence-based selection) but do not include these as baselines for their method. These feel like more important baselines than other preference optimization algorithms like SimPO.\n\n3. Currently, the authors only evaluate performance on model-based preference evaluations (a model rates how much it likes a response, or which of two responses it prefers). This evaluation can be biased and opaque. It may also be of interest to measure the effect of this data selection method on benchmarks with a ground-truth. For example, does this improve MMLU accuracy?", "theoretical_claims": "The paper does not include theoretical claims.", "experimental_designs_or_analyses": "In the definition of “learned step”, why is the reference model being considered? It feels like it should only depend on the model itself. In particular, even the reference model might be able to reliably distinguish certain preferred vs. rejected answers, but in this formulation it wouldn’t be distinguishing any of them.", "relation_to_broader_scientific_literature": "Previous work has suggested that language models should be trained on data on which they exhibit high confidence, e.g., on facts that are well-known [1]. This is the same principle as the authors apply to alignment here. These works should be discussed and cited.\n\n  [1] Ghosal, Gaurav, Tatsunori Hashimoto, and Aditi Raghunathan. \"Understanding finetuning for factual knowledge extraction.\" arXiv preprint arXiv:2406.14785 (2024).", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n\nThe authors provide a nice analysis of an intuitive principle for data selection. They are rigorous in explaining its effectiveness (they make an effort to rule out issues like labeling errors as alternative causes for why removing difficult examples helps). The experiments illustrating that model capacity plays a role in difficulty are quite convincing and valuable.\n\nWeaknesses:\n\nBesides the weaknesses described in the individual sections above, this paper may be of limited significance if these findings only hold in a preference optimization setting, and, in particular, for DPO (rather than for its variants or RLHF).", "other_comments_or_suggestions": "No.", "questions_for_authors": "The author’s seem to be hinting at a broader claim about datas selection: that difficult examples hinder performance. Why do they constrain their experiments to an alignment/preference optimization setting? Do these results transfer to other settings (e.g., SFT, pre-training)? If not, then what makes preference optimization particularly amenable to this sort of data selection?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper looks at data selection for alignment. The main claim is simple but reasonable: alignment is hurt by examples that are too hard for the model to handle. The notion of difficulty here is related to the model capacity, rather than noise or other causes. The authors then show a simple approach to detect points that are too hard based on looking at the validation loss. This leads to a straightforward data selection approach. The paper does a significant amount of experiments to validate these ideas.

This paper is simple but nice. The reviewers largely agreed with this; they asked a variety of clarifying questions as well for more evidence for some of the key claims (e.g., the choice of detection approach instead of others). The authors addressed most of these questions. Perhaps the main weakness is how useful the result is given that it is focused on alignment specifically, but nevertheless I think it is worth accepting a solid set of findings.
Decision: Accept (poster)