Towards Understanding Valuable Preference Data for Large Language Model Alignment
Authors: Zizhuo Zhang
Qizhou Wang
Shanshan Ye
Jianing Zhu
Jiangchao Yao
Bo Han
Masashi Sugiyama
Venue: ICLR 2026 Poster (2026)
Keywords: Large language model alignment
preference data
influence function
Primary area: foundation or frontier models, including LLMs
TL;DR: We assess preference data quality through our newly proposed truncated influence function (TIF), and then we propose a set of candidate scoring functions that are positive correlated with TIF to select valuable preference data.

Abstract: Large language model (LLM) alignment is typically achieved through learning from human preference comparisons, making the quality of preference data critical to its success. Existing studies often pre-process raw training datasets to identify valuable preference pairs using external reward models or off-the-shelf LLMs, achieving improved overall performance but rarely examining whether individual, selected data point is genuinely beneficial. We assess data quality through individual influence on validation data using our newly proposed truncated influence function (TIF), which mitigates the over-scoring present in traditional measures and reveals that preference data quality is inherently a property of the model. In other words, a data pair that benefits one model may harm another. This leaves the need to improve the preference data selection approaches to be adapting to specific models. To this end, we introduce two candidate scoring functions (SFs) that are computationally simpler than TIF and positively correlated with it. They are also model dependent and can serve as potential indicators of individual data quality for preference data selection. Furthermore, we observe that these SFs inherently exhibit errors when compared to TIF. To this end, we combine them to offset their diverse error sources, resulting in a simple yet effective data selection rule that enables the models to achieve a more precise selection of valuable preference data. We conduct experiments across diverse alignment benchmarks and various LLM families, with results demonstrating that better alignment performance can be achieved using less data, showing the generality of our findings and new methods. Our code is publicly available at~\url{https://github.com/tmlr-group/TIF_LossDiff-IRM}.

Published as a conference paper at ICLR 2026
TOWARDS UNDERSTANDING VALUABLE PREFERENCE DATA FOR LARGE LANGUAGE MODEL ALIGNMENT
Zizhuo Zhang" Qizhou Wang',
Shanshan Ye Jianing Zhu'
Jiangchao Yao
Bo Han 1,5t Masashi Sugiyama 4.5
'MMR Group, Department of Computer Science, Hong Kong Baptist University
University Technology Sydney *CMIC, Shanghai Jiao Tong University
'The University of Tokyo RRIKEN Center for Advanced Intelligence Project {cszzzhang, csqzwang, csinzuu, bhanml @comp hkbu. edu. hk shanshan. ye@st u au, Sunarkeres jtu edu. en suqišk.u-tokyo.c
ABSTRACT
Large language model (LLM) alignment is typically achieved through learning from human preference comparisons, making the quality of preference data critical to its success. Existing studies often pre-procest raw training datasets to identify valuable preference pairs using external reward models or off-the-shelf LLMs, achieving improved overall performance but rarely examining whether individual, selected data point is genuinely beneficial. We assess data quality through individual influence on validation data using our newly proposed truncated influence function (TIF), which mitigates the over-scoring present in traditional measures and reveals that preference data quality is inherently a property of the model. In other words, a data pair that benefits model may harm another. This leaves the need improve the preference data selection approaches to be adapting to specific models. To this candidate scoring functions (SFs) that computationally
end, we introduce simpler than TIF and positively correlated with it. They are also model dependent and can serve as potential indicators of individual data quality for preference data selection. Furthermore, we observe that these SFs inherently exhibit errors when compared TIF. To this end, we combine them offset their diverse error sources, resulting in a simple yet effective data selection rule that enables the models to achieve a more precise selection of valuable preference data. We conduct experiments across diverse alignment benchmarks and various LLM families, with results demonstrating that better alignment performance can be achieved using less data, showing the generality of our findings and new methods. Our code is publicly available at https: //gg://gtttt /ithtub. com/ tml r- wwwggrruupsooooo /TIF TIF LossDiff
INTRODUCTION
Reinforcement learning with human feedback (RLHF) has emerged as dominant fine-tuning paradigm for aligning large language models (LLMs) with human preferences (Bai et al., 2022b; Wang et al., 2025b). Whether through training explicit reward models (Lambert et al., 2024) or optimizing policy with implicit rewards (Rafailov et al., 2023), the success of RLHF hinges heavily on the availability of high-qualic preference data. Previous works (Shen et al., 2024; Pattnaik et al., 2024; Morimura et al., 2024; Deng et al., 2025; Chen et al., 2024) typically leverage external reward models off-the-shelf LLMs filter raw data, treating selected data reliable sources for training their models or releasing them for open-source use (Cui et al., 2023; Bai et al., 2022a). This pre-processing paradigm dominates the RLHF community to improve data quality, contributing remarkably to the success of many publicly available LLMs such as Llama (Vavekanand & Sam, 2024), Qwen (Yang et al., 2025) and DeepSeek (Guo al., 2025).
'Trainee at RIKEN AIP.
1 "Correspondence to Shanshan Ye
(shanshan.yee wtw.den..us.edu.... and
Bo
Han
(bhanml @ komp.hkbu.edu.hk)

---

Published as a conference paper at ICLR 2026
However, such pre-processing implicitly assumes that quality is an intrinsic property of data them- selves: regardless of training configurations or models, certain data are consistently presumed to be more valuable for alignment than others. Therefore, seemingly more realistic perspective is that the data quality is also property of the model (Grosse et al., 2023; Xia et al., 2024). That is to say, some data points may be beneficial for certain models configurations while being detrimental at others. We validate this new assumption as more reasonable based on the influence function (IF) (Koh & Liang, 2017), which quantifies the impact of each training data point on validation performance, thereby reflecting data quality. Nevertheless, we observe that the influence scores may overfit to the validation data, an issue that is particularly severe for open- we propose and verify simple modification to the original IF, namely fruncaled influence function (TIF), which is shown to be more reliable for preference data selection and results in better overall performance. Using TIF, we verify that data quality is model-dependent, varying across different models, with some data points proving beneficial for certain models yet harmful at others.
models like LLMs. To address this,
Our above analysis suggests a reasonable yet seldom-discussed viewpoint: preference data selection should be performed for specific models and explicitly related to the training process, Although TIF provides reliable and effective measure of data quality, its high computational cost on gradients limits its direct application for large-scale LLMs (Kwon et al., 2024). To address this, we introduce two simpler scoring functions (SFs) with lower computational costs yet sufficient potential to approximate TIF-loss difference (LossDiff) and implicis reward margin (IRM)-that are positively correlated with TIF yet require only forward passes. Considering that a single SF may exhibit specific errors relative to TIF, a combination indicator LossDiff- -RM is proposed to mitigates such errors, as their distinct bias sources may offset one another. Empirically, LossDiff- IRM achieves an average WinRate improvement of +13.58% over full-data training while using only 50%-644 64% of the data, across multiple LLM families, benchmarks, and alignment methods.
2 PRELIMINARY
To begin, we formalize pairwise preference data and introduce Direct Preference Optimization (DPO) as the base preference optimization approach used in our analysis and experiments, and then briefly review prior work on preference data selection.
Pairwise Preference Data. Pairwise preference dataset, denoted as D {d; = (xx(1, yic 3t >
(6)
is annotated by humans to reflect real human preference. Each pair consists of prompt and pair of responses: the chosen response yo and the rejected response yt, which means that the human annotator prefers the chosen response rather than the rejected one, denoted the y1-
Direct Preference Alignment. Traditional RLHF (Bai et al., 2022b) often follows a two-stage pipeline: first training reward model on preference data, and then optimizing the policy L.M using reinforcement learning (RI. algorithm such as PPO (Schulman et al., 2017). The two-stage RI.FF is relatively complicated and resource-intensive. Recently, many studies (Rafailov et al., 2023; Azar et al., 2024; Zhao et al., 2023; Meng et al., 2024; Wu et al., 2024b;a) have tried to bypass the need for an explicit reward model and RL learning, to directly optimize the policy model from preference data. Among them, DPO (Rafailov et al., 2023) is a milestone work that derives the optimal policy and substitutes into the Bradley-Terry (BT) model (Bradley & Terry, 1952), which yields its objective:
To (y x)) Tret (ye [cc)
To (x)
(Dpo D)- = -B(G..m..WEED WDDD log a log
,3 log
(I)
Teer T (x)
where B > 0 controls the strength of the KL penalty between Te and mief and is the sigmoid function. According to the derivation of DPO loss (Rafailov et al., 2023), the term inside the sigmoid can be interpreted implicit reward margin (IRM) between the chosen and rejected responses:
To (V (x) (Ber(B/.zz)
TT(()
IRM = =3loog
Blog
(2)
Tiet (v)
This reward margin serves as an important indicator monitored during training to track how well the model differentiates the chosen response from the rejected one. For an individual data pair, the margin is negatively correlated with the DPO loss: larger margins correspond to lower loss values. Intuitively, the DPO adopts a contrastive-like structure that directly trains the policy LLM on preference pairs by
2

---

Published as a conference paper at ICLR 2026
encouraging larger relative reward margin for the chosen response the rejected one. The other alignment method SLiC (Zhao et al., 2023) used in this work is introduced in Appendix B.1.
Preference Data Quality. Given annotator subjectivity and the open nature of the concept of preference, manually annotated preference data can be imperfect or noisy. Existing studies adopt multiple data processing strategies: Morimura et al. (2024) and Deng et al. (2025) filter out low-quality preference pairs based on external reward models. Pattnaik et al. (2024) organizes the preference data in curriculum based on metries such as GPT-4 score, external reward score, or log probability. Muldrew et al. (2024) and Shen et al. (2025) introduce active learning to improve data quality and annotation efficiency. Overall, these approaches mainly rely on external signals (e.g., GPT reward model scores, or active human labeling) and implicitly view preference data quality as a property of the data itself, while overlooking the role of models, training configurations, and optimization objectives in shaping utility of preference data. Detailed related work refers to Appendix A.2.
3 ANALYSIS: TRUNCATED INFLUENCE FUNCTION (TIF)
Building upon influence function (IF), this section takes a model-centric perspective to investigate what kind of preference data truly valuable for model alignment.
3.1 INFLUENCE FUNCTION
To quantify the quality of data sample d E Dunin classical idea is to measure its leave-one-out (LOO) (Evgeniou et al., 2004; Elisseeff et al., 2003) effect, which assesses the change in validation performance when the model is trained with versus without the data sample d:
LOO Effect(d)) = (0p... Dvai) - ((pun)(a)) Dvat):
where u(.) denotes a measurement to evaluate the model re, and 0D denotes the model trained on the dataset D. Intuitively, computing the exact LOO effeet requires training [Prainn + 1 separate models, which is infeasible in practice. To avoid repetitive retraining the model, influence function (IF) (Koh & Liang, 2017) provides first-order Taylor approximation of the LOO effect using gradient-based metric at the current parameters:
[F(; mo; Dvat) = (O0 val)) HJ vol(O:d) N DoL(O;Dra)) (ol(((d)
(4)
1
Vol (0: (B.d Vol(O:d),
(5)
Deal
where E(0; d) is the training loss for the data d, and He V2 D usin) is the Hessian matrix (Ham- pel, 1974) of the total training loss with respect to model parameters 0. In practice, computing and inverting the Hessian matrix often computationally intractable, especially for a large-scale model. A common approach is to approximate the Hessian Ho as the identity matrix by assuming that the loss landscape is locally isotropic near 9 (Koh & Liang, 2017: Wang et al., 2025a; Xia et al., 2024). This simplification reduces the IF to dot product between the gradient of the training data d and the expected gradient of the validation set. Intuitively, a higher IF means the training gradient better aligns with the validation gradient, suggesting the data is more beneficial for generalization.
Instantiation to DPO Loss. By taking the DPO loss in Eq. (1) into the IF formulation in Eq. (4), we instantiate the IF under DPO objective for a give preference pair d (zz Nao y1):
IFDPo ne; Dea) = 3(1 - o(A)))
B D
Σ (1-
(90) of
current preference pair direction
preference generalization dircction w.t.t. validation set
(6) where g. Va log (y, |z: denotes the gradient of the log- liielihooddd for response y, E (yurs V4
and Ag = ,3 log [o(w IF under pointwise objectives, which assesses the gradient consistency of a give data point with validation set (Koh & Liang, 2017; Wang al., 2025a), the DPO-based IF focuses the gradient
Ra [v] B log ro(p)z) is the reward difference term in the DPO loss. Unlike Tar(yu ø)
3

---

Published as a conference paper at ICLR 2026
Oreet
Margl
Salee bakua
loke)
God
Molle
13
12
Cooked 110
0.5
60
25
16
30 16 160
Figure 1: Analysis of Ia-ed data partition on the Qwen3-0.6B-Base and Llama-3.2-1B. From Left to Right: Training DPO loss, evaluation DPO loss and evaluation reward margin. Subsets of (small, medium, large) are denoted by {blue, orange, green), respectively, while different line styles indicate IF values computed using different epoch checkpoints. More analysis refers to Appendix C.I.
difference consistency, i.e., 9w - 9t. with the validation preferences, thereby serving as a proxy to assess quality of preference pair d. Vanilla IF implicitly assumes the validation set is oracle of generalization, thereby equating high-IF data with high quality. However, under this assumption, IF may overfit to the specific validation set. The instantiation to SLiC loss is provided in Appendix B.
3.2 ANALYSIS: TRUNCATED INFLUENCE FUNCTION (TIF)
Since preference alignment is open-ended and a fully reliable validation set is hard to obtain, we propose the Truncated Influence Function (TIF) by partitioning data into three regions for analysis.
minn D Denah from UltraFeed- back (Cui et al., 2023), containing 5,000, 3,000, 300 preference pairs for training, validation, and testing, respectively. (wwen3-066--Bl. 2024) are adopted as backbone models. Both models are first trained with supervised fine-tuning (SFT) on UltraChat-200K (Ding et al., 2023), followed by five epochs of DPO on Drrb At each- epoch DPO checkpoint, we compute the IF values of all training pairs using Eq. (5) with respect to Detub and partition them into three equally sized subsets (small-, medium-, and large-IF). Each subset then used to continue DPO training from the checkpoint, while we monitor the evaluation DPO loss and reward margin on Dpaub to examine which type of data more beneficial for alignment training.
Analytical Setup. We randomly sample a probing set De (Diihs
(Yang et al., 2025) and Llama-3.2-- IB (Vavekanand & Sam,
Figure 1 reports the training dynamics on Qwen3- -.o.o----BBBBB. -Bas and Llama- 3.2- IB with different IF- based partitions. Across both models, we observe several similar phenomena as follows:
Small-IF data: Training loss decreases as expected, but evaluation loss increases while the evalu- ation reward margin falls below zero. This indicates that small-IF pairs are largely uninformative and of low quality, with a high likelihood of being noisy or ambiguous. Learning from such data not only fails to help the model distinguish chosen from rejected responses but may also mislead it, thereby harming alignment training and preference generalization.
Large-IF data: Evaluation loss initially decreases but later rises, while the evaluation reward margin continues to increase. This mismatch between loss and margin trajectories counterintu- itive, since DPO loss and reward margin are normally negatively correlated at the per-sample level. The phenomenon indicates overfitting: training on large-IF data enlarges the margins of a small subset of pairs while reducing the margins of many other pairs. Owing the sigmoid saturation, overly large margins contribute little to the loss, whereas the diminished margins of the majority dominate it, ultimately causing the model to overfit to narrow portion of preference pairs.
4

---

Published as a conference paper at ICLR 2026
Table 1: Computational time and throughput rate analysis on probing set by Qwen3-0.6B-Ban. and Llama-3.2-1B using one H100-80GB GPU. Top: Exact IF, consisting of the validation -ngaaiiinnng computation and the per-pair IF anner-product computation. Bottom: LissDiff -Rff-IRM, consisting of one forward on the training model and one forward on a validation-aligned auxiliary model.
Computational Time
Throughput Rate (pair/sec)
IF Computation
Val Gradient
IF h 59 45 6 02 m 37
Total
Val Gradient
IF
Qwen3-0.6B-Base
17 31s 3 555 m 32
3h 16s
1.55
1,44
1.2ma-3,2-18
9 58 m s
4.71
4.35
LossDif-IRM Computation
Training Forward
Val Forward
Total
Training Forward
Val Forward
Dwena--.6B-Bass
min 5 2 m 32
59 m 27
2 2 m 59
76.92
84.74
Llama-3.2
32.86
33,80
Table 2: Overlap Coefficient analysis to assess the overlap of two selected sets.
Overlap Coefficient
LossDiff vs. IF
IRM IF
LossDiff-IRM vs. IF
Models
Epoch I ckpt
Epoch 2 ckpt
Epoch ckpt
Epoch 2 ckpt
Epoch ckpt
Epoch 2 ckpt
Qwen3-0.6B-Buse
0.6953
0.6639
0.6883
0.6470
0,7820 0.7655
0.7257
LJama-3.2-11 1B
0.6687
0.6582
0.6969
0.6025
0.6963
As training loss decreases, evaluation loss steadily decreases while reward margin increases, which aligns well with the intended learning dynamics of the DPO objective: improving the model ability to distinguish cumaan-pefferred (chosen) responses from rejected ones. This demonstrates that medium-IF preference pairs are high-quality data, providing the most effective and stable signal for alignment training and fostering better preference generalization.
Medium-IF data:
Overall, these findings differ from vanilla IF in traditional classification, where high-IF data is typically regarded the most valuable. In preference alignment, however, the most valuable data are the medium-IF preference pairs. This counter-intuitive finding is nonetheless reasonable: preference aligmment an open-ended task in which the chosen- rejected annotations inherently affected by annotator subjectivity, making the validation gradient an imperfect proxy for reflecting the real human preference direction. As a result, data with both extremely small and extremely large IF values is low-quality, which provide little useful signal for alignment training. Therefore, we propose the Truncated Influence Function (TIF), which offers a more robust criterion for assessing preference data quality in LLM alignment training:
(IF me: Dval) = [m < IF(d; me; Dval) < ónage)]
where &mall and Starge denote threshold percentiles that specify the boundaries of IF values. It can be observed that the criterion of TIF depends on the current model no, which suggests that the identification of valuable preference pairs is inherently model-dependent.
METHODOLOGY: LOSSDIFF-IRM, DATA SELECTION
Although TIF provides principled criterion for assessing the value of preference data, its computation requires gradients on both the training and validation sets, which becomes prohibitive in the large- scale model and dataset regime. As reported in the top of Table 1, computing exact IF a probing set of 5,000 pairs for a small Alama-3.2-11 BB model still takes about 10 hours, which is already prohibitively slow. This computational cost makes TIF impractical as a per-sample scoring function for preference data selection in alignment training at scale.
4.1 APPROXIMATION PROXY OF TIF
To address this challenge, we introduce lightweight, model-dependent indicators that require only forward passes yet track TIF well. Concretely, we use two approximation proxies that are positively correlated with IF the per-pair level for efficient preference data selection as follows:
Validation- scoring function: Loss Difference (LossDiff). To bypass costly gradient com- putations, we introduce an auxiliary model that is aligned on the validation set and approximate IF by the loss difference between the current model and this auxiliary model. Specifically, for each preference pair, we define its loss difference as follows:
LossDiff( d; no, To..) e(0: d) E(Ovat: d)),
(8)
5

---

Published as a conference paper at ICLR 2026
IF Lowdilf (iveal-----aa lpeck
1F No Leadli E. L
1F Laudet 33B Ep
Wi IRN CQUuSi-AA E1
IF'm BoN (quua Alt Exech
IF IRM dlane
1. Roach
IF vn BAS (1ame Rooch
Leg Sowwe Meeen 1000
Figure 2: Correlation analysis on Qwwen-006B--ell and Llama-3.2-1B. Top: correlation between loss difference (LossDifff) and IF. Bottom: correlation between implicit reward margin (IRM) and IF.
where a is the auxiliary model aligned validation set. An intuitive understanding is that larger value indicates that moving from 0 toward Oval reduces the loss on d, which is consistent with the direction favored by the validation objective. We formally demonstrate that the LossDiff has a positive correlation with the IF in Appendix B.2. Empirically, the top panel of Figure 2 shows strong Pearson (Cohen al., 2009) and Spearman (Hauke & Kossowski, 2011) correlations (e.g.. r = 0.77, 0.80 on Qwen-0.6B 0.6B-Base). Notably, this proxy requires only two forward passes per pair without backpropagation and remains tied to the validation set by construction.
Implicit Reward Margin (IRM). Empirically, we find that the IRM defined in Eq. (2) exhibits a strong positive correlation with IF at the per-pair level. As illustrated in the bottom panel of Figure 2, the Pearson and Spearman correlations between IRM and IF reach r == 0.67 and p = 0.66 on Llama-3. I-BB, respectively. Intuitively, IRM measures how strongly the current model n prefers the chosen response over the rejected response relative to reference, and thus reflects the model--percivvee mainly pushes the policy to enlarge that pair's margin; the validation objective is doing the same at the validation distribution level. Consequently, pairs with larger positive IRM tend to produce updates that more consistent with the validation objective and thus yield larger IF. Moreover, IRM is a validation freee scoring function and requires only forward computation from Ty.
Validation-free scoring function:
difficulty of a preference pair. When training on a pair, the update
From the Pearson and Spearman correlations, we also observe that the validation-based LossDiff correlates higher with IF than the validation-free IRM. This is expected, as LossDiff explicitly leverages a validation-aligee alignee auxiliary model, with the validation set serving as a reference direction for human preference. In contrast, IRM relies solely on the internal signals of current model, which makes more lightweight but also less anchored. More analysis is provided Appendix C.2.
4.2 LOSSDIFF-IRM PREFERENCE DATA SELECTION
Using scoring function of either LissDiff or IRM alone may introduce method-specifi errors. To offset the diverse errors, we propose a combined indicator: LossDiff-IRMM, which selects data falling within the intersection of the medium percentile ranges defined by LossDiff and IRM. Specifically, LossDiff-IRe -RMfff-IMMMM- selects a preference pair if and only if
LossDiff- Te;
Deal) I[Esmall
< LossDiff((d d; no; Dowi) Elagge]
A I/Tsmall < IRMM(: To) < Thangel
where LossDiff-IRMdd; ne; Daa)) E 10. (Esmall, Starge)) and (Tsmall, marge) are percentile thresholds that define the medium ranges for LossDiff and IRM, respectively. The procedure is as follows: we first warm up Te by training for a short stage (e.g., one epoch) on the training set, then obtain Te.. by training for IRM using Tp. Based on the combined rule in Eq. (9), we select preference data and retrain Ta for a longer stage (e.g., two epochs in our experiments).
stage the validation set. We then compute LossDiff using both my and Te... and

---

Published as a conference paper at ICLR 2026
Table 3: Performance of LossDift-IRM and baselines using DPO and SLiC. Cell background
colors indicate relative performance: darker colors denote better results within each model group
Milloc
1.51
1.43
700
71:31
107
16:41
6.34
73.55
1.3
1095
3:44
1.3
03:38
1:3
7521
605
0::51
6:1
$1.00
8.60
4.50
$4.38
1.80
02:1
1:31
01.5
1.3
22.13
61.3
LeudesiBM
3.30
65.15
Kocki
690
54.03
08.13
4.83
4.50
4123
2.6
OPPE
100
Tautte 1834
19:00
1838
F:hie F1MM JAPOA
Jome L.M 9XA63
236
2.47
3.10
131
1.56
3.0
2.51
Korkia
396
241
16:03
3.30
54
1.60
3018
04.53
4.89
0110
3.26
V4.3
1.80
216
1.41
14:44
1.0
16.16
2.89
$0.07
PuK002500 SEAO
57
687
6.85
1.56
6.64
STT
4.25
4.56
3.14
FalDa
200
235
07.36
741
55:11
8.15
52.05
131
20.
1006
6.6
23.40
1.8
0:00
6.30
6000
3.00
Tmon Malr
Edb
64000
11:08
9915
PaNe- OSCAT3
1.50
1.91
115
10V
10.00
14:0
71.48
1191
1.01
11.55
7.01
17.45
18:33
250
500
3.50
7000
00.11
Figure 3: Ablation study WinRate. Comparison of LossDiff-IRA. vs. its ablations that select data using only LossDiff ("w/ LossDiff") or only IRM ("w/ IRM"). Top: DPO training. Bottom: SLiC training. Single-Score results are provided in Appendix E.1 and Figure 12.
Empirical evidence. We quantify the overlap of two selected sets using the Overlap Coefficient Overlap(A, A, B) min(l m) E [0, 1). As reported in Table 2, the combined selector LissDiff- IRM achieves higher Overlap Coefficient with the exact TIF-selected set than using LissDiff or IRM alone across Qwen3-0.6B- BBase and Llama-3.2-1B This indicates that combining the two scoring functions yields selection that more closely approximates the TIF while while being more computationally efficient at scale. As shown in the bottom of Table 1, LossDiff-IRe requires substantially less time and achieves higher throughput; for example, on Llama-3.2-1B it takes about 5 minutes versus roughly 10 hours for exact IF.
AnB
5 EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Setup. Our experiments are conducted on diverse LLM families, including Llama-3.1- 8B (Vavekanand & Sam, 2024), Qwen3-8B-Base (Yang et al., 2025), and the Pythia series (2.8B/1.4B/410M) (Biderman et al., 2023). Following prior work (Ko et al., 2024; Wu et al., 2024a), each pretrained LLM is first initialized with one epoch of supervised fine-tuning (SFT) on UltraChat-200k (Tunstall et al., 2024), which serves the starting point for subsequent align- ment. We adopt UltraFeedback- Binaaarebbac-BBiaizzeeedd previous studies (Pattaikk et al., 2024; Gao et al., 2025; Ko et al., 2024). For validation set, perform stratified sampling to select 20% of the alignment data according to the GPT-4 score margin between the chosen and rejected responses, and use the remaining 80% as training set. We employ
Binarized (Cui et al., 2023) as the alignment dataset, consistent with
7

---

Published as a conference paper at ICLR 2026
Table 4: Performance comparisons of LossDifT-IRM with existing methods, including Cur-
riDPO (Pattnaik et al., 2024), MAP (Huang et al., 2025), and RS-DPO (Khaki et al., 2024).
UltraFeedbuck
AlpacaEval
Vieuna- Bennh
Arena-Hlard
Training Data
Single
WinRate
Single
WinRate 1
Single
WinRate
Single
WinRate
Llama-3. 1-8B (DPO)
CurriDPO-GEE 4PT4
5.47
74.23
5.53
75.01
5.84
74.06
4.55
79.77
CurriDPO-Reward Model
5.51
74.62
5.54
74.29
5.59
74.06
4.62
79,49
MAP
6.04
79.99
6.21
80.88
6.34
74.69
5.08
85,30
RS-DPO
75,98
6.39
84.84
7.04
82.81
4.69
82,05
LossDiff-IRM
6.54
83.97
6.84
87.08
7.06
86.88
5.59
88.40
Qwen3-80- (DPO)
CuriiPPO-GPTI
7.61
61.04
7.74
62.35
8.16
54.69
7.52
62.51
CurriDPO- Rewerrd--wrrddd Model
7.60
59.62
7.84
61.96
8.20
61.58
7.55
63.97
MAP
7.95
67.84
8.31
71.11
8.62
66.87
7.72
66.55
RS-DPO
7.88
64.87
8.36
74.07
8.93
71.90
7.48
61.32
LossDiff-IRM
8.05
67.32
8,36
71,52
8.72
67.19
7.83
68.63
Abodin
Jul tox boin
cull Sunes
Figure 4: Impact of noisy validation set. Performance curves across different noise rates {0.0, 0.2, 0.4]; dashed lines denote performance of Full-Data training.
DPO (Rafailov et al., 2023) and SLiC (Zhao et al., 2023) as core preference optimization algorithms used in our experiments. More details of experimental settings are provided in Appendix D.
Evaluation Setup. We evaluate the aligned LLMs open-ended generation tasks using both in-distribution (ID) and out-of-distribution (OOD) benchmarks. The ID evaluation is conducted on the UltraFeedback test split, while OOD evaluation includes AlpacaEval (Li et al., 2023), Vicuna- Bench (Chiang et al., 2023). and Arena-Hart (Li et al., 2024). We compare against full-data training (Full Data) and several preference data selection strategies commonly used in prior work (Pattnaik et al., 2024; Deng et al., 2025; Morimura et al., 2024): random sampling (Random), GPT-4 score filtering (GPT4), and external reward-model filtering (Reward Model). We also compare with several existing methods: curriculum learning- baanin CurriDPO (Pattnaik et al., 2024), margin-bases selection MAP (Huang et al., 2025), and rejection sampling-baad based preference data generation method RS-DPO (Khaki et al., 2024). All methods are trained under the same setup for a fair comparison. We report two evaluation metrics via LLM Ls--as-udgee (Zheng al., 2023a): Single Score (Single) and Length-controlled Win Rate vs. SFT (WinRate). Further details are in Appendix D.3.
5.2
EXPERIMENTAL RESULTS
5.2.1 MAIN PERFORMANCE OF LOSS-IRM FOR PREFERENCE DATA SELECTION
LossDifT-IRM achieves better performance. Table 3 summarizes the performance of Loss-IRM and competitors with DPO and SLiC. Across diverse LLM families, benchmarks and metrics, it can be observed that LossDiff Miff-IRMMoo- occupies more darker cells, and surpasses full-data training while using only about 65% of the data, indicating stronger performance. Specifically, compared to full-data training. Diff-iffI achieves average WinRate improvements of +11.4%% + +15.14%, Breen-88--Base and 2.B/aai2.8/L..4//41111
+ +16.63% + +88.28% and +17.71% on 1lama-3.1-8BB with DPO, respectively. Furthermore, we compute exact TIF on smaller Pythia-410M. Table 5 shows that training data selected by exact TIF and by LossDiff- -I achieves comparable performance. These results demonstrate the effectiveness and superiority of Loss-IRM in selecting real valuable preference pairs that really beneficial for current model alignment training.
LussDiff-IRM outperforms several data-centrie baselines. Table 4 reports the performance com- parisons of LossDiff-IRM with several existing data-centric methods. It can be observed that
8

---

Published as a conference paper at ICLR 2026
Table Exact TIF vs. LossDiff-IRM. Performance comparison of Pythia- 410M trained on data selected by Exact TIF or LossDiff-IRM.
UltraFedBback
AlpacaEval
Vieuna- Bench
Method
Single
WinRate
Single 1
WinRate
Single 1
WinRate
1
Pythia-410M. (DPO)
Full Data
2.81
75.25
2.77
73.51
3.10
69.37
Figure 5: Overlap Co-
Los-Diif-IRM
3.30
86.14
3.30
85.16
3.80
85.62
efficient of selections
Exact TIF
3,13
85,42
3.21
86.19
4.01
84.38
across five models.
Table 6: Performance of training DPO and SLiC on Selected vs. Dropped set by LossDiff-IRM. Comparison of training on Full Data, the LosDiff-IMM-seletee data, and the dropped data.
Training
UltraFeedback
AlpucaEval
Vicuma-Bench
UtraFeellack
AlpacaEval
Vicum-Bench
Single 1 WinRare
Singl
Winkanc
Single
WinRate
Singl:
WinRate
Single 1
WinRute
Single
WinRine
Llama-3.1-88
Owen3-S0- (DPO)
5:72
78.41
7.64
61.41
Selected Data
6.54
6,84
87.08
8J15
71.52
N.72
Drugped Duta
4,56
4,48
61.15
7.51
54.82
58.33
7.39
Llame-3.1-8
Qwen3-80-Box (SLiC)
5.09
70.72
72.13
71,88
7.55
59.54
59.71
8.08
Solecte Data
78.84
07.58
8.44
Drupped Data
4,33
59.37
7.37
56.67
55.79
8.21
56.72
LossDiff-IREE outperforms these competitors on the most cases. Specifically, the average improve- ment of WinRate achieves +2.62% over the best baselines across two models, especially achieving improvements of +6.45% on average on Llama-3.8-8B These results suggest the effectiveness of our IF-driven analysis and LossDiff-IRA data selection strategy. Additionally, compared to CurriPPO relied on external signals (GPT4 or reward model scores), MA and RS-DPO, which adopt implicit reward margins generate model-specifer preference pairs using the SFT model and thus partially incorporate the model's perspective, emerge as strong competitors. This aligns with the underlying philosophy of our idea: valuable preference data are depenede,,,,,nt, rather than relying on external heuristics. The cost analysis between LossDiff- IRM and RS-DPO is provided in Appendix E.5.
Preference data quality is model-dependent. We observe that LossDiff-IRMM outperforms selection based on Random, GPT4 score or external reward model, with only a few single-case exceptions, e.g., Qwen3-8B-Ban LossDiffIIRM. delivers average WinRate improvements of +4.07%. +3.84%, +8. 28%, +10.29% and +-8. 13% on Llama-3.11800 1-8B,, Qwen3-8B-BB
on Vicuna-BB Bench under SLiC. Under DPO, relative to the second- best result,
and Pythia-2.8BB/ 8BB.//..4///0MM,. respectively, Notably, GPT4 score selection can even reduce performance, such as on Ba-n--8--Base with DPO. Moreover, Figure shows that the Overlap Coefficient between selections varies across models; pairs within the same family (c.g.. Pythia) exhibit higher overlap than cross-family pairs. Altogether, these findings indicate that preference data quality is inherently model dependent, and static or model-agnostic selectors may benefit one model yet harm another.
LossDiff-IRM is compatible to different preference optimization methods. For both DPO and SLiC, Diff- IRM selection yields consistent and often substantial gains across diverse LLM families and architectures, indicating the generality of the LossDiff-IRM method. This compatibility stems from that the derivation and analysis of LossDiff-IRM criterion do not involve any strong assumption about certain preference optimization algorithm, so that diverse methods can be initiated under our LossDiff-IRM criterion, which serves as a plug-and-play preference data selection step.
5.2.2 FURTHER ANALYSIS
Combining Lisf and IRM outperforms either alone. Figure 3 reports WinRate for LossDiff- IRM and its two ablations: "w/ LossDiff" and "w/ IRM", which select pairs using only LossDiff or only IRM, respectively. Across both DPO and SLiC, the combined scoring function LossDiff-IRM performs better than either single variant. This matches our intent: combining the two scoring functions offset specific errors each incurs when used alone approximate TIF in data selection.
Dropped data is low-value for the current model alignment. We train DPO and SLiC using only the subset that LossDiff-IRM drops and compare against Full Data and the MosDiiff-IMMseleee

---

Published as a conference paper at ICLR 2026
AbonBall
Goeal - 1200- Boyel
0202
Toua 1200 Tayal
Tour (220 Tayal
Figure 6: Analysis of percentile thresholds Exmall: Clarge: Tsmalll Tharge of Llama-3.1-8B. We vary Esmall (100- Etagge and Tsmall (100- nargo) ) with in {0, 10, 20, 303. Further analysis of Qwen3- 8B-Base is illustrated in Figure 14 and Appendix E.6.
A4
Figure 7: Visualization of layer- wise IF value computed on Qwen3-0.ncm 0...--0..BBBBB B-Baase
subset. Table 6 shows that training "w/ Dropped Data" yields the worst performance; in particular, it reduces average WinRate by 12.59% relative to full-data training across two models and both alignment methods, and even drops to 46.88% on Aicuma-Benche with Owen3-8B-BBl (SLiC). By contrast, "w/ Selected Data" consistently improves alignment, indicating that LossDiff-IRM filters out low-value pairs and favors data that really benefits the current model alignment learning.
LossDiff exhibits certain robustness under validation-set noise. We corrupt the validation set by flipping the chosen/rejected labels at rates E {0.1, 0.2, 0.3, 0.4) and then re-run selection. Figure 4 shows the expected downward trend in performance as r increases, since LossDiff-IRM (and TIF) uses the validation set orient the score. Surprisingly, in many cases the selection upon noisy validation set still exceeds full-data training (dashed lines) under noisy validation set. We attribute this the fact that LossDiff-IRMM tends to rediu data that are less sensitive to validation noise. These results suggest the robustness of LossDiff-IRM to noisy validation set.
Analysis of percentile thresholds Esmall: Starge, Tsmall: narge The percentile thresholds of LissDiff- IRM are tuned as hyperparameters. We vary Esmall = (100- Étarge and Tsmall = (100- argoo) with in {0, 10, 20, 30). Figure illustrates the performance curves varying with different thresholds for Llama-3. 1-BB (DPO) across four benchmarks. It can be observed a rough trend: performance first improves and then degrades as the thresholds become stricter, that is, as more data are filtered out. This observation is expected, because initially, the threshold helps remove low-quality data; however, beyond a certain point, the filtering starts to exclude informative and high-quality data, leading to performance degradation. More analysis on Qwen3-883 -Bas is provided in Appendix E.6.
IF values are not confined to specific layers. Figure shows that IF may concentrate in certain layers or spread across all layers, implying that layer-wise computation alone cannot reliably approximate the exact IF and motivating the need for proxies such as our LossDiff-IRM. More visualization IF computed other LLMs are provided in Appendix E.2.
6 CONCLUSION
In this work, we propose the Truncated Influence Function (TIF) as principled lens to analyze preference data quality in LLM alignment. Unlike prior approaches that treat data quality as an inherent property of the data, our analysis adopts a model-side perspective and reveals that medium-IF pairs, rather than small- or large-IF ones, provide the most effective training signal. To make TIF practical at scale, we further introduce the LossDiff- -IM approximation, which closely matches TIF while being far more efficient. Experiments demonstrate that LossDiff DossDiifIIRM enables a "less is more" effect, where using fewer but higher-quality preference pairs yields better alignment performance.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
ZZZ, QZW, JNZ and BH were supported by RGC General Research Fund No. 12200725, RIKEN Collaborative Research Fund, and HKBU CSD Departmental Incentive Scheme. MS was supported by IST ASPIRE Grant Number JPMIAP25B1. JCY is supported by National Natural Science Foundation of China (No. 62306178) and STCSM (No. 22D22229005).
ETHICS STATEMENT
This study adheres to the Code of Ethics. It relies only on publicly available models and datasets, without the use of sensitive or privacy data, and poses no identifiable risks concerning privacy, security, or fairness. The work is conducted purely for scientific purposes, and no conflicts of interest involved.
REPRODUCIBILITY STATEMENT
We are committed to ensure the reproducibility of our proposed method. The detailed descriptions of approach and experimental settings are all provided in this paper. The corresponding source code released at https:/ / /gi thub. com/ moom/mmle smir-grouph /TIF Les___osDsfffff IRI... Both backbone models and datasets used in work publicly available. Furthermore, all parameters, hyper- parameters, and procedural steps required to reproduce our results are thoroughly recorded in the Experimental Details section and corresponding Appendix. We believe that these components provide the community with details necessary to verify and reproduce our work.
REFERENCES
Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang. Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, et al. A survey on data selection for language models. arXiv preprint arXiv:2402. 16827, 2024.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference Artificial Intelligence and Statistics, pp. 4447-4455. PMLR, 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204. 05862, 2022a.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.00 058622, 2022b.
Stella Biderman, Hailey Schoelkopf. Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-355, 1952.
Tianchi Cai, Xierui Song, Jiyan Jiang. Fei Teng. Jinjie Gu, and Guannan Zhang. Ulma: Unified language model alignment with human demonstration and point-wise preference. arXiv preprint arXiv:2312.02554, 2023.
Liang Chen, Xueting Han, Qizhou Wang, Bo Han, Jing Bai, Hinrich Schutze, and Kam-Fai Wong. Eepo: Exploration-en enhannced policy optimization via sample-then-fore forget. In The Fourteenth International Conference Learning Representations, 2026.
11

---

Published as a conference paper at ICLR 2026
Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, et al. Alpagasus: Training better alpaca with fewer data. In The Twelfth International Conference on Learning Representations, 2024.
Wei Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. Imsys. org (accessed 14 April 2023), 2(3):6, 2023.
Israel Cohen, Yiteng Huang, Jingdong Chen, Jacob Benesty, Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen. Pearson correlation coefficient. Noise reduction in speech processing, pp. 1-4, 2009.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, et al. Ultrafeedbcck: Boosting language models with scaled ai feedback. arXiv preprint arXiv:2310.01377 2023.
Xun Deng, Han Zhong, Rui Ai, Fuli Feng, Zheng Wang, and Xiangnan He. Less is more: Improving Ilm alignment via preference data selection. arXiv preprint arXiv:2502.14500 14560, 2025.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 0029-3051, 2023.
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, (6:30039-30069 2023.
André Elisseeff, Massimiliano Pontil, et al. Leave-one-out error and stability of learning algorithms with applications. NATO science series sub series iil computer and systems sciences, 190: 1are11-1110 1-100, 2003.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306 2024.
Theodoros Evgeniou, Massimiliano Pontil, and André Elisseeff. Leave one out error, stability, and generalization of voting combinations of classifiers. Machine learning,
Chengqian Gao, Haonan Li, Liu Liu, Zeke Xie, Peilin Zhao, and Zhiqiang Xu. Principled data selection for alignment: The hidden risks of difficult examples. arXiv preprint 0rX6v25502.09650 2025.
Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, et al. Studying large language model generalization with influence functions. arXiv preprint arXiv:2308. 03296, 2023.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in 1lms via reinforcement learning, arXiv preprint arXiv:2501. 2025.
Frank R Hampel. The influence curve and its role in robust estimation. Journal of the american statistical association, 69(346):383-333, 1974.
Jan Hauke and Tomasz Kossowski. Comparison of values of pearson's 's and spearman's 's correlation coefficients on the same sets of data. Quaestiones geographicue, 30(2):87-93, 2011.
Kexin Huang, Junkang Wu, Ziqian Chen, Xue Wang, Jinyang Gao, Bolin Ding, Jiancan Wu, Xiangnan He, and Xiang Wang. Larger or smaller reward margins to select preferences for Ilm alignment? In Forty-second International Conference Machine Learning, 2025.
12

---

Published as a conference paper at ICLR 2026
Saeed Khaki, JinJin Li, Lan Ma, Liu Yang. and Prathap Ramachandra. Rs-dpo: A hybrid rejection sampling and direct preference optimization method for alignment of large language models. In Findings of the Association for Computational Linguistics: NAACL 2024, pp. 1665-1680, 2024.
Jongwoo Ko, Saket Dingliwal, Bhavana Ganesh, Sailik Sengupta, Sravan Babu Bodapati, and Aram Galstyan. Sera: Self-reviewing and alignment of Ilms using implicit reward margins. In The Thirteenth International Conference on Learning Representations, 2024.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference machine learning, pp. 1885-1894. PMLR, 2017.
Keyi Kong, Xilie Xu, Di Wang, Jingfeng Zhang, and Mohan S Kankanhalli. Perplexity-wware
correction for robust alignment with noisy preferences. Advances in Neural Information Processing Systems, 37:28296-28321, 2024.
Andreas Köpf, Yannic Kilcher, Dimitri Von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Richard Nagyfi, et al. Openassistant conversations
converations--dmmoratiii democratizing large language model alignment. Advances in Neural Information Processing Systems, 11447669-4768.. 2023.
Yongchan Kwon, Eric Wu, Kevin Wu, and James Zou. Datainf: Efficiently estimating data influence in lora-tuned Iims and diffusion models. In The Twelfth International Conference on Learning Representations, 2024.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, IJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403. 13787, 2024.
Kemou Li, Qizhou Wang, Yue Wang, Fengpeng Li, Jun Liu, Bo Han, and Jiantao Zhou. Llm unlearn- ing with Ilm beliefs. In The Fourteenth International Conference on Learming Representations, 2026.
Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406. 1939, 2024.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacaeval: An automatic evaluator of instruction-folloving models, 2023.
Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. In The Twelfth International Conference on Learning Representations, 2024.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference- free reward. Advances in Neural Information Processing Systems, 57:124198-124235 2024.
Yuchun Miao, Sen Zhang, Liang Ding, Rong Bao, Lefei Zhang, and Dacheng Tao. Inform: Mitigating reward hacking in rlhf via information-theoreti reward modeling. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Tetsuro Morimura, Mitsuki Sakamoto, Yuu Jinnai, Kenshi Abe, and Kaito Ariu. Filtered direct preference optimization. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 22729-22770, 2024.
William Muldrew, Peter Hayes, Mingtian Zhang, and David Barber. Active preference learning for large language models. International Conference on Machine Learning, pp. 36577-36590. PMLR, 2024.
Koichi Nagatsuka, Clifford -edii-Bediako, and Masayasu Atsumi. Length- based curriculum learning for efficient pre-training of language models. New Generation Computing, 41(1):109-13) 2023.
13

---

Published as a conference paper at ICLR 2026
Jinlong Pang, Na Di, Zhaowei Zhu, Jiaheng Wei, Hao Cheng, Chen Qian, and Yang Liu. Token clean- ing: Fine-grained data selection for Ilm supervised fine-tuning. arXiv preprint arXiv:2502.0196. 2025.
Pulkit Pattnaik, Rishabh Maheshwary, Kelechi Ogueji, Vikas Yadav, and Sathwik Tejaswi Madhusud- han. Enhancing alignment using curriculum learning & ranked preferences. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 12891-12907) 2024.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon IIm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.0/ 2023.
Yulei Qin, Yuncheng Yang. Pengcheng Guo, Gang Li, Hang Shao, Yuchen Shi, Zihan Xu, Yun Gu, Ke Li, and Xing Sun. Unleashing the power of data tsunami: comprehensive survey on data assessment and selection for instruction tuning of language models. Transactions on Machine Learning Research, 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances 2023.
in Neural Information Processing Systems,
Gayathri Saranathan, Mahammad Parwez Alam, James Lim, Suparna Bhattacharya, Soon Yee Wong, Martin Foltin, and Cong Xu. Dele: Data efficient Ilm evaluation. In ICLR 2024 Workshop on Navigating and Addressing Data Problems for Foundation Models, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint
2017.
Judy Hanwen Shen, Archit Sharma, and Jun Qin. Towards data-centric rlhf: Simple metrics for preference dataset comparison. arXiv preprint arXiv:2409.09603 2024.
Yunyi Shen, Hao Sun, and Jean-Francor Ton. Active reward modeling: Adaptive preference labeling for large language model alignment. In Forty-second International Conference on Machine Learning, 2025.
Liping Tang, Nikhil Ranjan, Omkar Pangarkar, Xuezhi Liang, Zhen Wang, Li An, Bhaskar Rao, Linghao Jin, Huijuan Wang, Zhoujun Cheng, et al. Txt360: A top-quality I1M pre-training dataset requires the perfect blend, 2024.
Lewis Tunstall, Edward Emanuel Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro Von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of Im alignment. In First Conference on Language Modeling. 2024.
Raja Vavekanand and Kira Sam. Llama 3.1: An in-depth analysis of the next-generation large language model. Preprint, July, 2024.
Qizhou Wang, Jin Peng Zhou, Zhanke Zhou, Saebyeol Shin, Bo Han, and Kilian Q Weinberger. Rethinking Ilm unlearning objectives: A gradient perspective and go beyond. In The Thirteenth International Conference on Learning Representations, 2025a
Yue Wang, Qizhou Wang, Zizhuo Zhang, Ang Li, Gang Niu, Bo Han, and Masashi Sugiyama. What is preference optimization doing, how and why? Arxiv Preprint, 2025b.
Zhichao Wang. Bin Bi, Shiva Kumar Pentyala, Kiran Ramnath, Sougata Chaudhuri, Shubham Mehrotra, Xiang- Bo Mao, Sitaram Asur, et al. A comprehensive survey of Ilm alignment techniques: Rlhf, rlaif, ppo, dpo and more. arXiv preprint arXiv:2407.16216 2024.
Junkang Wu, Yuexiang Xie, Zhengyi Yang. Jiancan Wu, Jiawei Chen, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. Towards robust alignment of language models: Distributionally robustifying direct preference optimization, arXiv preprint arXiv:2407.0788 00780, 2024a.
14

---

Published as a conference paper at ICLR 2026
Junkang Wu, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. beta-dpo: Direct preference optimization with dynamic beta. Advances in Neural Information Processing Systems, 37: 17:129944-12996 2024b.
Shengguang Wu, Keming Lu, Benfeng Xu, Junyang Lin, Qi Su, and Chang Zhou. Self-evolved diverse data sampling for efficient instruction tuning. arXiv preprint arXiv:2311.08182, 2023.
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning. In International Conference on Machine Learning, pp. 54104-54111 54132. PMLR, 2024.
Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy S Liang. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems, 36: 34201-34227, 2023.
Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: pushing the boundaries of Ilm performance in machine translation. In Proceedings of the 41st International Conference
Machine Learning, pp. 55204-55224, 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint ar arXiv:2505.09388, 2025.
Chi Zhang, Huaping Zhong, Kuan Zhang. Chengliang Chai, Rui Wang, Xinlin Zhuang, Tianyi Bai, Jiantao Qiu, Lei Cao, Ju Fan, et al. Hamnsssing diversity for important data selection in pretraining large language models. arXiv preprint arXiv:2409. 16986, 2024a.
Jiazheng Zhang, Wenqing Jing, Zizhuo Zhang, Zhiheng Xi, Shihan Dou, Rongxiang Weng, Jiahuan Li, Jingang Wang, Mingxu Chai, Shibo Hong, Tao Gui, and Qi Zhang. Two minds better than one: Collaborative reward modeling for Ilm alignment. arXiv preprint arXiv:2505. 10597, 2025.
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.02388 2024b.
Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, al. Instruction tuning for large language models: A survey, arXiv preprint ar arXiv:2308. 10792, 2023.
Zizhuo Zhang. Jianing Zhu, Xinmu Ge, Zihua Zhao. Zhanke Zhou, Xuan Li, Xiao Feng, Jiangchao Yao, and Bo Han. Co-rewarding: Stable self- supervised rl for eliciting reasoming in large language models. In The Fourteenth International Conference on Learning Representations, 2026.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305. 10425, 2023.
Lianmin Zheng, Win-Lin Chiang. Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li. Eric Xing, et al. Judging llm-as-a-judge with mt-bennh and chatbot arena. Advances in Neural Information Processing Systems, 46626696566663,,, 2023a.
Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Yuhao Zhou, Limao Xiong, et al. Delve into ppo: Implementation matters for stable rlhf. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023b.
Zhanke Zhou, Rong Tao, Jianing Zhu, Yiwen Luo, Zengmao Wang, and Bo Han. Can language models perform robust reasoning in chain-of- of-hh-ugttth prompting with noisy rationales? In NeurIPS, 2024.
Zhanke Zhou, Xiao Feng, Zhaocheng Zhu, Jiangehao Yao, Sanmi Koyejo, and Bo Han. From passive to active reasoning: Can large language models ask the right questions under incomplete information? In ICML, 2025a.
Zhanke Zhou, Zhaocheng Zhu, Xuan Li. Mikhail Galkin, Xiao Feng, Sanmi Koyejo, Jian Tang, and Bo Han. Landscape of thoughts: Visualizing the reasoning process of large language models. arXiv preprint ar arXiv:2503.22 265,, 2025b.
15

---

Published as a conference paper at ICLR 2026
LLM USAGE STATEMENT
Here we clarify how Large Language Models (LLMs) are used in this work. In preparing the manuscript, LLMs served only as a writing assistants for writing improvements and were not involved in research ideation or the generation of core content. For research methodology, LLM is a core component of our proposed method. Specifically, we utilize the Llama-3.1-88 Qwen3- 8B-Base and Pythia series as our backbone models to validate our proposed method.
A BACKGROUND AND RELATED WORK
A.1 SUPPLEMENT TO PRELIMINARY
Reinforcement Learning with Human Feedback (RLHF). RLHF aims to align the given policy model To with human preference by optimizing the model to maximize the expected reward value obtained from the reward model (Zhang et al., 2025). The reward model is trained on a preference dataset, quantifying the human preference into scalar values. Typically, the Bradley-Tene -erry (BT) model (Bradley& & Terry, 1952) is used to estimate the probability distribution that chosen response yw is preferred over a rejected one y/: as follows:
exp (r(z, Mw))
pwu > vı(=) = exp (r(z, ym.)) + exp (r(z, vt) = (r((,, Bwm r(x, 9(),
(10)
where r(x, y) denotes the latent reward function, and ((.) is the Sigmoid function. Due to unobserv- ability of the reward function, the traditional RLHF typically follows two-stage pipeline: a reward
model ra (Ir, y) is pretrained on preference data; then, the policy model is updated based on the pretrained reward model using reinforcement learning (RI.) algorithm like PPO (Schulman et al., 2017). The overall objective of RLHHE can be formulated as follows:
CRLMP D) = EE yost((- D.-DDy--att- [r v)] + (vo|
(11)
where [-Il-11-] is the KI.-divergence regulation term that constrains the policy model mo to optimize within the surrounding landscape of the reference model Tef avoiding policy collapse or training instablity during alignment, and is a hyperparameter to control trade-off between reward maximiza- tion and KL penalty. The reference model is often the supervised fine-tuned (SFT) model (Zhang et al., 2023) used initialize the policy LLM.
A.2 RELATED WORK
LLM Preference Alignment. LLM has achieved remarkable success across a wide range of tasks (Li et al., 2026; Chen et al., 2026). LLM preference alignment aims steer LLM behaviors toward responses that better reflect human preferences and values (Wang et al., 2024). A common approach RLHF that trains a reward model to provide reward signals for RL training (Bai et al., 2022b; Zhou et al., 2025a;b). Though effective, the two-stage RLHF pipeline is complex and resource-intensive, suffering from issues like reward hacking (Miao et al., 2024) and unstable optimization (Zheng et al., 2023b; Zhang al., 2026). To address these limitations, recent studies (Rafailov et al., 2023; Azar et al., 2024; Zhao et al., 2023; Meng et al., 2024; Ethayarajh et al., 2024) have proposed multiple alignment learning objectives that bypass explicit reward modeling, offering a simpler formulation for direct LLM preference alignment, such as DPO (Rafailov et al., 2023), IPO (Azar et al., 2024), SLiC (Zhao al., 2023; Liu al., 2024), SimPO (Meng al., 2024), KTO (Ethayarajh et al., 2024), CPO (Xu et al., 2024) and more (Wu et al., 2024b;a; Liu et al., 2024). Among them, DPO serves as a milestone work that derives the close-form expression of the optimal policy and substitutes into the Bradley-Terry (BT) model, effectively hiding reward learning within the policy optimization process. SLiC adopts hinge loss to enlarge the margin between the chosen and rejected responses. SimPO simplifies the DPO formulation and achieves reference model-free optimization. KTO employ prospect theory to directly maximizes the utility of generations rather than log- likelloooood of preferences. Overall, while many efforts have been devoted to improving alignment algorithms, relatively little attention (Deng et al., 2025; Gao et al., 2025) has been given to understanding the preference data quality, especially from model perspective.
Data Selection for LLMs. Data selection (Albalak et al., 2024) aims to filter out low-quality or noisy data (Zhou et al., 2024) and retain high-quality data for better training and generalization,
16

---

Published as a conference paper at ICLR 2026
which plays pivotal role in enhancing the performance and efficiency of LLMs across difference training stages, including pretraining (Penedo et al., 2023; Tang et al., 2024), supervised fine- tuning (SFT) (Pang et al., 2025; Qin et al., 2024), and preference alignment (Shen et al., 2024; Gao et al., 2025). Most existing approaches assess data from multiple considerations, such as importance (Xie et al., 2023) and diversity (Zhang et al., 2024a), using variety of metrics such as text length (Nagatsuka et al., 2023), perplexity (Kong et al., 2024), text embeddings (Saranathan al., 2024), external signals from humans (Wu et al., 2023) or ChatGPT (Chen al., 2024). We notice that most studies for LLM data selection primarily focus on the first two stages, i.e., pretraining and SFT, whereas the selection of preference data for alignment has received comparatively limited exploration. Existing studies (Pattnaik et al., 2024; Morimura al., 2024; Deng al., 2025) often employ off-the-shelf LLMS or pretrained reward models to pre-process preference data. For example, CurriPPO (Pattnaik et al., 2024) leverages the GPT-4 or reward model scores to organize curriculum learning. fDPO (Morimura et al., 2024) integrates a reward model into the DPO training process, filtering preference data based reward scores. Muldrew et al. (2024) and Shen et al. (2025) incorporate active learning into alignment to improve data quality and annotation efficiency. Overall, these methods share common trait: they rely external signals- whether from powerful GPT models, pretrained reward models, or active human labeling-and thereby treat data quality as an inherent property of the data itself, while overlooking the role of the model and training objectives. Motivated by this limitation, we examine preference data quality from the model's perspective to better understand which data are truly valuable for alignment.
B FORMAL DERIVATION
B. FORMULATIONS SLIC LOsS
Definition of SLiC Loss. Unlike DPO loss in Eq. (I), which optimizes log-ratio difference between chosen and rejected responses, SLiC (Zhao et al., 2023; Liu et al., 2024) minimizes a hinge loss on normalized log- oogliikllhoood, formulated as follows:
To (1oll))) Lsii (1.. max (0. (Blog
Ta (3h [z)
B log
(12)
Tref you (z)
Tie (1I (x)
Intuitively, SLiC loss enforces at least fixed margin between the chosen and rejected responses. When the margin is satisfied, the loss becomes zero, making the optimization focus on those pairs that violate the margin constraint.
Instantiation of SLiC Loss for Influence Score. By taking SLiC loss into the influence function formulation in Eq. (5), the SLiC-based influence function is derived as follows:
IFsic (d; no; Deat) = I
|Dail
Uninge
9t
(13)
preference pair direction
preference generalization direction validation set
where
(o (x) log
To To (x)
Ininge (d) I[1 Ao 0] and Ag 3 log
(14)
13rr(W/l (z)
Tee
y1
Both DPO and SLiC assign higher IF values preference pairs whose gradient difference direction (i.e., 9he g)) is consistent with that of the validation preferences, and negative influence scores when they oppose. In DPO, the IF is further scaled by 1 a (Aa), assigning near-zero scores pairs with large Ag. i.c., those already well learned by the model. Similarly. SLiC sets the IF to exactly zero for saargn--atisficn pairs (Ag 1). These well learned pairs naturally fall into the medium-If region, and are also considered as high-qualit, data that promote preference generalization.
B.2 RELATIONSHIP BETWEEN INFLUENCE INFL.NCTION AND LOSSDIFF
To analyze the positive correlation between influence function defined in Eq. (4) and LossDiff defined in Eq. (8), provide formal justification as follows:
17

---

Published as a conference paper at ICLR 2026
Lemma B.1 (Loss Difference (LossDiff) Correlates with Influence Function (IF)). Assume that the validation aliggned model New is obtained by performing a single gradient descent step from the current model the loss, i.e. r 0 o Dual). where ni the learning rate. For give training preference pair d, the loss difference and the influence function are positively correlated:
LossDiffi di ne, Te ((B: E(Ovat; d) x I [d:; ne; Dvat);
(15)
where [(d; Ro; Dmmt) is the influence function defined in Eq. (5).
Formal Derivation. By applying first-order Taylor expansion of the loss E(0; d) at 0;, we get:
((Oval; d) Vol(Od)) (Oval -0..
(16)
Then, rearranging terms gets:
E(#:d) E(O d) =
00..
(17)
Now assume 0eal - (0; vval), where > is the learning rate. Substituting this into the equation yields:
E(0: d) d(Oval: = n Vol(0;d) VoL(0; (V(()) Dra) & Vol(0; d) VoL Pva) = I(d: To: Dva)).
(18)
(19)
(20)
Hence, the LossDiff defined Eq. (8) positively correlated to the IF defined Eq. (4).
The formal justification assumes that the validation-aligned model To is obtained by performing a single gradient descent step from the current model тa on the alignment loss computed over the validation set. While this assumption does not precisely reflect the actual training procedure, it serves as a reasonable local approximation. In preference alignment methods, they all explicitly implicitly have KL divergence term in preference optimization objectives (Bai al., 2022b; Rafailov et al., 2023). The KL regularization term constrains both the Te and the Te.,, to remain close the reference model in parameter space. Since both 0 and Oun lie in small neighborhood around the same reference model, it reasonable to assume that Pual can be approximated by single gradient descent step from on the preference loss. This justifies the assumption used in Lemma B.
B.3 FORMAL ANALYSIS OF THE RELIABILITY OF MEDIUM-IF DATA SELECTION
We demonstrate that the reliability of selecting high-IF data decreases as the quality of the validation data decreases. We provide the derivation pipeline as follows:
Suppose the validation dataset is corrupted w.p. 77. Then, the expected validation gradient consists of the clean (c) and noisy (n) parts, following Is = (1 n)ge + 19n. For a training sample i with the gradient gi. we have influence weight as
W: 9. 9i (1 n)g. 9:
19.. 9s.
(21)
which can be further written as w + 8,, with wt the clean weight and 8; = -n ge) 9i the bias term. Assuming that gn - 9c is independent on g., with zero mean and isotropic variance o I, then have Var(6;) |1:11 2 and Var(w;) = o + r², with of Var(w,) and 7² Var(6:). Accordingly, we have the expected correlation between wt and w as
Cov(w) wwi)
/ Vollo +
(22)
Var(( Var(w:)
As observed, when the corruption rate increases, the correlation p between the true and observed scores decreases, leading to lower probability p(i Tume (k))..)). Hence, the top-k sampling strategy becomes less reliable under high noise levels. Similar derivations hold for bottom-k sampling, making middle-k sampling tend to be more reliable.
18

---

Published as a conference paper at ICLR 2026
Quee
Wir plo
120
119
Marge
20
180
159
Figure 8: Analysis of based data partition with overlapping splits on Qwen3-0.6B-Bas and Llama-3.2-1B, From Left to Right: Trainin DPO loss, evaluation DPO loss and evaluation reward margin. Subsets of (small, medium, 100g are denoted by fblue, orange, green], respectively, while different line styles indicate IF values computed using different epoch checkpoints. Previous analysis refers to Section 3.2 and Figure 1.
Table 7: Further Overlap Coefficient analysis on (3.cc- DPO checkpoints to assess the overlap between two selected sets.
Overlap Curfficient
Lundifff IF
IRM IF
Losdiff-IRM vs.
Upoch ckge
Upoct dopt
Opech copt
lipcch 3 cigt
lpocc ckge
Upoch 5 dkyc
3
4 clop
Upoch dape
Owen3 0.68 Buse
0U6193
D.6345 8.6299
066400 0,2922
0.6153 0,5645
0.5985 0.5112
0.6000 0.5199
0,4977 0,6657
0.6770 0,6643
0.6747 0,6673
Llama 3.2
0.6319
C ADDITIONAL ANALYSIS
C.I MORE ANALYSIS OF IF ON QWEN3- -B. 6B-BASE AND LLAMA-3.2-1B
As a supplement to Sec 3.2, we investigate whether introducing more medium-IF data into the small-IF and large-IF subsets can mitigate the adverse effects of extreme IF values. To this end, adopt overlapping splits, dividing the data into large-60%, medium-60%, and small-60% subsets based on their IF scores, where the small-60% and large-60% subsets partially overlap with the medium-60% subset. Figure 8 shows the training dynamics of such overlapping data partition on Qwen3-0.6B- Base and Llama-3.2. 3.2- BB. We observe similar phenomena shown as non-overlapping splits in Figure 1: small-IF data continues to be uninformative and large-IF IF data still drives overfitting. This suggests that extremely small- or large-IF pairs are detrimental, as even introducing smoother overlaps fails to mitigate their negative effects.
C.2 MORE ANALYSIS OF CORRELATION
As supplement to Figure 2 and Table 2, which analyze the 1poch-1,2 checkpoints of Qwen3-0.6B Base and Llama-3.2211: B, Figure and Table present the results for the remaining epoch-3,4.5 checkpoints. We observe that as training progresses. the correlation between IRM and IF decreases more sharply than that between LossDiff and IF. For example, at epoch-5 of Llama-3.2-1B3 the Pearson coefficient between IRM and IF drops to 0.0636. Nevertheless, the Overlap Coefficients of LossDiff IRM remain consistently above 0.65, indicating a relatively high level of agreement with the exact based selection and exceeding those of LossDiff or IRM alone. This supports design choice: the errors of LossDiff and IRM in approximating IF are diverse and may partially offset each other, so combining them yields selection that more closely matches the exact IF-based criterion.
19

---

Published as a conference paper at ICLR 2026
(Qwo334AA Epoch
Egoot
(Qweed Egust
Loes Difforrze atw.orrzeee..ooooooooo (LAKCH))
Diforrte
Differecee
P
Purmar
Ogvermie 8
Aiflienoe (IF)
Furetion
luclicace (IF)
(Qweez Egoth
(Qweed Egoch 4)
(Qwea? Egecb
Stgllahh Rowand Magic (1RM)
Ieplict Rooand Magic (TAM))
Iro Roward
83543
Speurein 1-10000
liflienoe (EF)
Furetion
luclucace (IF)
LeudDiff (Llum-332-18 Epsch
(L.mmm3.2-1B Epich-6)
(Lhmms332-1B
Low.Dineoeeesaa (LoeDudh
6.0
Loss Differooe (LOWEENCT
4.30
Differsse
-100
2.27
-100
E751
Farction
luclucnce
(Lbmms33211
Ingiich Bo Magih (RO0)
Imspickk Denund Magii (RA3))
Imsphem Mergi
San 0.90
Sponrale p 8.67)
Figure 9: Further correlation analysis (p0ch-(3,4,55) checkpoints of Qwen3-0.6B-Baan and Llama-3.2-1B. First Row: correlation between Lisf and IF on Qwen3-0.6B Base. Second Row: correlation between IRM and IF on Qwen3-0.6% Base. Third Row: correlation between LossDiff and IF on Llama-3.2-11. Fourth Row: correlation between IRM and IF on 1.ama-3.2-11
D MORE EXPERIMENTAL DETAILS
D.1 DATASET DETAILS
The details of the datasets used in this work is introduced follows:
20

---

Published as a conference paper at ICLR 2026
Table Statistics of the datasets used in this work.
Instances
Dataset
Purpose
Unit
Train
Val
Eval
UltraChat-200K
SFT
207,866 48,908
23,110 10
Dialogues
UltraFeedback- -iinariineedd
Pref. Train
12,227
Pairs
UltraFeedback Test set
Evaluation
1,000
Pairs
AlpacaEval
Evaluation
805
Prompts
accun--Bench Arena-Hart
Evaluation
80
Prompts
Evaluation
500
Prompts
UltraChat-200K' (Tunstall et al., 2024): This is a heavily filtered subset version of Ultra- Chat (Ding al., 2023), which was originally used for training ZePhyr-7B--3 model. The filtering process removes dialogues containing grammatical errors or assistant responses with phrases such "I do not have emotions" or "I don't have opinions." After filtering, training set of UltraChat- 200K contains 207,865 multi-turn dialogues generated by ChatGPT, covering wide range of topics. Currently, it is widely adopted for supervised fine-tuning (SFT) of LLMs in research community (Ko et al., 2024; Zhang et al., 2024b). We also use this dataset to perform SFT on LLMs as preparation for subsequent preference alignment.
BrraFeeedDak-BBnniiieedd (Cui et al., 2023): This is preprocessed pairwise version of the UltraFeedback dataset, designed for LLM preference alignment. The dataset contains 64k prompts collected from diverse sources. For each prompt, four responses are generated by different LLMs and then evaluated by GPT-4 along four axes: fostuutionnffolooiingg truthfulness, honesty, and helpfulness. To construct preference pairs of the UltraFeedback DicrrFeeddbac-BBiaazeee,,,, the response with the highest overall score is selected as the "chosen" response, while one of the remaining three is randomly selected the "rejected" response. We perform stratified sampling based on the GPT-4 score difference between chosen and rejected responses, holding out 20% of the training split validation set, using the remaining 80% as our full training set in this work. The resulting dataset contains 48, .098 training pairs and 12,227 validation pairs.
UltraFeedback Test Set (Cui al., 2023): This Binarized dataset and constructed using the same preprocessing procedure. It contains 2,000 high-quality preference pairs. To reduce the cost of LL.M-based evaluation in our experiments, we randomly sample 1,000 pairs from this set to form the UltraFeedback evaluation dataset used in this paper. To reduce the cost LLM-as-a-Judge (Zheng et al., 2023a) evaluation, we randomly sample 1,000 pairs from this set to construct the UltraFeedback benchmark used in this work.
is provided alongside the UltraFeedback-
AlpacaEval (Li et al., 2023): This is a lightly modified version of the AlpacaFarm (Dubois et al., 2023) evaluation set, containing 805 challenging prompts spanning wide range of topics. Following previous studies (Ko et al., 2024; Gao et al., 2025), we employ AlpacaEval to evaluate the instruction-followe followwng capability of the trained L.MSs in this paper.
Vicuna-Bench (Chiang et al., 2023): This is a dataset containing 80 diverse questions originally used evaluate the Vicuna series of LLMs. Following previous work (Pattnaik al., 2024; Ko et al., 2024), we adopt it to evaluate the open-ended irestion-answerigg ability of the model.
Area-Hard (Li et al., 2024): This is a benchmark containing 500 challenging prompts curated by BenchBuilder, which utilizes LLM judges to estimate human preferences.
The statistics information above datasets used in this paper is summarized in Table 8.
'ntts://huggingffcc..c hwt.r//ggw...ittuoooooo/ninnnnnn
hhtp::/hhnggnngiooooooaoeeeeelee ooditesss//iiigggggggoooihiiaodd binarized
bhttss//huggingfacee.ou
"https//huggingffcccco htw.:hgggggnggngooaaaec///nsaaaa eval
21

---

Published as a conference paper at ICLR 2026
Table 9: Training Setup Details.
Stage
Hyperparameter
Llama-3, 3,1-8B
Qwen3-6r Base
Pythia-410M Pythia
Pythia-2.88
Leurning rate
Optimizer
AdamW
SFT
Scheduler
Cosine
Epoch
Batch Size
64
Gradient accumulations
2
Learning rate
2c-4
5e-7
Optimizer Scheduler Epoch
AdamW Linear
Batch Size
32
Alignment
Gradient uccumulations
16
8
0.1 (DPO) 0.1 (SLIC)
TLORA
OLORA
32
32
Please act as an impartial judge and evaluate the quality of the response provided by an Al assistant to the user question displayed below. Your evaluation should consider factors such as the helpfulness, relevance, accuracy, depth, creativity, and level of detail of the response. Begin your evaluation by providing a short explanation. Be as objective possible. After providing your explanation, please finally rate the response on a scale of to 10 by strictly following this format: "[[rating]]". for example: "Rating: [[5]]".
[Question] {question]
[The Start of Assistant's Answer] fresponse]
[The End of Assistant's Answer]
Figure 10: Pointwise single-answer grading prompt to compute metric of "Single".
D.2 TRAINING DETAILS
We conduct experiments using various LLM families, including Llama-3.1-88 (Vavekanand & Sam, 2024), Owen3-8B-Basd (Yang al., 2025) and the Pythia series (Pythia-2.88 8B/1 .4B/410M) (Biderman et al., 2023). Due to GPU limit, Ilama-3. 1-8B and Qwen3-8B BB-Base are trained using LoRA with TLORA = 32, aLoRA 32 and drop_out = 0.05, whereas the Pythia models are trained full- parameter setting. As described in the previous section, first perform one epoch of SFT on each model based on UltraChat-2000 dataset to initialize the alignment learning.
We then apply two preference aligmment algorithms, i.e, DPO and SLiC, validate the effectiveness of our proposed LossDiff- D... For both DPO and SLiC, we train for two epochs with the Adam W optimizer and linear learning rate scheduler. The batch sizes and gradient accumulation steps set to {16, 16, 32, 8, 2} and [1, 1, 1, 4, 16) for the five models, respectively. The hyperparameter of KI. pernalty 3 in DPO and SLiC is set 0.1 for both DPO and SLiC following previous work (Ko et al., 2024). All experiments are conducted using bfloat 6 dtype in our experiments. All experiments are conducted
NVIDIA H100-80GB GPU using the Hugging Face TRL library. All training hyperparameters refer to Table
22

---

Published as a conference paper at ICLR 2026
Please act as an impartial judge and evaluate the quality of the responses provided by two Al assistants to the question displayed below. You should choose the assistant that follows the user's instructions and answers the user's question better. Your evaluation should consider factors such as the helpfulness, relevance, accuracy, depth, creativity, and level of detail of their responses. Begin your evaluation by comparing the two responses and provide short explanation. Avoid any position biases and ensure that the order in which the responses presented does not influence your decision. Do not allow the length of the responses to influence your evaluation. Do not favor certain names of the assistants. Be as objective as possible, After providing your explanation, output your final verdict by strictly following this format: "[[A]]" if assistant A better, "[[B]]" if assistant B is better, and "[[C]]" for tie.
[User Question] {question]
[The Start of Assistant A's Answer]
(response_at a] [The End of Assistant A's Answer]
[The Start of Assistant B's Answer] {response_ bb [The End of Assistant B's Answer]
Figure 11: Pairwise comparison prompt to compute metric of "WinRate vs. SFT".
D.3 EVALUATION DETAILS
To evaluate the aligned LLMs, we use the vilm? library to accelerate inference and generate responses with sampling temperature set to 1.0, top-p 0.95, and a maximum generation length 512 tokens to control the generation length. For LLM-as-a-judke evaluation, we adopt the open-soure Qwen3-32B to
adopt GLM-4-Plus Pythia
experiments, while for Llama-3.1-8B and Qwen3-8B-BB reduce API token costs. We adopt types of prompting strategies: (1) pointwise single-answer grading prompt, which yields the Single Score, and (2) a pairwise comparison prompt, which produces the Win Rate vs. SFT. The temperature of the judge is set to 0.0 for both two metrics. The prompt templates for both evaluation metrics are shown in Fig. 10 and Fig. 11. The WinRate is computed by assigning a weight of wins, 0.5 to ties, and 0 losses. Formally:
Thwin + 0.5 X Phie
WinRate
(23)
12win Ruie + Tipos
where Thwin Thies and Moss denote the number of wins, ties, and losses, respectively. To mitigate position bias, we repeat the pairwise evaluation with swapped answer orders and report the averaged Win Rate. For the GPT-4 and RM baselines, high-quality preference pairs are selected based on GPT-4 score differences from UltraFeedback and reward differences computed by the OpenAssistant reward model", respectively.
We compare our LossDiff-IRA with several data-centric preference alignment methods, including:
CurriDPO (Pattnaik et al., 2024) a curriculum learning- based method, which orders preference pairs to organize curriculum learning based on various creteria, such as GPT4 score (CurriDPO- GPT4) and reward model score (CurriDOO-Ree Model).
"http://docss. http://ddoc..lll...aa. www.ssswlmmmmmii...... ai/en/latest/ / 'htps://higmodell.c/
23

---

Published as a conference paper at ICLR 2026
Figure 12: Ablation study on Single Score. Comparison of LossDiff-IRA and its ablated variants that select data relying solely on LossDiff ("w/ LossDiff") or solely on IRM ("w/ IRM"). Top: Training with DPO. Botton: Training with SLiC. The ablation of WinRate is provided in Figure 3.
Table 10: Concrete performance of ablation study of LossDifl-IRM with DPO, which is corre- sponding to Figure and Figure 12.
Dataset Ratio
UltraFeedback
AlpacaEval
Vieuna-Bench
LLM
DPO
Single
WinRate
Single
WinRate
Single
WinRate
LossDiff-IRM
64%
6.54
83.97
6.84
87.08
86.88
Llama-3.1-8B
w/ Diff
80%
6.34
82.42
6.49
84.36
79.69
w/ IRM
80%
5.72
76.92
5.80
77.25
5.97
76.88
LossDiff-IRM.
64%
8.05
67.32
8.36
71.52
67.19
Qwen3-8B-Base
w/ LossDiff
80%
7.89
64.71
8.26
68.10
65.62
w/ IRM
80%
7.61
61.73
7.94
63.88
8.22
59.38
LossDiff-IRM
4.90
79.62
5.30
76.03
5.74
82.50
Pythia-2.8B
w/ Liff
4.74
76.83
5.22
72.44
81.01
w/ IRM
80%
4.85
75.40
5.12
72.38
5.64
73.12
Loss IRM
4.23
78.49
4.49
76.43
76.88
Pythia-1.4B
w/ Lifs
80%
4.09
75.20
4.48
75.75
4.99
75.62
w/
80%
4.06
74.22
4.35
74.41
72.50
Loss IRM
64%
3.30
86.14
3.30
85.16
85.62
Pythia-410M
w/ Lifs
80%
3.07
80.64
3.11
82.21
3.54
80.63
w/ IRM
70%
3.11
82.58
3.17
82.96
3.73
85.00
MAP (Huang et al., 2025) is margin-based preference data selection metric, called alignment potential, which integrates both explicit and implicit reward margins to quantify preference data.
RS-DPO (Khaki et al., 2024) combines rejection sampling (RS) with DPO to generate high- quality preference data for DPO training. RS-DPO requires an external reward model to quantify preference pairs during rejection sampling.
E ADDITIONAL EXPERIMENTAL RESULTS
E. ABLATION STUDY: SINGLE SCORE
As a supplement to Figure 3, Figure 12 reports the Single Score for LossDiff-IRM and its two ablated variants: "w/ LossDiff" and "w/ IRM", which select data depending on LossDiff or IRM alone. Similar to the Win Rate results in Figure 3, the full LossDiff-IRM consistently outperforms its ablations. This results corroborates the higher Overlap Coefficients LossDiff-IRM shown in Table 2 and Table 7, indicating that the combination of the two scoring functions provides more reliable criterion for data selection than either one alone. Furthermore, Table 10 and Table 11 summarize the concrete performance of ablation studies with DPO and SLiC respectively.
24

---

Published as a conference paper at ICLR 2026
Table 11: Concrete performance of ablation studt of LossDitt-IRM with SLiC, which is corre- sponding to Figure 3 and Figure 12.
Dataset Ratio
UltraFeedback
AlpacaEval
Vicuna-Bence
LLM
SLiC
Single
WinRate f
Single
WinRate
Single t
WinRate
LossDiff-IRM
64%
5.94
79.51
5.84
78.84
5.85
76.56
Llama-3.1-8B
w/ Liff
80%
5.66
76.44
5.76
77.58
5,33
70,94
w/ IRM
5.36
72.73
5.34
72.64
74.37
LossDiff-IRe
64%
7.87
64.40
8.11
67.58
61.12
Qwen3-8B-Baie
w/ Liss
64%
7.82
62.42
8.12
65.27
8.28
58.13
w/ IRM
7.55
59.62
7.82
62.11
8.36
55.63
LossDiff-IRM
64%
4.82
76.36
5.02
68.85
5.47
74.38
Pythia-2.8B
w/ Diff
80%
4.51
71.01
4,81
66.48
5,30
70.63
w/ TRM
80%
4.53
71.74
4.84
66.79
5.25
67.50
LossDiff- -RM
64%
4.14
74.25
4.39
72.69
4.88
71.25
Pythia-1.4B
w/ Diff
80%
3.78
67.87
4.07
68.06
4.78
66.25
w/ IRM
3.89
70.29
4.17
67.29
4.74
66.87
LossDiff-II -RM
64%
3.30
86.14
3.30
85.16
3.80
85.62
Pythia-410M
w/ Liff
80%
2.95
75.20
2.97
76.53
3.71
72.50
w/ TRM
70%
2.97
78.54
3.08
79.39
3.69
77.50
Layer-wise IF Value
Dat
8:00
8.09
0.81
011
A/2
0.00
8:90
2.91
0X2
C/E
8/11
0.93
9.41
0.00
0.51
.008
40/0
-3.00
8.11
-1.91
0.50
Dat
8.50
6.41
001
0.01
8.00
6:11
0.88
6.11
0.50
Dat
0.30
0.63
8.6
6:81
028
461
0/6
0.
391
6.99
662
8:64
6.00
0244
800
4/0
4.50
81
8:01
682
046
406
0/0
9.1
01
000
641
Data
4181
8.81
8:02
016
414
8:62
6666
8/11
4.81
8.41
8:11
6.6
116
6/6
4.8
8.91
041
Data
E20
8:61
8.00
AUK
0.H
9:04
6.55
0/00
+U00
6.00
0.1
3.M
Data
4.87
-8.6
6.87
4.14
-32
4.18
+322
-0
+4.44
4.19
-0.1
+4.34
-066
801
+.01
4.86
82
+4.0
-6.1
-043
400
4.00
4.16
-22
630
Duta
-001
4,00
0.12
4.9
Mevor
leyer_2
Aever
loyr
leyot Anye
loyer
Hoyat.
Layer
Layet
Aayor
Lawe,
laver
lawe,
laver,
Layer
layer
lays
laves 32
lame
cadred nos
Jayns segge
Figure 13: Visualization layer-wise IF value computed on Pythia-410M.
E.2 MORE VISUALIZATION OF IF
As a supplement to Figure 7, Figure 13 visualizes additional layer-wise IF values computed on the Pythia-410M model using the full UltraFeedbace training and validation sets. Consistent with earlier observations, no single layer dominates the IF values, suggesting that approximating IF using gradients from only one or few layers is not reliable and motivating the need for exploring effective correlated approximation proxies such as our LossDiff-IRM
E.3 RESULTS OF DIRECT TRAINING ON TRAIN, VAL, OR TRAIN+VAL SETS
Since this work introduces an additional validation set, a natural question is how models perform when trained directly on this validation set or on the union of training and validation sets. As supplement, Table 12 reports the results of models trained on the training set, validation set, and the combined training + validation set. under both DPO and SLiC. We observe that none of these settings clearly outperforms the others, despite differences dataset size. This highlights that existing datasets such UltraFeedback contain substantial amount of low-quality data, randomly
25

---

Published as a conference paper at ICLR 2026
Table 12: Results of training on training, validation, or training validation sets, respectively.
Training Data
UltraFeebbakk
AlpacaEval
Vicuna-Bench
LLM
Method
Single
WinRate
Single
WinRate
Single
WinRate
Val Train +
DPO
5.28
72.77
5.34
73.26
5.83
73.44
DPO
5.50
75.16
5.47
73.24
5.50
71.95
Train Train
DPO
5.77
77.61
5.87
78.41
6,04
73.75
DPO LossDiff-IRA -RM
6.54
83.97
6.84
87.08
7.06
86.88
Llama-3.1-8B
Val
SLiC
5.19
72.50
5.16
72.39
5.29
67.50
Train Val
SLaC
5.03
67.41
5.07
69.82
5.15
66.56
Train
SLaC
5.09
70.72
5.13
72.13
5.40
71.88
Train
SLiC LossDiff-IRM
5.94
79.51
78.84
5.85
76.56
Val Train + Val
DPO
7.83
61.73
8.07
67.81
8.36
65.33
DPO
7.72
62.01
7.88
63.77
8.18
52.50
Train
DPO
7.64
7.92
63.85
8.21
62.14
Train
DPO LossDHT- IRM
8.05
67.32
8.36
71,52
8,72
67.19
Qwen3-88B-Bsee
Val Train Val
SLiC
7.71
60.62
8.07
64.59
8.38
58.44
SLiC
7.56
58.44
60.93
8.22
60.59
Train
SLaC
7.55
59.54
7.61
59.71
8.05
$4.69
Train
SLiC LossDiff-IRM
7.87
8.44
Val Train + Val
DPO
4.63
70.88
4,80
64.51
5.49
69.37
DPO
4.54
71.39
4.86
63.67
5,35
66.25
Train
4.60
70.53
67.50
5.35
Pythia-2.8B
Train
DPO LossDif--RRM
4,90
79.62
$,30
76.03
5.74
82.50
Val
SLiC
4.42
67.70
4.71
62.11
4.89
61.88
Train Val
SLiC
4.33
64.29
57.49
4.84
56.25
Train
SLiC
4,36
67.46
4,48
61.66
4,90
65.00
Train
SLIC LossDiff-IRM
4.82
76.36
5.02
5,47
74.38
Val
DPO
3.69
66.45
4.65
65.65 65.65
Train Val
DPO
3,80
65.50
4.01
64,30
4.69
65.62
Train
3.70
66.17
4.71
64.38
Pythia-1.4B
Train
DPO LOssDHT-IRM
4.23
78.49
4,49
76.43
5.28
76.88
Val
SLaC
3.76
66.75
4,02
63.33
4.64
59.38
Train
59.38
Train
SLiC
3,66
63.58
3.98
63.68
4,67
60.00
Train
SLAC LossDifF-IRM
4.14
74,25
4.39
72.69
4.8%
71.25
Val Train Val
2.72
72.57
73.04
72.50
DPO
2.77
73.62
2,80
73.85
3.31
73.75
Train
DPO
2.81
75.25
2.77
73.51
3.10
69.37
Pythia-410M
Train
DPO LossDif-IRM
3.30
86.14
3.30
85.16
3,80
85.62
Val Train +
SLáC
2.68
71.13
2.73
71.74
3.51
66.25
SLiC
2.83
73.17
2.84
73.60
3.21
63.12
Train
SLaC
2.81
73,80
2.82
73.91
3.31
71.25
Train
SLIC LossDiff-IRM
3.07
80.08
3.09
84.39
3.83
79.36
Table 13: Performance of introducing more validation data. The validation set is extended by introducing validation from OASST (Köpf al., 2023) and GoldenHH (Cai al., 2023).
Validation
UltraFeedhace
AlpacaEval
Vicuna-Bench
LLM
DPO
Dataset
Single
WinRate
Single f
WinRate
Single
WinRate
NIA Ultra
Full Data
4.60
70.53
4,95
67.50
5.35
67.50
LossDiff-IRM
4.90
79.62
5.30
76,03
5.74
82.50
Pythia-2.8B
Ulira + OASST
LossDiff-IRM
4.88
78.71
5.11
74.16
5.85
76.88
Ultra GoldenHH
LossDiff-IRM
4,99
80.07
5.28
5.89
74.38
N/A
Full Data
3.70
65.88
3.99
66.17
4.71
64.38
Ultra
LossDiff IRM
4.23
78.49
4.49
76.43
5.28
76.88
Pythia-1.4B
Ulra + OASST
LossDiff-IRM
4.17
77.26
4.44
77.11
77
Ultra GoldenHH
LossDiff-IRM
4.36
80.12
4.69
79.02
5.39
81.01
N/A
Full Data
2.81
75.25
2.77
73.51
3.10
69.37
Pythia-410M
Ultra
LossDiff- IRM
3.30
86.14
3,30
85.16
3.80
85.62
Ultra + OASST
LossDif-IRM
3.11
84.77
3.19
85.24
3,96
86.25
Ultra GoldenHH
LossDif-IIMM
3.25
84.62
3.28
86.71
3.73
83.75
splitting out validation set or simply enlarging the training set by taking their union does not yield performance gains. In contrast, when applying LossDiff-IRM, IRM data selection the training performance improves significantly, further confirming that LossDiff- Diff-IRMM effectively identifies the valuable preference data beneficial for current model alignment.
26

---

Published as a conference paper at ICLR 2026
Table 14: Time cost analysis between RS-DPO and LossDitt-IRM.
RS-DPO
LossDiff- -RM
Trained Model
Prompts
Time
Throughput (prompts/sec)
Time
Throughput (prompis/sec)
Llama-3. 3.1-B
48, 48,908
34 min
2.97
2 h min 2h 43 min
6.08
Owen3-8B-Ban
48,908
50 min
2.81
5.03
Queeat 4Dt Bow BB
Oneeh 6ll Bee (DP1))
Qpeeas sDt Srince
Oreeh sil then (DP0)
Greal
Boyil
Swet
Tomal
Dagel
Figure 14: Analysis of percentile thresholds Esmall, Starge: Timall: Farre of Qwen3-8B-Base. We vary Ssmall = (100- Starge) and smalll = (100- -aargo) with in {0, 10, 20, 30). Analysis for Llama-3. 1-8B is illustrated in Figure 6.
E.4 IMPACT OF INTRODUCING MORE VALIDATION DATA
Since LossDiff-IRM relies on validation set as reference for data selection, Figure has examined the effect of noisy validation sets. A remaining question is whether enlarging the validation set by incorporating data from multiple sources can further improve its effectiveness. As a supplement, Table 13 reports the performance of LossDiff-IRRA IRM when the original Ultra validation set is extended with additional data from OASST (Köpf et al., 2023) and GoldenHH (Cai et al., 2023), resulting in {Ultra OASST) and {Ultra GoldenHH), J. each containing 22,227 preference pairs. We observe that extending Ultra with OASST or GoldenHH yields only marginal improvements on certain models benchmarks, without consistent or significant gains. Notably, regardless of whether the validation set is Ultra, (Ultra + OASST). or {Ultra + GoldenHHH,. applying LossDiff-IRM consistently outperforms training on the full dataset. These results suggest that large validation selection. LossDiff-IRA. remains
preference set is not strictly necessary for effective LossDiff-Ine certain robustness even when the validation set differs from the training distribution, as long as provides a rough reference direction for distinguishing preference data quality.
E.5 TIME COST ANALYSIS BETWEEN LOSSDIFF-IRE AND RS-DPO
High-quality preference data selection and generation are two types of data-centric methods. For example, RS-DPO (Khaki al., 2024) focuses on generating new high-quality preference pairs through rejection sampling using the SFT model, whereas our Loss IRM focuses on analyzing existing annotated preference pairs identify valuable preference pairs. Specifically, RS-DPO requires an external reward model to evaluate the responses generated by the SFT model. Given N prompts and K sampled responses per prompt, RS-DPO needs to score N x K responses with the reward model in order to construct contrastive preference pairs. For LossDifT-IRM, it requires scoring all preference pairs with both the current model and an auxiliary model. For the same scale of N preference pairs, LossDiff- IRM requires N forward passes with the current model and N forward passes with the auxiliary model, resulting in a total of 2NV forward passes, where the two models share the same architecture and differ only in parameters. To facilitate direct comparison, Table 14 reports the data processing time cost of RS-DPO (including data generation and rejection sampling) using the OpenAssistant-11 1.4B reward model with K=8, as well as the data-selection time of LossDiff-IRM. All experiments were conducted on a single 11000-00GB GPU.
The dominant cost of RS-DPO comes from using the reward model to evaluate all N x K responses generated by the SET model. Even using a relatively small 1.4B reward model, the time cost is actually higher than LossDiff-IRM. IRM. Therefore, the time cost of LossDiff-IRM is at least par with RS-DPO, and even often smaller, especially when considering that RS-DPO scales with K or uses a larger reward model.
27

---

Published as a conference paper at ICLR 2026
Lime-l (DPO)
Gewnl-lBeliie (DPO)
Qwen (OPO)
0001
3358
3335
0003
0003
3010
-25
205
LoWD2
Figure 15: Visualization of distribution of LissDiff and implicit reward margin for Llama-3.1-87 and Qwen3-8B-Ban respectively.
Table 15: Concrete performance of LossDift-IRM with noisy validation set, which is correspond- ing Figure 4.
UltraFeedback
AlpacaEval
Vicuna-
Method
Single f
WinRate f
Single 1
WinRate
Single
WinRate
Llama-3.
Full Data LossDiff-IRM.
5.77
77.61
5.87
78.41
6.04
73.75
6.54
83.97
6.84
87.08
7.06
86.88
+
Noise Rate 0.1
6.44
83.79
6.69
85.99
6.76
81.56
+
Noise Rate = 0.2
6.19
80.76
6.35
81.30
6.67
78.44
+
Noise Rate 0.3
6.19
80.71
6.24
82.03
6.54
82.50
+ Noise Rate 0.4
5.95
79.02
6.01
80.37
6.54
78.75
Qwen3-8B-Ban
Full Data LossDiff-IRM.
7.64
61.41
7.92
63.85
8.21
62.14
8.05
67.32
8.36
71.52
8.72
67.19
+
Noise Rate 0.1
7.70
63.78
8.11
68.75
8.47
65.62
+
Noise Rate = 0.2
7.99
66.70
8.26
69.57
8.61
61.25
+
Noise Rate 0.3
7.97
65.98
8.15
69.16
8.65
66.25
+ Noise Rate 0.4
7.84
63.90
8.26
68.35
8.49
65.62
E.6 MORE ANALYSIS OF PERCENTILE TURESHOLLD
As supplement, Figure 14 provides the analysis of percentile thresholds Esmall Starges Tsmall Thrge for Owen3-8B-Bass The rough trends are similar as for Llama-3, 1-8B: 1.tnaa...--BB.. performance first improves and then degrades as the thresholds become stricter. Following this observation, we finally set Esmall and Esmall as 10, Tsmall and Tsmall 10 for all models.
E.7 VISUALIZATION OF DISTRIBUTION OF LOSSDIFF AND IRM
Figure 15 visualizes the LossDiff and IRM distributions for Llama-3. 1-BB and Dwen3-8B-Bass respectively. We observe that both LossDiff and IRM exhibit distributional shapes resembling unimodal, Saussina-like distributions, where the majority of data lies in the middle region and only small number of data fall in the extremes. In our data selection strategy, both criteria concentrate data in their middle ranges, taking their intersection would not result in an extremely small subset. The overlap between their medium regions remains sufficiently large and high-quality for alignment.
E.8 ADDITIONAL SUPPLEMENT
As supplement Figure 3 and Figure 12, report the corresponding numerical results in Table 10. In addition, Figure shows the impact of noisy validation sets on LossDiff-IRR. under different noise rates, with the detailed results provided in Table 15.
28


Reviews:
[{"review_id": "Ch37JeTvIC", "reviewer": "Reviewer_kSkJ", "summary": "This paper studies the problem of identifying valuable preference data for aligning large language models (LLMs) with human preferences. The authors argue that data quality should be viewed as a model-dependent property, not intrinsic to the data itself, and further introduce the Truncated Influence Function (TIF) to evaluate the per-sample effect of preference pairs on validation performance, showing that “medium-IF” data tend to yield the best alignment results.\n\nTo make this computation feasible at scale, they propose two lightweight proxies—Loss Difference (LossDiff) and Implicit Reward Margin (IRM)—and a combined rule LossDiff–IRM for preference data selection. Experiments on various LLM models as well as alignment methods show that the proposed selection could achieve better alignment with about half of the dataset, improving winrate over full-data baselines.", "questions": "Please see the weakness section.\n\nIn summary, this paper provides a well-motivated perspective on preference data valuation, supported with solid empirical evidence. I currently tend to recommend acceptance. However, I'm willing to re-evaluate this work according to the further dicusssions.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper is well-motivated. The idea that data valuation is model-dependent is insightful and challenges a long-standing assumption in RLHF and preference optimization pipelines.\n\n\n2. The proposed Truncated Influence Function (TIF) seems new in the literature and provides a principled mechanism to quantify individual data influence. \n\n3. Results span several LLM families, datasets, and alignment methods, demonstrating generality and robustness.", "weaknesses": "1. The “medium-IF” hypothesis is interesting, but primarily from the empirical observations. It would be better if the authors could elaborate more in a theoretical or formal way. \n\n2.  It seems a commonly used benchmark, arena-hard, is not included. Can authors elaborate more on it?\n\n3. (Mirror Issue) The presentation could also be improved. For example, the captions of Figure 1 and Figure 3 refer to the materials in the appendix. From my personal perspective, it might be better to put the related paragraphs back into the main paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0qyKcj2SMA", "reviewer": "Reviewer_8ymy", "summary": "The paper reframes preference-data quality as model-dependent and introduces a Truncated Influence Function (TIF) lens showing that medium-IF pairs—not very small or very large ones—drive the most stable alignment gains. To avoid the high cost of exact IF on LLMs, the authors propose two forward-pass proxies—Loss Difference (LossDiff) and Implicit Reward Margin (IRM)—and a combined LossDiff–IRM selector that closely tracks TIF.", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. **Principled, model-aware data valuation.** The paper grounds selection in a TIF-motivated view of *model-dependent* data value and shows that combining **LossDiff** and **IRM** outperforms either proxy alone; moreover, pairs discarded by the selector are empirically low-value for alignment, confirming the criterion’s discriminative power.  \n\n2. **Strong empirical generality.** Across models, objectives (DPO/SLiC), and benchmarks, the method achieves higher win rates using reduced data (e.g., 64% subset with top performance).", "weaknesses": "1. **Incomplete positioning vs. recent work.** The paper discusses Filtered DPO, but does not engage with several highly relevant baselines:\n\n   * [1] margin-based preference selection for alignment quality (ICML 2025),\n   * [2] RS-DPO (rejection sampling + DPO for cleaner preference data),\n   * [3] active preference learning for LLMs (querying informative pairs instead of passively filtering).\n     These works target the same core problem — selecting / curating high-value preference pairs — and should be compared both conceptually and empirically.\n\n2. **Practicality.** The proposed selectors still require rescoring large volumes of pairs with both the current model and an auxiliary model. The paper does not report the real cost (GPU hours, throughput) or show that this is cheaper/more scalable than RS-DPO-style sampling or active preference acquisition.\n\n3. **Robustness.** The “medium-IF is best” claim is convincing on the reported setups but is not stress-tested across broader domains, model sizes, or stages of alignment; it is unclear how stable this curriculum is outside the presented benchmarks.\n\n[1] Larger or Smaller Reward Margins to Select Preferences for Alignment? ICML 2025.  \n[2] Rs-dpo: A hybrid rejection sampling and direct preference optimization method for alignment of large language models. NAACL 2024.  \n[3] Active preference learning for large language models. ICML 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D5Ae5ei8a6", "reviewer": "Reviewer_CD9n", "summary": "This paper proposes a novel data selection method for preference data, inspired by two points:\n\n1. Preference data selection based on influence function (IF);\n\n2. DPO formulation implicitly encodes the reward difference, which can be leveraged for better estimation of IF.\n\nThrough preliminary experiments, the authors reveal an interesting phenomenon: data with medium IF values (i.e., within a truncated IF range) yields the best performance. They further propose a new data selection strategy to avoid the heavy computational cost of IF estimation, and demonstrate positive experimental results.", "questions": "How are the thresholds set? I did not find sufficient details in the paper regarding the choice or tuning of these thresholds.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea is novel and surprising. Traditionally, it is believed that maximizing the margin (i.e., the loss difference) for single-step optimization is most beneficial for learning. However, this paper shows that truncated medium IF actually performs best.\n\n2. The authors also carefully consider the high computational cost of IF computation in practice and propose an approximation strategy.\n\n3. In the preliminary experiment, both DPO training and validation reward margins are computed on subsets from UltraFeedback, which share similar distributions, while in the main experiments, OOD situations are also tested, and the core design of this paper works on both situations, proving its value.", "weaknesses": "1. Although the analysis is thorough, the method itself is relatively simple, and the level of technical innovation is somewhat limited.\n\n2. The proposed LossDiff-IRM takes the intersection of two selection criteria. However, model performance also depends on data scale. If the intersection yields a very small subset, it may pose a risk of data scarcity, which should be addressed.\n\n3. Since pairwise preference data and DPO-style training are becoming less common, the strong coupling between the proposed data selection and model completions might limit its applicability to more recent paradigms such as RL-based methods. It would be helpful to include a discussion on how such approaches might extend to the prompt-level selection process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KRdhxdEHsA", "reviewer": "Reviewer_sQAd", "summary": "This paper studies the data quality of preference pairs for LLM alignment.\nThe authors begin with influence function analysis of each preference pair's impact, which measures the consistency of datapoint's gradient with the overall gradient of validation set.\nThe analysis shows the medium-IF preference pairs are more valuable, so the authors design the TIF metric: $\\mathbb I[\\delta_{low} < IF < \\delta_{high}]$ to select medium-IF data for training.\nFurthermore, to reduce the computation cost of IF, the authors introduce two simplified data quality metrics:\n1. LossDiff: the difference of loss between $\\pi_\\theta$ and a validation model $\\pi_{val}$;\n2. IRM: the implict reward margin.\n\nBoth metrics correlate well with IF and can be used to approximate TIF for data selection.\nEmpirical results on different models, DPO variants, and benchmarks demonstrate the effectiveness of the proposed data selection methods in improving alignment performance.", "questions": "Can you provide more comparison with prior model-specific data selection methods?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Clear Motivation and Well-Justified Methods**: Data quality metrics based on model state serve as an intuitive and reasonable approach to improve preference optimization performance. The paper's derivation from influence function analysis to practical data selection metrics is well-motivated and clearly presented.\n2. **Stable Improvements**: The proposed data selection methods consistently enhance alignment performance across different models, DPO variants, and benchmarks, demonstrating their robustness and effectiveness.\n3. **Comprehensive Ablation**: The paper includes several ablation studies to provide further insights including model-specific selection, noisy robustness, and optimized layers.\n4. **Presentation Quality**: The paper is well-written and easy to follow, with clear explanations of the proposed methods and experimental results.", "weaknesses": "My main concern with this work is the lack of comparison with prior model-specific data selection methods.\n\nThe authors state in L56-58:\n> Our above analysis suggests a reasonable yet seldom-discussed viewpoint: preference data selection\nshould be performed for specific models and explicitly related to the training process\n\nHowever, there are already several works discussing model-specific data selection. For example, implicit margin based selection methods [1,2,3,4] also take the model state $\\pi_\\theta$ into consideration when selecting data. \nSo I think it would be better not to claim that this is a \"seldom-discussed viewpoint\".\n\nMoreover, [3,4] utilize IRM to select data in a similar way: They prioritize preference pairs with the smallest absolute IRM value, i.e., medium-IRM data, which is similar to the TIF metric proposed in this paper.\nSo it would be better to cite and compare with these prior works.\n\n---\n\n**References**\n\n[1] Morimura, Tetsuro, et al. \"Filtered Direct Preference Optimization.\" Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.\\\n[2] Deng, Xun, et al. \"Less is more: Improving llm alignment via preference data selection.\" arXiv preprint arXiv:2502.14560 (2025).\\\n[3] Huang, Kexin, et al. \"Larger or Smaller Reward Margins to Select Preferences for LLM Alignment?.\" Forty-second International Conference on Machine Learning. \\\n[4] Yang, Sen, et al. \"Not All Preference Pairs Are Created Equal: A Recipe for Annotation-Efficient Iterative Preference Learning.\" Findings of the Association for Computational Linguistics: EMNLP 2024. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)