Data Selection for LLM Alignment Using Fine-Grained Preferences
Authors: Jia Zhang
Yao Liu
Chen-Xi Zhang
Yi Liu
Yi-Xuan Jin
Lan-Zhe Guo
Yu-Feng Li
Venue: ICLR 2026 Poster (2026)
Keywords: Data Selection
Preference Alignment
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Large language models (LLMs) alignment aims to ensure that the behavior of LLMs meets human preferences. While collecting data from multiple fine-grained, aspect-specific preferences becomes more and more feasible, existing alignment methods typically work on a single preference and thus struggle with conflicts inherent in such aggregated datasets. As one early attempt, in this paper, we propose a data-centric approach to align LLMs through the effective use of fine-grained preferences. Specifically, we formulate the problem as a direct fine-grained preference optimization and introduce preference divergence (PD) that quantifies inter-aspect preference conflicts. Instead of directly tackling the consequent complicated optimization, we recast it as a data selection problem and propose a simple yet effective strategy, which identifies a subset of data corresponding to the most negative PD values, for efficient training. We theoretically analyze the loss-bound optimality of our selection strategy and conduct extensive empirical studies on varied settings and datasets to demonstrate that our practical selection method could achieve consistent improvement against standard full-data alignment, using even just 30% of the data. Our work shares a line that LLM alignment using fine-grained preferences is highly feasible.

Published as a conference paper at ICLR 2026
DATA SELECTION FOR LLM ALIGNMENT USING PREFERENCES
Jia Z Yao Liu', Chen-Xi Zhang! Yi Liu", Yi-Xuan Jin',
Lan-Zhe GuolAt Yu-Feng Lil2t
National Key Laboratory for Novel Software Technology, Nanjing University
2 School of Artificial Intelligence, Nanjing University
3 Algorithm Tech, Taobao & Tmall Group of Alibaba
4 School of Intelligence Science and Technology, Nanjing University
z zhangjia, guolz, liyf nju. edu.
ABSTRACT
Large language models (LLMS) alignment aims to ensure that the behavior of LLMs meets human preferences. While collecting data from multiple fine- grained, aspect-specific preferences becomes more and more feasible, existing alignment methods typically work on single preference and thus struggle with conflicts inherent in such aggregated datasets. As one early attempt, in this paper, we propose a data-centric approach to align LLMs through the effective use of fine-grained preferences. Specifically, we formulate the problem as a direct fine- grained preference optimization and introduce preference divergence (PD) that quantifies inter-aspect preference confticts. Instead of directly tackling the con- sequent complicated optimization, we recast it as a data selection problem and propose simple yet effective strategy, which identifies subset of data corre- sponding to the most negative PD values, for efficient training. We theoretically analyze the loss-bound optimality of our selection strategy and conduct extensive empirical studies on varied settings and datasets to demonstrate that our practical selection method could achieve consistent improvement against standard full-data alignment, using even just 30% of the data. Our work shares a line that LLM aligmment using fine-grained preferences highly feasible.
INTRODUCTION
Reinforcement learning from human feedback (RLHF) (Christiano et al. 2017 Ouyang al. 2022
Bai et al.] (022) plays a pivotal role in aligning large language models (LLMs) Naveed et al. 2024 Brown et al. 22200) Meta Al 2023). However, standard RLHF methods of online RL (Schulman et al. 2017 Shao et al.) 2024) are often burdened by substantial computational overhead and the complex, multi-stage training process. As an efficient alternative, methods like Direct Preference Optimization (DPO) (Rafailoy et al. 2023) directly align LLMs by fine-tuning on an offline dataset of human preferences, bypassing the need for the complex and unstable RL-based methods.
The efficacy of DPO is tied to the quality of the offline preference dataset. The common practice is to collect data based on an overall "better-than" preference, but it often suffers from ambiguity and overall judgment, some studies (Ji et al.) 2023 Wu et al. 2023] Rame et al. 2023) argue that the overall preference be decomposed into multiple compatible fine-graie -neegraided aspects, which also term
intractability in annotation (Bakker al. 2022] Casper et al. (2023). Instead of relying
sub- prefereeeess for simplicity. Collecting fineggrained preferences is more feasible as the underlying criteria simpler, which in turn enhances the tractability and consistency of the annotations.
Fine- grained judgment provides a more feasible pathway for discerning the preferred response. However, this approach faces two main challenges. 1) Existing DPO-like methods are mainly devel- oped for single preference and fail to handle different fine-grained preferences. 2) More critically,
Work done during the authors' internship at Alibaba Group.
Corresponning authors.

---

Published as a conference paper at ICLR 2026
Preference Conflict Case
Preference Conflict Statistics
Please describe a time when you faced rejection overcame
Query
recenily applying
but unfertunately
28 6% All Consensus
R.A
didn't get interview. Though discouraged...
Explicit Conflict
As model, hoven't experienced rejection The funilionaal serse. However, I can share un
R.B
33.2%
4.7% Neutral
better
better
Tie
Parilal Consensus
Inst-Folowing
Truthfulness
Honesty
Helpfulness
Conflicts between the fine-grained and overall preferences commonly occur, and only part of the samples show complete consisteney across all fine-grained aspects.
Figure 1:
collecting fine- theeggraned preference data introduces severe data quality issues, such redundancy, noise, and especially preference conflicts, leading to training inefficiency and performance degrada- tion. For instance, the statistical analysis of the widely-used preference dataset UltraFeedback (Cui al. 2023) in Figure [T which provides preference of fine- fine-grained aspects, reveals that nearly 30% of samples exhibit explicit preference conflicts. These challenges raise critical research question: How can we effectively harness these fine-grained preferences for robust model alignment, espe- cially in the presence of inherent and severe preference conflicts and noise?
In this paper, we study LLM alignment using fine-graines preferences through a data-centric view. We first formulate the problem as direct fine-grain fine-grained preference optimization (DFPO) that utilizes all fine-grained preferences for alignment. Motivated by the insight that the DFPO assigns varying importance fine-grained preference data, introduce preference divergence (PD) quantify inter-aspect preference conflicts. Instead of directly tackling the consequent complicated optimiza- tion, recast it data selection problem and propose simple yet effective strategy, which identifies a subset of data corresponding to the most negative PD values, for efficient training.
We theoretically analyze the loss-bound optimality of our selection strategy. Moreover, extensive empirical evaluations on varied settings and datasets demonstrate that: Leveraging fine-grained pref- erences with no additional annotation efforts, our method consistently outperforms full-data align- ment, even using just 30% of the data, highlighting its effectiveness in filtering high-quality data from the large datasets. Distinguishing itself from other heuristic data filtering methods, our work is established on theoretical selection guidance and, to the best of our knowledge, is the first work to utilize data selection among fine-grained preferences that contain noise and conflicts to facilitate more robust and efficient alignment. To summarize, our contributions are as follows:
To study LLM alignment using fine-grained preferences in the presence of data issues like noise and conflicts, we formulate the direct fine-grained preference optimization and intro- duce the preference divergence to measure inter-aspect preference conflicts.
We recast the complicated fine-grained preference optimization a data selection problem and propose a simple and effective data selection method: identifying a subset of samples corresponding the most negative estimated PD values for efficient training.
(3) We develop theoretical study to analyze the loss-bound optimality of strategy, and conduct extensive experiments to show that our selection method could outperform the standard full-data alignment using even just 30% of the data, sharing line that LLM alignment using fine-grained preferences highly feasible.
2 RELATED WORK
2.1 LLM ALIGNMENT WITH PREFERENCE
Preference alignment is crucial for ensuring that the behavior of LLMs adheres to the expectations of human values. Popular alignment methods include supervised fine-tuning (SFT) (Wei et al.,
2

---

Published as a conference paper at ICLR 2026
2022
reinforcement learning fine-tuning (RLFT) (Schulman al. 2017 Shao et al. 2024, Hu et al. 2025), and DPO-like approaches (Rafailvv et al. 2023 Azar et al,
Ouyang et al. 2022).
2024
Ethayarajh et al. 2024 Meng et al. 2024). which directly optimize the model on preference
data. While the majority of existing work concentrates on aligning LLMs using an overall "better- than" preference, a few studies (Ji et al. 2023) Wu al. 2023 Rame et al. 2023 Zhou al. 2024) have made preliminary attempls at multi-objective preference optimization, aiming to achieve the Pareto frontier by applying different weights to various sub-preferences. In contrast, our work aims to effectively use fine- mee-griined preference in the presence of severe data quality for efficient training.
2.2 LLM ALIGNMENT WITH DATA SELECTION AND FILTERING
Data selection and filtering are widely recognized as critical factors in both the pre-training and post- training stages of LLMs. For pre-training, studies (Xie et al.) 2023 Gu et al. 2025 Wang et al.) 2025) focus on filtering for higher-quality subsets to enhance model capabilities. This challenge becomes even more acute in the post-training step (e.g.. SFT, DPO, RLFT). Unlike pre-training, where scaling laws (Kaplan et al. 2020) may compensate for moderate noise, alignment datasets typically orders of magnitude smaller, making the quality of the data critical. A growing body of research confirms that harmful or redundant examples can drastically degrade fine-tuning outcomes and prevent the model from learning the desired behaviors (Kung et al. [023] Xia et al. 2024 Li iet al. 2024; Zhang et al., 2025). These studies emphasize the necessity of active and effective data curation for aligmment. Existing data selection methods for DPO or RLFT (Deng al.
2025 Lee
et al. 2025 Gao et al. 2025 Li et 20255) are primarily limited to an overall preference. The base idea of these methods often involves using an internal or external reward model measure sample difficulty and perform filtering. Our study stems from key insight from the DFPO, motivating our data selection strategy for more effective L.M alignment using fine-grained preference data.
3 USING FINE- GINEGGRINNED PREFERENCE FROM A DATA SELECTION VIEW
We formalize settings of LLM alignment using fine-grained preferences and derive the direct fine- grained preference optimization objective ($3.1). Insight from DFPO motivates to introduce the preference divergence ($3.2) and formulate an alternative data selection problem. For this new problem, we then theoretically propose selection strategy with the loss-bound optimality ($3.3).
3.1 DIRECT FINE-GRAINED PREFERENCE OPTIMIZATION
Problem Formulation. We consider an alignment setting using fine-graines preferences. A sub- preference dataset, DR, is a collection of preference data (x* y uf*) where for each prompt = the response V is preferred over y/ under the specific fine-grained criterion k. The entire dataset D D= ( x - ot k E [k], (ax ( yh: v/) E D& then aggregated from K such sub sub-preference datasets from different aspects. We assume that each sub- sub-preference k is modeled by correspond- ing latent reward model, (1, y), such that for any given sample, the winning response assigned a V v vft) E D2-- The aggregation of these sub- aub-rreference data can introduce preference conflicts, which we formally define as follows. The goal is to use this aggregated fine-gra fine-grained preference dataset D) for effecctiee LLM alignment.
higher reward than the losing one: Ta (2* v > Th (x* wf¹),
Definition 3.1 (Preference Conflict). Assume there is ground-truth reward model r" for the overall preference. A conflict between fine- grained and overall preferences occurs for sample (k, vi. vf) when Tx(a*, v(t) > (x vf*) while r (x* vt) < r r*(x", v²). Note that this conflict arises primarily from data quality issues rather than inherent trade-offs or incompatibility between the definitions of different fine-grained aspects.
Definition 3.2 (PPO Using Fine-Grained Preferences). Given an initial policy model Trefs and assuming that the latent reward model ry (x, y) for each fine-grained aspect is available, the standard PPO objective for RL fine-tuning (Schulman al. 2017) using multiple fine-grained preferences
3

---

Published as a conference paper at ICLR 2026
can be formulated as follows,
arg max D.g.Daaa TW
Σ Tk (x(2,))) BBErDD [DLL (TO(-)a) [ree [+)(()))
(1)
DEPO Objective. We derive the direct fine-grained preference optimization objective by extend- ing the principle of DPO (Rafailov et al. 2023) to the fine-grained preference alignment setting, resulting in the following loss function (see Appendix A.11. for full derivation from Eq. ①
(DFFO(() = - EEWD log
KMM(=)) Adu(z)
(2)
PD term
Here, Mz(z) represents the preference margin,
To [y (x*) Traf (y 2**)
To (of (2
Ma(z) = 3 log
B log Teef (v/ lx*)
(3)
And we introduce (d as the preference divergence (PD) term, formally defined as: Adx(2) Φia(a. v/n)) -
(4)
(x(I, = Σ (x,
(5)
k'
3.2 SELECTION INSIGHT FROM PD TERM
The key distinction between DPO and DFPO is the PD term, which functions implicit data weighting mechanism. We analyze two opposing scenarios provide an intuition of role:
(²)(2) > 0: A positive PD term indicates that the sub sub-preference of aspect conflicts with the majority of other aspects. Foreing the model to learn from such sample may be detrimental to overall behavior. The positive PD term in DFPO reduces the sample's impact on the loss, thereby mitigating the preference margin.
(Φ)(2) O: A negative PD term suggests the sub uub-preference of this data aligns well with the consensus of others, potentially high-quality, reliable sample. The negative PD term in DFPO up-weights the sample's priority, reinforcing the log- bog-probabilityy margin.
The analysis above reveals that the PD term implicitly re-weights samples by measuring the sensus or conflict among fine-grained preference aspects. Despite its potential, DFPO still faces practical challenges, such high computational cost and the risk of instability from unavailable or unreliable reward models. However, given the varying value and importance of samples, a potential solution arises: Can design strategy filter the dataset in advance? Such a strategy aims to curate a high-quality subset to improve aligument performance while enhancing training efficiency.
3.3 DATA SELECTION PROBLEM AND STRATEGY
Accordingly, instead of using PD terms for the consequent complicated optimization, we propose to utilize them as the basis for data selection. The target is to find subset of samples for standard DPO, such that the resulting policy should minimize the DFPO objective. We formalize this selection problem and present our key theorems below, which ground the validity of the proposed selection strategy. Proofs are deferred to Appendix A.2
Definition 3.3 (Data Selection Problem for DFPO). Assume the d% are known. Give a dataset D consists of data from x sub-preference dataset Dz. supervised fine-tuned model myefs the DPO objective CDPO. the DFPO objective LDPPO. a selection budget X. The goal is to find selection strategy that selects for subset C D for DPO training, which results in optimal LDFPO:
D arg min CDFOO D),
DCD
(6)
s.t. Ty = arg min ( D), DD/lD X.
ne
4

---

Published as a conference paper at ICLR 2026
Theorem 3.4 (Loss Bounds of DFPO in Data Selection Problem). Consider the learned policy g was only trained on the subset D. Assume Tg gives preference margin on D bounded by Ma(z) e [c1, c2l and suboptimal expected and bounded preference margin and loss on : D, such that ED,5 [- log (M:M))))))) 1,, DD,DD [Ma(z)] S Sc. Then, the DFPO loss bounded as follows,
DFPO (D) S LDFPO S Drro (D), clower (D) - -og (kez + D [ (1 - Ao + Ep (8)
DDFO (D) = i llgg a (RC1 + ( - -E ( - 4).
(9)
Theorem 3.5 (Selection with Loss-Bound Optimality). Let any selection strategy be partition of the dataset D into D and D \ D, and regard the loss bounds of LDrro as function of D. Assume normalized Tı E [0, r) and under the mild condition that 2(6-12 r S C2 Co, the strategy 6. that optimizes both bounds is to select samples with the most negative PD term,
arg - (- {=|Di/|>k
(10)
Theorem 3.5 demonstrates that the strategy of prioritizing samples with the most negative PD values guarantees minimizing both bounds, potentially leading to better performance than other strategies.
4 THE PROPOSED PD SELECTION METHOD
4.1 PD TERM ESTIMATION
To bridge the gap between the theoretical selection strategy and practical method, the most crucial obstacle is the lack of ground-truth latent reward gap of samples for computing the PD term, we can only access one specific fine-grained preference for each sample. To this end, we propose the PD term estimation method, which utilizes smaller proxy model to explicitly learn the preference pattern of each sub- sebb-reference and mutually extrapolate and estimate the pseudo-reward for samples in other sub-preference datasets. Specifically, we train the reward models f.. by contrastive learning with B BT-model [Bradley & Terry 1952) for each sub-preference k:
fk arg min ED, [- log (rra vir
r(
(11)
The obtained reward models can be used to predict and estimate the pseudo-reward gap of their sub-preference for samples gathered from other aspects.
Afx(2') = (a v
(z vi A2' & DR-
(12)
To ensure the comparability of pseudo-reward gaps across proxy reward models of different sub- preferences, we apply quantile scaling and normalize the resulting scores into the range of [-1, 1], and thus yield the final PD terms:
9: = (+(AA(())) Ak' # k, 2 Dw
(13)
APx(z) -1,
AFx(2) Clip
(14)
PD(z) = Σ AFw(z), E D.
(15)
where P,, denotes the y-quantile of value set. The underlying insight is that fine- inn-grained preference patterns are easier to capture, which allows the use of smaller model and less data for reward modeling and prediction, thereby ensuring both precision and efficiency.
5

---

Published as a conference paper at ICLR 2026
Gather Fine-Grained Preference Data @ Reward Models Data Sampling
① Sub-Preference Reward Training
The dataze! aggregates dala from diveesee fin- grained proferences
P
Dg
D
Balance somoolng rallo Rondom sampling
Roward loss function length
D,
D,
Estimation across
Selection based Estimated
(2),) ED;
PD values
Curaled SUb aab-cataser B
Seleci based on PD values
Figure 2: The overall workflow of the proposed PD selection method.
4.2 LENGTII BIAS MITIGATION
The estimation of PD terms relies on proxy reward models trained on fine- fine-griined preference data. However, these models are susceptible to length bias, vell-documented issue where longer re- sponses are favored regardless quality (Singhal et al. 2024 Huang et al., 2024: Lambert et al.) 2024). If left unaddressed, this bias would propagate into the PD term estimation and corrupt the data selection. Therefore, we employ two strategies to mitigate the underlying length bias reward modeling, which allows to obtain more reliable reward gaps for PD term estimation.
Length Balanced Sampling. First, to mitigate the intrinsic bias towards longer responses, we employ balanced sampling strategy. We partition each sub-preference dataset D₁, into two disjoint subsets based on length difference: D = {= E Dx (en( 2 (yn)) and D = {a E Dr (w) < (en(x:)) Let fir I fü = / D/ be the frequencies of these samples, compute an adjusted ratio using balance temperature т: fo exp(ff /r)/(cxp(/f1 /T) + exp(f% /T)). Given a specific sampling ratio pr for training the reward model, we sample p,. ft and pr f data from D and Dj respectively, to obtain more balanced training set D/-
Length Reward Penalty. Second, we introduce an explicit penalty term into the reward modeling to discourage length bias. We hypothesize that the total reward r(z, y) can be decomposed into quality component (2, y) and length component r2 y)- We model the length component as simple linear function of the response length (y) p len(y), where p is the length penalty coefficient. To encourage the model to focus on fitting the intrinsic quality rather than the superficial length feature, add the length penalty term directly into the reward loss function:
C(r) = Ed; [- log (r(x, yw) - r(z, u)) - pálen(z))]
(16)
Here, Alen(=) len(yw) len(n) is the length difference between the chosen and rejected re- sponses. Then, the estimation of pseudo-reward gaps from Fq. (12) can be refined by:
Arx(2') (a" w "x at p=len(zz).
(17)
4.3 PD SELECTION METHOD
Our method involves several steps, as illustrated in Figure 2 and Algorithm [ First, we collect the aggregated dataset from multiple sub-datasets of different fine-grained preferences. Then, for each sub- sub-rreference, we train a de-biased reward model using a smaller proxy model and leverage these models estimate the PD term for each sample across the entire dataset. Subsequently, select data subset by retaining the samples corresponding to the most negative PD values within the selection budget. Finally, this curated dataset will be used align the L.M via standard DPO.

---

Published as a conference paper at ICLR 2026
Table Performance comparison of different methods. We report the win rate (WR) and the length- controlled win rate (LC) for AlpacaEval 2, the average win score (AW) across the five test sets, and GPU hours required for (selection and) training. Results of Qwen2.5 are Appendix C.2
Dataset
UltraFeedback
HelpSteer
AlpucaEval 2
Pairwise
GPU
AlpucaEiall 2
Pairwise
GPU
Model
Method
WR1
LC₁
AW
Hours
WRr
LCr
AW
Hours
INIT
SFT
7.08
14.00
0.64umms
0.0
4.52
8.25
0.76,am
0.0
Llaama3. 3,1 1-8B
OVA.
14.35
19.96
1.00aam
33.6
5.75
9.95
1.00.am
10.0
AVG.
16.63
22.21
1.15.m04
33.6
6.18
9.55
1.03gaus
10.0
FULI
ALL
15.18
21.14
1.08amm
33.6
5.55
1.02auus
10.0
DMPO
19.42
24.73
1.25umns
42.1
6.01
10.35
1.09
13.7
RAND
14.72
19:56
1. 1.02auus
10.1
5.53
8.50
0.98gum
5.0
SELT
RAF
19.64
23.34
1.18aamm
16.2
6.58
10.46
1.10
PD (rati))
18.85
25.38
1.23
10.1
6.57
12.41
1.13,0m1
5.0
PD
21.00
1.24
18.6
7.55
12.13
1.19
RAND
RAY
UMPO
Llama-IIS
Quen-1I8
25
14
IiC-Wa Hale
12
12
10
15-
30
40
50
20
40
50
20
30
20
30
40
Selection Budgot (%)
Selection Budget (%)
Selection Budget
Selection Budget (%)
Figure 3: Performance variation with different selection budgets for settings in $5.2.
Notably, our approach holds significant practical value. While it leverages fine- one-grained preference, introduces no additional annotation overhead. This can be achieved by viewing the entire dataset as a collection of disjoint subsets, where each subset is annotated against one specific sub-preference. Consequently, each sample still requires only preference annotation. Annotating based sub proferefeeece simplifies the judgment criteria, leading to easier and more feasible collection of preference data. The learned patterns from these uub-preferences then generalized across the entire dataset to facilitate the final filtering of a high-value subset based on our method.
EMPIRICAL STUDY
We validate our proposed method through comprehensive empirical study. We begin by detailing the experimental setup 15.1. We then present the main results in $5.2, demonstrating the general effectiveness of our method across different models and datasets. To provide deeper insights, first, we investigate the detrimental impact of preference conflicts on alignment and show how data selection helps ($5.3). Following this, we study the effect of varied selection budgets on alignment performance ($5.4). Moreover, $5.5 explores the sensitivity different proxy reward models. And $5.6 validates the effectiveness of our method through comprehensive ablation studies. Lastly, $5.7 further applics our approach to a proprietary real-world Taobao Live application.
5.1 EXPERIMENTAL SETUP
Fine-Graiie -en--Grinned Preference Dataset. We construct two fine-grained preference datasets from UI- traFeedback (Cui et al. 2023) and HelpSteer (Wang et al., [2023b] 2024) to simulate the aggregation of data from diverse preferences, leveraging the provided fine men-grained aspects from them. To further validate the effectiveness and robustness, we also created three datasets with different conflict levels.
The detailed description is provided in Appendix C.1

---

Published as a conference paper at ICLR 2026
Table 2: Performance under different conflict levels. The results for OVA., and AVG, are presented in Table as these methods are unaffected by the varied conflicts since they re-label the data.
Dataset
Conflict Level 10%
Conflict Level 20%
Coonflict Level.30%.
AlpucaEival
Pairwise
Alpucatival
Pairwise
Alpacalival 2
Pairwise
Method
WR,
I.C
AW
WR+
1.C+
WR+
T.C,
AW
15.18
21.14
1,88
13.28
18.07
13.40
16.44
0.96
FULL
DMPO
19.42
24.73
1,25
17:53
23.28
1.17a006
15.43
20.99
I 1.092000
RAND
14.72
19.56
1.02
14.06
18.45
1.0
13.17
18.82
1 1.00g.0s
SELT (30%))
RAF
19.64
23.34
1.18g0
19.51
22.62
1.19
18.23
21.76
1.172006
PD (rati.)
18.85
25.38
1023
18.04
24.81
1.21
18.65
24.96
1.23
(ours)
21.00
26.11
1.24
19.48
25.17
1.23
20.40
24.71
1.21 1.21 2000
RAND
RAF
(D
(10(ouns)
OVA
AVG
ALL
DMPO
Conflict Level 10%
Conflict Level 20%
Conflict Level 30%
L.C-Wi Rate
2.5
25
25
20-
20
20
30
50
15
20
50
15
20
Selection Budget (%)
Selection Budget (%)
Selection Budget (%)
Figure Performance variation with different selection budgets for settings in $5.3.
Evaluation Protocols. A) AlpacaEval 2 Benchmark (Li al. 2023 Dubois et al. 2024). We evaluate models on the widely recognized AlpacaEval 2 leaderboard. This automated bench- mark evaluates model outputs against those from GPT-4 (OpenAI| 2024). using the AlpacaFarm dataset (Dubois et al. 2023). B) Pairwise Evaluation. To assess performance on open-ended gen- eration tasks, we also adopt the aa---as-a-uugge paradigm (Li al., 2025a Zheng et al. 2023) to conduct pairwise evaluation on five commonly used open-ended test datasets spanning diverse domains. The win score is reported for comparison. More details are provided in Appendix C.4
Candidate Strategies. We evaluate our proposal against categories of methods: Full-data alignment (FULL) and Selection methods (SELT). The FULL category utilizes the entire dataset, namely: FI) OVA.: re-labels the data using the overall preference provided by the dataset for DPO; F2) AVG.: re- re-labels the data using the average rating of fine- fine-grained preferences provided by the dataset for DPO: F3) ALL; directly use the entire fine-grained preference datasets with preference conflicts for DPO; F4) DFPO: applies DFPO on the entire dataset. The SELT category selects data subset based on different strategies for DPO, namely: S1) RAND: selects samples randomly. S2) RAF: Train unified reward model to estimate self-agreement for filtering, following the strategy of (Deng et al. 2025 Lee et al. 2025) S3) PD (rati.): Use the ground-truth discrete rating of each sub-preference for PD selection strategy. S4) PD (ours): selects data by our proposed method.
5.2 RESULT I: IMPROVED PERFORMANCE WITH REDUCED COST
Table presents the main results. For all SELT methods, we set the selection budget A=30% for UltraFeedbaal and A=50% for HelpSteer, and use OVA. as the baseline for all pairwise evaluations.
Our selection method achieves superior performance with reduced training cost. explicitly filters out conflicting and low-value data to retain subset of high-quality samples. The PD (rati.) also has comparable performance. Its reliance on pre-defined preference ratings makes it less sus- ceptible to length bias. However, the ratings are limited to a few discrete values, leading to potential inaccuracies. Additionally, its annotation requirement is costly, as it necessitates explicit and accu- rate ratings across all fine-grained aspects for every sample. whereas we require only single binary preference for one aspect. We also observe that DFPO yields considerable improvements over FULL methods. But it has limitations: estimation emrors in PD terms directly hurt the policy update, and
8

---

Published as a conference paper at ICLR 2026
30%
43%
6B Oveed 1:1
20-
15
1065
20% Contint Londl
306
Figure 5: Comparison of differ- Figure 6: Comparison of differ- Figure 7: Performance variation ent proxy reward models.
ent ablation strategies.
of different ablation methods.
low-value samples, although weighted still consume computational resources and are learned by the policy model, potentially creating performance bottlenecks.
is noteworthy that eurated subset can outperform the full-data alignment, phenomenon also reported in previous studies (Deng et al. 2025 Lee et al., 2025 Gao et al. 2025). Even for method (AVG.) with better preference consistency, there exist samples that are challenging, ambiguous, or exceed the model's capacity, which can be detrimental to model alignment. In contrast, the OVA. and ALL methods are directly susceptible to inherent preference noise and conflicts.
5.3 RESULT II: PREFERENCE CONFLICT HURTS BUT DATA SELECTION HELPS
The results on varied levels of conflicts are reported in Table We observe that as preference conflicts in the dataset increase, directly applying DPO to the entire dataset (row ALL) leads to severe degradation alignment performance. This issue requires careful consideration because ag- gregated datasets, collected from multiple fine-grained preference aspects, inevitably contain such explicit confficts that directly harm the effectiveness of alignment. Without proper data cura- tion, the truly valuable preferences overwhelmed by conflicting and low-value preference data, thereby harming the alignment outcome. contrast, PD (ours) and PD (rati.) exhibit robust per- formance across varying conflict levels. These results validate that effective data selection, which leverages fine-grained preferences to filter out harmful samples, can help mitigate these data issues achieve robust and efficient alignment, even using datasets with explicit conflicts and noise.
5.4 RESULT III: STABLE SUPERIORITY ACROSS SELECTION BUDGETS
To further study the selection dynamics of different strategies, incrementally increase the lection budget and report the corresponding performance in Figure 3 and Figure 4 As the budget increases, the performance of all methods exhibits trend of initially improving and then converging or even declining. This initial improvement is expected, as tiny budget provides insufficient train- ing, leading to weak performance. Conversely, as the budget expands excessively, the performance eventually converges to that of direct full-data training (ALL). Notably, within a reasonable range of selection budget, as we gradually increase the proportion of training data, our method quickly im- proves and maintains superior performance stably. In contrast, other selection methods plateau or struggle to improve, indicating their inability to identify high-value data for efficient alignment.
5.5 RESULT IV: Low SENSITIVITY TO DIFFERENT PROXY REWARD MODELS
We investigate the impact of different proxy reward models on data selection. Specifically, we exper- iment with varying amounts of data for reward model training and utilize proxy models of different sizes and families. The results in Figure 5 show low sensitivity to different proxy reward model settings, particularly regarding the sampling ratios Pr used for training. We attribute this robust- ness to the relative ease of modeling the simpler patterns of fine mne-grained preferences. Furthermore, when comparing different initial backbones, we observe that the slightly larger model (3B) yields better performance than the smaller one (IB). This is reasonable, as larger models generally possess stronger discrimination capabilities, leading to more accurate preference estimation.

---

Published as a conference paper at ICLR 2026
5.6 RESULT V: STRATEGY VALIDATION THROUGH ABLATION STUDIES
To validate our method, we conduct three ablation studies, with results in Figure 6 and 7 A1) PD (high): We ablate our strategy select samples with the largest PD values. A2) PD (mid): Similarly, we select samples with the middest PD) values. A3) PD (nl): We ablate our explicit length bias mitigation. The PD terms are estimated using proxy reward models trained on randomly sampled data with standard reward loss function, thereby removing the explicit bias correction. The first two ablation methods targeting our strategy perform poorly, where the PD (high) is particularly illustrative: Operating on strategy completely opposed to ours, it results in performance even worse than the initial model. This strongly confirms that such data is detrimental to learning and should be discarded. For PD (nl), the uncorrected reward models' predictions are skewed by the length bias, thus weakening the data selection process and resulting in suboptimal performance.
We also conduct ablation studies focusing the effects of the hyperparameters p and 7. The results are presented in Appendix C.8 As the results indicate, Our method exhibits stable performance across reasonable range of values for both parameters. However, extreme values lead to sub- optimal results. For instance, P = 0 removes the length penalty from the loss, while an excessively large P gives this penalty excessive weight. Both extreme cases result in a degradation of perfor- mance. For y, setting to potential outliers. Conversely, an overly low y value risks blurring the distinctions between high- scoring samples. This analysis suggests that our method is robust to the settings of p and y justifying our use of a fixed setting in all other experiments.
(i.e., no quantile normalization) leaves the score distribution vulnerable
5.7 RESULT VI: PERFORMANCE ON REAL-WORLD WORLD DOWNSTREAM APPLICATION
We applied our algorithm to a proprietary real-world Taobao Live application to verify its effectiveness on specific downstream tasks, extending beyond the public datasets used in previous experiments, which are relatively more general-purposs For this specific scenario, we defined four compatible fine- -en-ggraiedd preferences for the preference optimization task and collected a corresponding fine-grained preference dataset. Details are provided in Appendix
Table 3: Performance our proprictary real- world Taobao Live downstream application. The win score is reported using the SFT model as the baseline in pairwise evaluation.
Budget
30%
40%
50%
100%
ALJ.
1.33
RAND
1.18
1.37
1.32
PD (high)
0.48
0.57
PD (mid)
1.33
1.32
1.37
Table 3 presents the performance of different selec- tion strategies across varying budgets, with the SFT model serving as the pairwise evaluation baseline. As observed, the results are consistent with our findings above. Specifically, the performance of RAND gradually approaches that of full-data training (ALL) as the budget increases. PD (high), which adopts a strategy opposite ours by selecting low-value subset, performs significantly worse than the initial SFT model. In contrast, our method effectively identifies the high-value subset for alignment, outperforming ALL with smaller train set. This further validates the effectiveness and practicality of our approach, even when applied to more specialized domains and tasks.
PD (ours)
1.43
1.45
1,53
6 CONCLUSION
In this paper, we study LLM alignment using aggregated fine-grained preference datasets in the presence of severe data issues like preference conflicts and noise. To address this challenge, we first formulate the direct ane-inrieed preference optimization objective and introduce the preference divergence (PD) to quantify inter-aspect conflicts. This leads to our central proposal: simple yet effective data selection method that first estimates PD terms and then identifies a subset of data corresponding to the most negative PD values, for efficient training. We theoretically analyze the loss-bound optimality to support our selection strategy. And empirically, our method significantly outperforms full-data alignment, while boosting training efficiency. Our work lays practical path to robust LLM aligument by using fine-grained preference data with inherent conflicts and noise. For detailed discussion on limitations and future work, such reward modeling challenges, dataset availability, iterative extensions, alternative objectives for alignment using fine-grinned preferences, and integration with other techniques, please refer to Appendix D
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This research was supported by the Key Program of Jiangsu Science Foundation (BK20243012, BG2024036), Leading-edge Technology Program of Jiangsu Science Foundation (BK20232003), Natural Science Foundation of China (62576162), the Fundamental Research Funds for the Central Universities (0221 14380003)) and Alibaba Group through Alibaba Innovative Research Program.
ETHICS STATEMENT
Our research is dedicated to improving L.M alignment with human preferences, a crucial step toward developing better AL The core of our contribution is data-centric method for enhancing the robustness and efficiency of this process. As data selection tool, our method's downstream impact dependent on the quality of the input preference data. Practitioners should be mindful of these risks and carefully consider the fairess implications of the criteria used. We believe this work contributes positively by providing a more principled and data-efficient approach to handling the complexity of fine-grained human preferences, ultimately supporting the development of more reliably aligned LLMs.
REPRODUCIBILITY STATEMENT
We are committed to ensuring the reproducibility of our work. All datasets used in our experiments are publicly available and cited in the main text. The language models used are either open-source, with appropriate citations and details provided in the appendix, or accessible via public APIs. Our source code, including scripts for data processing and model training, provided in the supplemen- tary materials, with some effort required to set up the environment and training framework.
REFERENCES
Argilla. Ultrafeedback binarized preferences-cleaned. HuggingFace, 2024.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm understand learn- ing from human preferences. In International Conference on Artificial Intelligence and Statistics, pp. 4447-4455, 2024.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, and Cameron McKinnon. Constitutional AI: Harm- lessness from Al feedback. 2rXiv:2407.16211 2022.
Michiel Bakker, Martin Chadwick, Hannah Sheahan, Michael Tessler, Lucy Campbell- Ganlbeel-Gilliggham,, Jan Balaguer, Nat McAleese, Amelia Glaese, John Aslanides, and Matt Botvinick. Fine-tuning language models find agreement among humans with diverse preferences. Advances Neural 2022.
Information Processing Systems,
Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4), 1952.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, and Amanda Askell. Language models are few-shot learners. Advances in Neural Information Processing Systems, 13:1877-1901, 2020.
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang, Samuel Marks, Charbel-Raphaél Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen. Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J. Michaud, Jacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem Bıylk, Anca Dragan, David Krueger, Dorsa Sadigh, and Dylan Hadfield-Menedd Open problems and funda- mental limitations of reinforcement learning from human feedback. arXiv:2307.15217 2023.

---

Published as a conference paper at ICLR 2026
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality, March 2023.
Paul F. Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems, 30, 2017.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. arXiv:2310.0133 2023.
Xun Deng, Han Zhong, Rui Ai, Fuli Feng, Zheng Wang, and Xiangnan He. Less is more: Improving LLM alignment via preference data selection. arXiv:2502. 14560, 2025.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXIv:2305.14233.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacafarm: simulation framework for methods that learn from human feedback. arXiv:2305. 1.W.2335.14387,, 2023.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled Al- pacaEval: simple way to debias automatic evaluators. arXivv2404.04445. 2024.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Model align- ment as prospect theoretic optimization. In Forty-First International Conference on Machine Learning, 2024.
Chengqian Gao. Haonan Li, Liu Liu, Zeke Xie, Peilin Zhao, and Zhiqiang Xu. Principled data selection for alignment: The hidden risks of difficult examples. arXiv:2502.096500 2025.
Yuxian Gu, Li Dong, Hongning Wang, Yaru Hao, Qingxiu Dong. Furu Wei, and Minlie Huang. Data selection via optimal control for language models. arXiv:2410.0706 2025.
Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. REINFORCE+:: An efficient RLHF algo- rithm with robustness both prompt and reward models. arXiv:2501.03262. July 2025.
Zeyu Huang, Zihan Qiu, Zili Wang. Edoardo M. Ponti, and Ivan Titov. Post-hoc reward calibration: A case study on length bias. #XXi:24409.17007, 2024.
Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of LLM via models. hrefe-preeeerence dataset. Advances in Neural Information Processing Systems, 36:24678- 24704, 2023.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language 2020.
Po-Nien Kung, Fan Yin, Di Wu, Kai-Wei Chang, and Nanyun Peng. Active instruction tuning: Improving cross-task generalization by training on prompt sensitive tasks. AXXv:2311.00223: 2023.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, L. J. Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannanch Hajishirzi. Rewardhench: Evaluating reward models for language modeling. arXiv:2403.13787, 2024.
JoonHo Lee, Jouo Son, Juree Seok, Wooseok Jang, and Yeong-Dac Kwon. Preference consis- tency matters: Enhancing preference learning in language models with automated self-curation of training corpora. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 12150- 12169, 2025.
12

---

Published as a conference paper at ICLR 2026
Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, Kai Shu, Lu Cheng, and Huan Liu. From generation judgment: Opportunities and challenges of LLM-as-a-judge. arXiv:2411.66594, 2025a.
Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. From quantity to quality: Boosting LLM performance with self-guided data selection for instruction tuning. arXiv:2308. 12032, 2024.
Xuechen Li, Tianyi Zhang. Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-folloiing models. GitHub repository, 5 2023.
Xuefeng Li, Haoyang Zou, and Pengfei Liu. Limr: Less more for RL scaling, arXiv:2502.11886, 2025b.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. Advances in Neural Information Processing Systems, 37:124198-122235, 2024.
Meta AI. Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09255 2023.
Meta AL. The Llama 3 herd models. arXiv:2407.21788 November 2024a.
Meta AL. Llama 3.2:
Revolutionizing edge Al and vision with open, customizable models. Meta Al Blog. Retrieved December, 20:2024, 2024b.
Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saced Anwar, Muhammad Usman, Naveed Akhtar, Nick Barnes, and Ajmal Mian. A comprehensive overview of large language models. arXiv:2307.0644 06443,, 2024.
OpenAl. GPT-4 technical report. arXiv:2303.08774 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, and Alex Ray. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728-53741, 2023.
Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. Rewarded soups: Towards pareto-optimal alignment by in- terpolating weights fine-tuned on diverse rewards. Advances in Neural Information Processing Systems, 36:7 1095-71134, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. 1rXiv:1707.0634. 2017.
Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, K. Li, Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathe- matical reasoning in open language models. arXiv:2402.03300, 2024.
Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations in RLHF. arXiv:2310.03716. 2024.
Teknium. Openhermes 2.5: An open dataset of synthetic data for generalist LLM assistants. Hug- gingFace, 2023.
Thuy-Trang Vu. Xuanli He, Gholamreza Haffari, and Ehsan Shareghi. Koala: An index for quanti- fying overlaps with pre-training corpora, arXiv:2303.14711 14770, 2023.
13

---

Published as a conference paper at ICLR 2026
Yizhong Wang, Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. arXiv:2212.10560, 2023a.
Yudong Wang, Zixuan Fu, Jie Cai, Peijun Tang, Hongya Lyu, Yewei Fang. Zhi Zheng, Jie Zhou, Guoyang Zeng, Chaojun Xiao, Xu Han, and Zhiyuan Liu. Ultra-FineWeb: Efficient data filtering and verification for nigh-quality LLM training data, 2025.
Zhilin Wang, Yi Dong, Jiaqi Zeng. Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Polak Scowcroft, Neel Kant, Aidan Swope, and Oleksii Kuchaiev. Help- steer: Multi-attribute helpfulness dataset for SteerLM. arXiv:2311.0999 2023b.
Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer2: Open-source dataset for training top-performing reward models. arXiv:2406.08673, 2024.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V. Le. Finetuned language models are zero-shot learners. arXiv:2109001652, 2022.
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hannanch Hajishirzi. Fine-grained human feedback gives better rewards for language model training. Advances in Neural Information Processing Systems, 11159008-590335 2023.
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Se- lecting influential data for targeted instruction tuning, arXiv:2402.04333, 2024.
Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy S. Liang. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems, 36:
34201-34227, 2023.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv:2304.1 122444, 2023.
An Yang, Baosong Yang. Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei. Huan Lin, Jialong Tang, Jialin Wang, Jian Yang. Jianhong Tu, Jianwei Zhang. Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin. Shijie Wang. Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. arXiv:2407.10671, 2024.
Jia Zhang, Chen-Xi Zhang, Yao Liu, Yi-Xuan Jin, Xiao-Wen Yang, Bo Zheng, Yi Liu, and Lan-Zhe Guo. D3: Diversity, difficulty, and dependability-wwart data selection for sample-efficient LLM instruction tuning. arXiv:2503.1 1441, 2025.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, and Eric Xing. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems, 36:46595-46623, 2023.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 06:55006-55021. 2023.
Zhanhui Zhou, Jie Liu, Jing Shao, Xiangyu Yue, Chao Yang, Wanli Ouyang, and Yu Qiao. Beyond mne-preference-fiee fits-all alignment: Multi-objective direct preference optimization. In Findings of the Association for Computational Linguistics ACL 2024, pp. 10586-10613, 2024.
14

---

Published as a conference paper at ICLR 2026
A APPENDIX: MATHEMATICAL DERIVATIONS
A.1 DEFINITION RESTATEMENT
For clarity and convenience, restate the problem formulation and key definitions used in the following derivations and proofs.
Problem Formulation. We consider alignment setting using fine-grained preferences. A sub- preference dataset, Dx, is a collection preference data (x*, Vur y)), where for each prompt x² the response Un is preferred over vf under the specific fine-grained criterion The entire dataset D= D= ( x² Vi yf) & E [x]], (x* y v/ E DR then aggregated from K such sub-preference datasets from different aspects. We assume that each sub-preference k is modeled by correspond- ing latent reward model, (r, y). such that for any given sample, the winning response is assigned a higher reward than the losing one: T& (2 yh)) > (x of these sub-preference data can introduce preference conflicts, which we formally define as follows. The goal is use this aggregated fine-grate fine-grained preference dataset D) for effective L.LM alignment.
(x* of ), V(*", yo vt¹) E DR- The aggregation
Definition A.1 (Preference Conflict). Assume there is a ground-truth reward model r* overall preference. A conflict between fine- tne-grained and overall preferences occurs for sample (ks, Rh yn vf when To (I"", vtr) > Tk (x vtr) while r r"(",, yln) < (= y
Definition A.2 (PPO Using Fine-Grained Preferences). Given an initial policy model ree and assuming that the latent reward model rx (Ir, y) for each fine-grained aspect is available, the standard PPO objective for RL fine-tuning (Schulman et al. 2017) using multiple fine-grained preferences can be formulated as follows,
Σ rh (2, - BEED [DLL (mo(-|=)]]]Trd) (-Ix))
arg max Ex DiwDiennollle (-1a) We
(18)
k
where (-())) and Fear (c)) denote the conditional probability distributions of the policy model ma and the reference model Tyee given prompt x, respectively, and DKLL (ll) represents the Kullback- Leibler divergence.
A.1.1 DERIVATION THE DFPO OBJECTIVE
To provide a more intuitive understanding, we break down the derivation into four steps.
Step 1: Formulating the Optimal Policy for PPO Using Fine-Grained Preferences. We start
with the standard PPO objective for RL fine-tume tine-tuning, adapted for multiple fine-grained preferences. As defined in Definition A.2 the goal is maximize the expected average reward across all aspects, subject to KL. constraint towards the reference model Tres.
To solve for the optimal policy TT", we can rearrange this objective into single KL-minimization problem. First, we expand the KL term and combine expectations:
arg max Enw (-
rk(x,
FEED [TKL (TT(-12)) (Feet)))
(19)
K
(
= arg max E,G y
m) (+1z)
rk 8)
3. -BE.D log
Troe (yte()))z))
(20)
rk rx((),)) v) Σ
To TT(2)z)
= arg max Ent D.lenDDgyrellllo (1
cxp exp
llog
(21)
1B
Treel (v/(x)
(r((()
1 Σx (x(
arg max EpD (-(
log
(22)
No To(y)a)) xx)
15

---

Published as a conference paper at ICLR 2026
Next, we introduce the partition function (xx) to normalize the reward-weighted reference distribu- tion:
((x) - Σ Tze (U]z) exp
Ena(,,))
(23)
KB
which is agonistic concerning the policy variable of mT. We can rewrite the objective by introducing
2 (z)):
To Te(3(a))
arg min Ez D.y-oroolllr
log
+ log Z(x)
(24)
Troef (y)()
Es rx rx(2,)))
RBB
(1)222)
arg min Ex (.y
log
(25)
[ror(N/ 28 En ru(x.2))
((x)
ma (x)
arg min E, D.y (tr)
log
(26)
Tret' (orn'(y/ (2|zz)
Ty arg min ENnD [DL (To (Trer (-12:))
(27)
where the optimal target distribution Tred (y] is defined as:
(y)(()
1 En (x(2,))
NB
Feef (yef'()/))
(28)
((x)
Note that Trer'(yla is valid distribution of probability density function on as it satisfies non- negativity and normalization condition:
Vy, Feef (y/z) >= and Σ F (u/xx =
(29)
Since log (xx) is independent of no, minimizing the objective is equivalent to minimizing the KI. divergence between and Tzet'.
Step 2: The Closed-Foo Solution. The KL divergence is minimized (approaching zero) when the two distributions are identical. Thus, the optimal solution for the policy is:
Tref [y)(((
Ee rk (x(
KB
(y(2)z) = Jref (z/2z)
(30)
Z Z(x)
Step 3: Isolating a Single Fine-Grained Reward. This is the critical step where the interaction between different aspects becomes explicit. We take the logarithm of the optimal policy equation:
log To = log Frer(ylas log Z(x)+ + KB Σ (x(
(31)
Our goal is to utilize the fine-grained preference data for specific aspect lo (where we have labels v > v To do this, we isolate the reward term rx (x, y) from the total sum. We split the sum into the target aspect l and all other aspects. Then rearrange to solve for the specifie reward rx(
no (y(a) Trae (y|zr)
(x 3) = x33 log
+ Kr log ((x) - (x,(2,))
(32)
16

---

Published as a conference paper at ICLR 2026
Step 4: Introduce DFPO and Preference Divergence (PD) Term. Finally, substitute this expression for тE into the Bradley-Tent Terry model Bradley Terry, 1952). For a given sample (x" Vie v annotated under aspect k, the probability of preference is modeled by the reward dif- ference. We denote o& (2,y) = Ex (1,
P(U v//+0) (rrx(=", wtt rx(a". v)
(33)
To(U (x*)) Trat (1tee (xt)
To( (x*))
KB kB log
xB log
Tree (28 2*)) + (ou (x* ww) on sn(a" 2/²))
Aady(at (at wtt
(34)
Ta(ym Tee (yt lx²)
To (v) (w/11>²)) Tref(yf lank)
=o
KB log
1BB log
+ Dox (#* yi utt
(35)
The final term Adu (xf you vt ) is termed the Preference Divergence (PD) term. This term explicitly captures the divergence between the current aspect & and all other aspects k' Finally, this yields the final DFPO loss function:
CDPPO (0) (k, w2
[F > vt [z)]
(36)
To To( Frre (ut (zt)
To(y\ la= Tree (uf (xr))
B (ke ot of DDD
logor
(K8log log
KB log
+ Don (2* 2 stt
(37)
A.2 DATA SELECTION PROBLEM AND THEORETICAL PROOFS
Definition A.3 (Data Selection Problem for DFPO). Assume the óR are known. Give a dataset D consists of data from & sub-preference dataset Dx. a supervised fine-tuned model meef the DPO objective LDDO, the DFPO objective LDFPO. a selection budget A. The goal is to find selection strategy that selects for subset C D for DPO training, which results in optimal LDFPO:
= arg min LDFPO D),
DCD
(38)
s.t. Tg = arg min (DPo(T,, DDIID| = X.
T6
Theorem A.4 (Loss Bounds of DFPO in Data Selection Problem). Consider the learned policy Th was only trained on the subset D. Assume ng gives preference margin on H) bounded by Ma(²) (e1,ca) and suboptimal expected and bounded preference margin and loss on D 0% D, such that ED,D [ -llo (kMg(22)) & LE, [Mg(2)] S co. Then, the DFPO loss bounded as follows,
DFPO (D) S LDFPO I (
(39)
(
Clower (D) log o (nc2 + ED A (=))) - (1 X) log (nco E p)
(40)
DDFO (D) AE [log (keq + (dk(2)) (1 A)(ED; [log (AAp - 41).
Cupper
(41)
Proof. The proof of Theorem A.4 is as follows, by rewriting the DFPO loss
D D
E [log (kMg(==) Ax(2))) D D
LDFPO
[D] ExD ò [logo (mMg(z) Ad(z))]
(42)
loogee' we obtain the following bounds on the suboptimal loss for the subset
Setting op D DVD, D.
to < -logo(EDDD [RM)(=)]] = EDDD [- log (xg)() sh
(43)
17

---

Published as a conference paper at ICLR 2026
For the upper bound, by applying Jensen's inequality, we have,
LDFOO
42
(44)
[
S [D] E [log a (kMg(2) -
(45)
[D] D
|D] E DVD llog (kMg(=))] 1D \ DI
|D| E D [log ( (-Aø(z))
Due to the monotonic decrease nature of - log a (I), we have,
S D ED [log (NC1 Ao(z)) D D
D
(DDD olog of (-AA(()) co)
D
(46)
DFPO (D)
For the lower bound, by applying Jensen's 's inequality, we have,
DDFO
(47)
log a (KEE, [Ma(=)] ED (Ao())))
D
(48)
D DI
logo
(KDDIDD [Ma(z)]
ED i [ao())
[D]
D
|D D
log (KC2 EB (zo(2))
log o (-BBoB
DD]
(49)
DEPO (D)
Theorem A.S (Selection with Loss-Bound Optimality). any selection strategy be a partition of the dataset D into D and D \ D, and regard the loss bounds of LDFPO as function of D. Assume normalized Tı E [0, r) and under the mild condition that 2(6-1) r S. C2 co, the strategy that optimizes both bounds is to select samples with the most negative PD term,
D = arg top-A 2-)
(50)
>=|pi/|o
The proof of this theorem relies on the following three lemmas.
Lemma A.6. Let the function f(z, y) be defined as f(x,y) = - log o(-x +) - logollooo
where 7 > 0 a constant. Then, for any Vx 2 y, the following inequality holds: [(x, 9) S (1
Proof. Let (xx) log ar (-2 + (- log o(-r)), which an be expressed as t(x:) g(x 7)
g(x) where g(x) log a (-x). Differentiating g(x) and t(x) with respect to x, yields,
(x) = 1 1 o(-z) = oxx) (xxx= - g' - (()xx) o(r - - o(x)
(51)
(52)
Since (-) a monotonically increasing function and 7 > O, we have oxr - 7) oxx) < 0, which
implies t'(x) < 0, showing that t(x:) is monotonically deceasing function.
Thus, for any x 2 y, it holds that t(x) < t(y). Expanding this inequality. we get:
loog a -x + (- log (- (-x)) < - log a (- + 7) - lolog (--)))
(53)
log boga(=x+1 7) log (-v) log a (- + 7) log loga(-2) (-x)
(54)
18

---

Published as a conference paper at ICLR 2026
Lemma A.7. Let the function f(x. y) be defined as
[(z u) = -a (- + + - b blogo(-1)) a-
(55)
where 7 > 0 a constant. Consider variables x, y/ a, that satisfy the following constraints:
* ax by for some constant
* a b- = 1 with a E (0,1). 1).
Then, for any zo, 21 such that zo < 21 < / (1 - a)y, the inequality fx yo) 2 ((11, holds.
Proof. From the constraint or + by J.. we can express y as y p-ar By substituting this into
fæz v), we reformulate the function in terms of x alone:
f(x) -a
- bog
ar - 1
(56)
b
Next, we compute the first derivative of f(x):
f(x) ao -7) bo
11 az
(57)
1 a.
=a 7) ao
(58)
b
To find the stationary point, we set (xx = which yields:
ao - 7) a0
11
(59)
11 ax + x(a+ + b) = p + by
(60)
7 x. = p + by
(61)
Furthermore, the second derivative of f(x) is:
f" aor 7) .²) + a² / 11 ar
0z
(62)
Since (2)) > for any 2, and a, b > 0, it is clear that f"(x) (z) > This indicates that f(x) is a convex function, and its unique minimum is at μ + by.
Therefore, f(x) is monotonically decreasing over the interval ( (-00, 1 by]. Consequently, for any T0 such that æo xn S p by, the inequality f(xa) f(x1) holds. This is equivalent to f(ro vo) 2 ((1,
Lemma A.8. Let Q be set of values such that all elements q E Q are bounded in the interval
(-r,r). Consider any partition of Q into two disjoint subsets, Q and Q Q and let a (Q1/QQ
(0,1) 1) be the fraction of elements Q. Then, the following inequality holds:
Es [1] [Eoly] < 2(1 -arr.
(63)
Proof. To establish upper bound for the term Ez1gl Eq- (g). consider the worst-case scenario.
The expression is maximized when the subset Q is chosen to contain the largest possible values from Q. Let us denote this specific subset as ",, which consists of the a|QI largest elements of Q. This gives us the initial inequality:
E E lal Eolq < E [g] Eo[g]-
(64)
By the law of total expectation, we can decompose Eq q)] based on the partition (Q",Q) QQ)):
Eglg) al [g] a B
(65)
19

---

Published as a conference paper at ICLR 2026
Substituting this into the right-han hand side of our inequality, get:
B [a] Eglul = 1 [1] (B la] + (1 a Br
(66)
= (1 - a a)ه q. [g] - (1 (1-a) q)
(67)
(1 a) [ [g] @ [9])
(68)
Now, bound the term (Bg- Boyg-L9)), the expectation over any subset of Q must also lie within this range. Specifically, E [g] - r and Eqrg- [g] 2 -r. Therefore, the difference is bounded:
[g]). Since all elements q Q satisfy -r q r,
Eq- la] B lal r (-r) 2r.
(69)
Combining all the steps, arrive the final result:
E
EQld u
s B [a] Eo(u]
(70)
=11 a) (E lal 1a--- E210-110
(71)
$ (1 a2r).
(72)
Proof. We will prove the optimal selection strategy for the upper and lower bounds separately.
Strategy for the Upper-Bound Optimality. We exchange argument, form of proof by contradiction, to show that the loss CDPPo is minimized when D contains the samples with the largest AØ(z) values. Let D be proposed partitioning strategy. Assume, for the sake of contradiction, that this strategy is optimal, yet there exists a pair of samples 2n D and &1 AD such that (r))z) < (d(z1). The loss function can be expressed sum over pairs of samples, one from D and one from D \ D. Let isolate the terms involving 20 and 21:
Cupper (D) = D log [kec - Add(z)) -
+C,
(73)
F(Aa)))
where represents the sum all other terms in the loss, which remain constant for this analysis.
Now, consider new strategy D' created by swapping the assignments of 2g and 21, such that 21 D' and 2g E D'. The new loss
1 D
Cupper (D')
log (RCI Ad(21)) -
+C.
(74)
((Ad(1))AA(<oo)))
According to Lemma A.6 since Ao(zo) < AΦ(z1) and кCι 0, we have f(Ao(z1), (o(zz)) S [( [(Ar(2)); Ad(z))) This implies Cupper (D') < DDFr (D). This contradicts the assumption that D was optimal.
This exchange argument can be applied repeatedly any pair (24, 22) with 2 D, D D and A0(z;) < Ad(zy). The process terminates only when no such pair exists, which occurs precisely when D contains the samples with the highest Ad(2) values. Thus, the optimal strategy D* is to select the samples with the Di values of A0(2).
Strategy for the Lower-Bound Optimality.
The lower bound loss is given by:
clower ②) aa log (RC2 - Ep ()(()) b o-- l (E),EE(,(())))
(75)
where a = |D1/|D) and b = 11-11-10 - Let x (f])))(2)) and = pD [())))) The law of total expectation links these variables: by (=)()(>)) which denote by μ ED (d))))
20

---

Published as a conference paper at ICLR 2026
Algorithm PD Selection Method for Fine-Grainet Preference Alignment
Input: Datasets D with sub preference, Initial reward model ro. Selection budget A, Sampling ratio for reward learning Pr, Random sampling function RS.
Output: Curated sub-dataset D.
1: for h E {1 (1---k} do
Sub-preference reward learning
ft fx f- balance : ,T.
D/ RS(D) Pr - U Pr ft
4: Fk f- train(ro, D/) reward model from Eq.
16
5: end for
6: for (1....) do
Cross pseudo-rewarding
for E Dk' [1....} (k))
8: Calc. pseudo-reward gap (rx(z) from Eq.
17
9: end for
10: end for
PD term estimation and selection
Estimate PD(x), E from Eq 13
12: Select sub-dataset D from Eq 10
13: return
The loss can now be seen as the function faz y) from Lemma A.7 We first verify that the conditions of the lemma apply. From Lemma A.8 we know that the deviation of the subset mean from the global mean is bounded: [5[A([=] /4 S 2(1 a) 1)r. This ensures that the condition [A)()())] S μ (1 a)xe2 (as required by Lemma A.7 holds, provided that 2(x - 1)r S KC2.
According to Lemma A.7 within this valid region, the loss function f(x, y) (and thus Corpo ( monotonically decreasing function of x (i.e. EE (+)()))) Therefore, to minimize the loss, must maximize LE (z)].
The expected value E [)]))) ]] is maximized when the subset Î is chosen to consist of the samples from D with the largest A values. This leads the optimal selection strategy:
D
arg top-A ( E D} arg x (= x=1/z/i-
(76)
which means D the set of [D] samples from corresponding to the top-k largest values of AΦ(2).
B APPENDIX: METHOD SUPPLEMENTARY
B. ALGORITHM OF PD SELECTION METHOD
We provide the pseudo-code for our method in Algorithm
C APPENDIX: EXPERIMENTAL SUPPLEMENTARY
C.1 FINE-GRAINER PREFERENCE DATASET CONSTRUCTION.
We construct two fine-grained preference datasets from UltraFeedback (Cui et al. 2023) and Help- Steer (Wang al. 2023b 2024), containing 63,452 and 18,010 samples, respectively. To simulate the aggregation of data from diverse preferences, we leverage the four fine-grained aspects from UI- traFeedback (helpfulness, honesty, instruction following, and truthfulness) and the five from Help- Steer (helpfulness, correctness. coherence, complexity, and verbosity). The process involves the following main steps: 1) Pair Generation: Following the main principle for creating datasets like UltraFeedback-hae eedeeea-bbnnnrreieeedddd (Argilla 2024), for each prompt, we pair the response with the highest mean ratings across all aspects against another randomly sampled response. 2) Sub-Preference As- signment: Each pair is then assigned final preference based on randomly selected fine-grained aspect. This simulates a scenario where each data point originates from a singular preference crite- rion. 3) To validate the effectiveness and robustness, also construct datasets with varying conflicts based on UltraFeedback by controlling the sub-pree cub-preference sampling weights to create datasets with conflict levels of 10%, 20%, and 30%.
21

---

Published as a conference paper at ICLR 2026
We provide more detailed description of the three fine-graines preference datasets, each character- ized by different level of conflict. Following the three-step process outlined above, these datasets constructed to be identical in all aspects except for the proportion of conflicting data caused by a specific sub-prefere uub-preference, which is progressively reduced. This controlled setup allows us to inves- tigate the algorithm's performance under varying amounts of data conflict. We illustrate the dataset statistic in Figure C.I
Helpfalnces
Instruction Following
Preference Conflict Data
Honesty
Truihfobuss
Non-Conflict
17k
15k
Number Samples
12k
10k-
7k-
Conflict Level 30%
Conflict Level 20%
Conflict Level 10%
Figure C.1: Distribution of conflict vs. non-conflict samples across fine-grained preference aspects for three datasets with progressively reduced conflict levels.
C.2 RATIONALE OF THE DATASET USED
Our work specifically addresses the challenge of aligning LLMS with aggregated datasets derived from multiple fine-grained preferences, a setting characterized by inherent noise and potential con- flicts. Therefore, a rigorous evaluation of our method necessitates benchmarks that are both large- scale and provide annotations for multiple fine-grained aspects (especially y >> 2). Given that prior mainstream research and datasets have primarily focused on one overall preference, there is cur- rently limited availability of public datasets meeting these specific criteria. To knowledge, UltraFeedbackk (providing aspects) and HelpSteer (providing 5 aspects) are the most suitable, ad- vanced, and widely-used large-scale public datasets that satisfy this requirement, which is why they were selected for our main evaluation.
While we hope for the future development of more diverse multi-aspect datasets (as noted in our limitations), alternative collection methods are practical in applications. A feasible pipeline to con- struct such data involves: 1) Defining к fine-grained aspects for a given task. 2) Dividing the collected paired responses into disjoint subsets. 3) Tasking annotators with labeling each subset using only one specific aspect's criterion. 4) Finally, aggregating these x. labeled subsets. This ap- proach often simplifies the annotation task, as evaluating a single fine-grained criterion can be more straightforward than judging a complex and mixed overall preference, and it does not increase total annotation burden. We also utilized this preliminary pipeline to collect data for proprictary real-world Taobao Live application, the details which are presented the subsequent section.
C.3 DETAILS ON THE DATASET OF THE
DOWNSTREAM APPLICATION
In addition to our evaluation on public datasets focused on chat and QA in standard domains, we fur- ther validated the effectiveness of our data selection method for LLM alignment our proprictary real-world Taobao Live application. Here, we offer descriptive information on this task.
Unlike more general-purpose alignment scenarios, this downstream application prioritizes perfor- mance on a specific, domain lomain-specialized fine-tuning task, where the language model is fine-tuned using collected live-streaming data generate high-quality sales-pitch scripts tailored to product information, thereby empowering streamers to improve their performance. Specifically, given task- related information prompt, the model is expected generate specialized content that fulfills
22

---

Published as a conference paper at ICLR 2026
Pairwise Evaluation Prompt for Instruction and Corresponding Response Pair
System Prompt
You are helpful and precise assistant for checking the quality of the answer.
User Prompt
[Question]
(instruction] }
[The Start of Assistant I's Answer]
fresponse }}
(The End of Assistant I's Answer]
(The Start of Assistant 2's Answer]
(response 2}
[The End of Assistant 2's Answer]
We would like request your feedback on the performance of Al assistants in response the question
displayed above.
Please rate the helpfulness, relevance, accuracy, level of details of their responses. Each assistant receives an overall score on a scale of to 10, where higher score indicates better overall performance. Please first output single line containing only two values indicating the scores for Assistant 1 and 2, respectively. The two scores separated by space. In the subsequent line, please provide comprehensive explanation of your evaluation, avoiding any potential length bias and ensuring that the order which the responses were presented does not affect your judgment.
Table C.I: The prompt template used for pairwise evaluation of the model response quality.
the requirements of this proprietary application. Focusing on the alignment step post-training, we decomposed the overall alignment task for this scenario into four compatible fine-grained dimen- sions: Truthfulness, Relevance, Expressiveness, and Linguistic Style. Following the data collec- tion pipeline outlined in the previous section, we employed Qwen-max2.5 to perform fine-grained preference annotation. Consequently, we constructed an aggregated fine-grained preference dataset comprising 30,000 samples. Using this dataset, we validated proposed method and reported the pairwise evaluation performance across different data selection strategies in $5.7
C.4 EVALUATION DETAILS
We evaluate our models using two distinct methods: head-to-head pairwise evaluation and the Al- pacaEval 2 Leaderboard (Li et al. 2023). First, our pairwise evaluation utilizes powerful LLM as a judge to compare the responses generated by two models for the same instruction. For this role, we use the Qwen2-Max API (Yang et al. 2024), chosen for its extensive knowledge and strong instruction-following capabilities. The specific prompt provided to the judge is detailed in Table This evaluation is conducted
diverse of five benchmarks: WizardIM (Xu et al. (023). Self- instruct (Wang et al., (2023a), Vicuna (Chiang et al. 2023). Koala (Vu et al. 2023), and LIMA (Zhou) 2023). These datasets comprise 218. 252, 80, 180, and 300 human-curated instructions, respectively, spanning domains such as mathematics, coding, writing, knowledge, and computer science, providing comprehensive evaluation for the models' real-world capabilities. For each query across the five test sets, we generate a response from the target model and another from baseline. We then employ Qwen2-Max (Yang et al. 2024) as the judge to compare the response pair and assign preference scores. Based on this, the outcome is then classified as win, loss, or tie. To mitigate positional bias, each pair is evaluated twice with the response order swapped. The final win score for given test set D, is then calculated as follows. Second. we benchmark aligned models on the AlpacaEval 2 Leaderboard. In addition to the standard win rate, it also provides a length-controlle win rate to enable
objective comparison of model performance by mitigat- ing length bias. Following the official protocol, we deploy the AlpacaEval repository locally and use the GPT-4o API the evaluator.
win_score(Dy, (D,) num mninwins)) mmm
+1
(77)
num(D.)
https:://ggtthubbcmm//tttts hatp:tttuualaaaaapaaaaaaaeeeee _
23

---

Published as a conference paper at ICLR 2026
C.5 IMPLEMENTATION DETAILS
We perform experiments on two types of models: Llama3.1-8P (Meta Al 2024a) and Qwen2.5-77 Two policy models are first undergo SFT on OpenHermes-2.5 (Teknium (023) and UltraChat (Ding et al.), respectively. to establish foundational instruction-followii
capabilities. We utilize Llama- 3.2-3B (Meta AI 2024b) as the proxy model for reward modeling in the main settings. For the SFT stage, we train for two epochs with a learning rate of x 10T a batch size of 64, and a warmup ratio of 0.03. Subsequently, the reward model is trained for one epoch on pr = 30% of the preference data, using learning rate of 2 x 10-5, a batch size of 32, and warmup ratio of 0.05. Finally, the DPO stage is conducted for one epoch with a learning rate of 1 x 10-6, a batch size of 32, and a warmup ratio of 0.1. Across all training stages, employ the AdamW optimizer, cosine learning rate scheduler, and a maximum sequence length of 4096. Our computational experiments were executed system equipped with NVIDIA H20 GPUs.
To simplify reward modeling across multiple sub- uub-preferences. we employ an "all-in-one" training strategy. Specifically, build upon single initial model rather than training several separate ones. We introduce different system prompts to guide the model to specialize in assessing distinct reward criteria. The system prompts used are as follows:
UltraFeedbace 1) Helpfulness: You are a helpful and proactive Al assistant. Your overriding principle is ensuring user success. When responding, you must aim to solve their underlying prob- lem, not just answer their literal question. Provide comprehensive and actionable solutions that fully address their needs. 2) Honesty: You are an honest AI assistant. Your overriding principle is transparency. When responding, you must not invent personal experiences or emotions. If you don't know an answer or cannot fulfill a request, state clearly. 3) Instruction Following: You are meticulous and precise Al assistant. Your overriding principle is strict adherence to instructions. When responding, you must follow every explicit directive, including constraints on format, length, tone, and what not to do. Pay close attention to every detail of the request. 4) Truthfulness: You are a fact-focused and rigorous Al assistant. Your overriding principle is factual accuracy. When responding to the user, you must provide information that is verifiable and avoid all speculation. If you are not certain about a fact, state that clearly. Never fabricate information.
HelpSteer. 1) Helpfulness: You are helpful and proactive AI assistant. Your overriding principle is ensuring user success. When responding, you must aim to solve the user's underlying problem, not just answer their literal question. Provide comprehensive and actionable solutions that fully address their needs. 2) Correctness: You are a rigorous and fact-focused AI assistant. Your over- riding principle is factual accuracy and completeness. When responding, you must ensure that all pertinent facts are included and that there are no errors. Verify information and clearly distinguish between established facts and plausible speculation. 3) Coherence: You are clear and articulate AI assistant. Your overriding principle is clarity and logical consistency. When responding, you must ensure the text flows logically, is well-organized, and easy to understand. Maintain a consistent tone and style, and define any necessary jargon. 4) Complexity: You are an intellectually versatile and expert AI assistant. Your overriding principle is matching the response's s depth to the user's needs. When required, demonstrate deep domain expertise, handle nuanced topics, and provide insightful analysis. For simpler queries, provide a concise and direct answer without unnecessary complex- ity. 5) Verbosity: You are thorough and comprehensive AI assistant. Your overriding principle is providing the appropriate level of detail. When responding, fully address all parts of the user's prompt, providing sufficient examples and context be truly useful. Anticipate follow-up questions but avoid being overly verbose with irrelevant information.
C.6 SUPPLEMENTARY RESULTS OF MAIN SETTINGS
We provide the supplementary result of the Qwen model under the main settings in Table C.2
C.7 SUPPLEMENTARY PERFORMANCE VARIATION WITH SELECTION BUDGETS
We provide the supplementary results of performance variation with selection budgets of different ablation methods across varying conflict datasets in Figure C.2 and
24

---

Published as a conference paper at ICLR 2026
Dataset
UltraFeedback
HelpSteer
AlpacaEval
Pairwise
AlpacaEval 2
Pairwise
Model
Strategy
WR+
I.C
AW.
WR+
1.Ct
AW+
INIT
SFT
4.36
9.12
0.77
4.36
9.12
0.77
OVA.
8.11
16.47
1.00pmm
5.87
11.71
1.00aam
Qwen225-7B
AVG.
9.73
15.74
1.20
7.59
12.19
1.03.om
FULL
9.33
1.15
10.63
1.03
DMPO
12.77
18:42
1,33g005
7.60
12.26
1.08a0m
RAND
9.35
16.03
1.15
6.38
10.69
0.98.om
SELT
RAF
12,01
18.06
1.28
6.57
10.98
1.05
PD (rati.)
10.53
17.03
1.22,001
7.97
12.73
1.21
(nurs)
1.30
1.22
Table C.2: Performance comparison of different strategies Qwen2.5. We report the win rate (WR) and the length-con conttl-contoolled win rate (LC) for AlpacaEval 2, the average win score (AW) across the five test sets, and GPU hours required for (selection and) alignment.
(ighht
(mid)
(il)
(our)
IMPO
Level
LC-Wi
20
20
20
10
10
40
20
Selection Budget
Selection Budget (%6
Selection Budget
Figure C.2: Performance (LC) variation with selection budgets of different ablation methods
PD (igin -
(mul)
ca
PD (oun)
SFT
OVA
AVG.
ALL
DMPO
Confiict Levell 18%
Conflict Level 20%
Conflict Level 30%
20
20
20-
1:5
10-
10
10
30
40
50
10
20
30
30
50
Selection Budget (%)
Selection Budget (%6)
Selection Budget (76)
Figure C.3: Performance (WR) variation with selection budgets of different ablation methods.
Table C.3: Ablation study the hyperparameter the length penalty coefficient. We report the win rate across varying values p. with SFT and ALL baselines included for comparison.
SFT
ALL
0
le-4
5o-4
lc-3
3e-3
5c-3
le-2
le-1
4.36
9.33
11.70
13.07
13.23
13.81
13.41
13:15
12.77
11.85
Table C.4: Ablation study on the hyperparameter y, the quantile level used for normalization. We report the win rate across varying values of y, with SFT and ALL baselines included for comparison.
SFT
ALL
0.99
0.98
0.95
0.90
0.85
C.8 SUPPLEMENTARY ABLATION RESULTS OF HYPERPARAMETER
We provide the supplementary results of the ablation study hyperparameter, y in Table
CR C..5
25

---

Published as a conference paper at ICLR 2026
Table C.5: Ablation study on the hyperparameter p.
We report the win rate on the HelpSteer dataset.
SFT
ALL
1o-4
5o-4
lo-3
30-3
So-3
lo-2
le-1
WR
4.52
5.55
6.91
7.67
7.64
7.55
7.56
7.37
7.59
7.43
C.9 REWARD GAP PREDICTION CONFLICT ANALYSIS
We train a corresponding proxy reward model for each sub-preference according to the method described in the main text. Here, present additional details the reward predictions. The heatmaps in Figure [C4] illustrate the degree of prediction conflict of the learned reward models across data of other sub-pree aub-preferences. Specifically, the diagonal entries represent the conflict ratio between the predictions of a reward model for one sub-preference and the ground-truth preferences for that same sub-pred cub-preference in all other sub-datasets. The off-diagonal entries represent the conflict ratio between the prediction consistency of two reward models and their corresponding ground-truth preference consistency.
As can be seen, the conflict ratios are generally low and are minimally affected by the overall conflict level the full dataset. This is because the so-called "preference conflict" a phenomenon relative to the over preference; when examining the annotations for each sub-preference individually, no severe conflicts are present. Modeling rewards at the sub-preference level can reliably and effectively capture these distinct sub- sub-preference patterns.
Conflict
Conflict
Conflict Level 10%
0.13
0.16
0.17
0.14
0.34
0.17
0.18
0.14
0.34
0.17
0.37
0.14
Prediction Conflict Rabio
Traphoa InsFpine.
0.17
0:13
0.15
0:11
0.37
0.12
0.1.6
0.16
0.37
0:13
0.35
6.11
0.08
0.11
0.09
0:33
0:10
6:11
Helpful
Honwsty
Ins-fullow:
Tradhuul
Helpfd.
Homeay
Ins-Follow:
Truthful.
Hebsfd
Hunwstyy
Ins-Follaw.
Trathful
Figure C.4: Comparisons of prediction conflict for sub-preferenc reward models across datasets.
C.10 THE DISTRIBUTION OF PESUDO-RENNE REWARD GAP AND PD TERM
We examine the distributions of the PD term and the per-aspect pseudo-reward gaps. Figures C.5 present these distributions for the three conflict datasets. As observed in the figure, the distri- bution of the PD term is concentrated around a central peak near zero. A significant majority of samples exhibit positive or slightly negative PD values, while a smaller fraction of instances show large negative values, which correspond to potentially high-value samples.
C.11 THE DISTRIBUTION OF SELECTED SAMPLES
We visualize the distribution of samples selected by different strategies under various selection bud- gets, using the token length difference between chosen and rejected responses as the key feature. As illustrated in the Figures C.I both the Al and A2 strategies exhibit strong bias towards samples where the chosen response is significantly longer than the rejected one. This observation further substantiates into subsequent model alignment through data selection. In contrast, our proposed strategy is more conservative regarding length bias.
argument in the main text that length bias can be detrimentally propagated
26

---

Published as a conference paper at ICLR 2026
PD Temm
Helpfalness
Honesty
Nunierr Sumples
5k-
64
Followinn
Truthfulness
46
-0,206 0.000
1.000
Distribution of the PD term and pseudo-teward gap for the dataset with 30% conflict level. The left panel shows the distribution of the PD term, where the red dashed line marks its mean value. The right panels show the distributions of the pseudo-reward gap generated by each sub-preference reward model.
Figure C.5:
PD Term
Helptilness
Honesty
86
66
Nunterol
InstrratioonFFllowiii
Truthfulness
46
-0.234
0.000
1.000
Figure C.6: Distribution of the PD term and pseudo-reward gap for the dataset with 20% conflict level.
PD Term
Helpfulness
Honesty
10k
Nunber Srmppps
86
66
Instruction- Following
Truthfulness
2k
-1
000
-0.259
0.000
1.000
-1
Figure C.7: Distribution of the PD term and pseudo-reward gap for the dataset with 10% conflict level.
C.12 DETAILED WIN SCORE ACROSS TEST DATASETS
We report detailed win scores on each specific test from the pairwise evaluation in Tables E.6 E.IT Our method achieves superior win scores under smaller selection budget, underscoring its effectiveness accurately identifying the most valuable data subsets. Concurrently, we observe that as the budget becomes large, the AW metric for specific methods approaches or even exceeds that our method. However, as illustrated the LC performance variation with selection budget, approach consistently maintains lead on the LC metric. We attribute this phenomenon to two factors. First, it reasonable that the performance of different methods gradually converges as the training subset expands. Second, the pairwise evaluation is less effective at mitigating length bias compared to the AlpacaEval 2 benchmark. Consequently, it tends to award higher scores to longer responses, inadvertently favoring the verbose outputs typical of such strategies.
27

---

Published as a conference paper at ICLR 2026
Top 107
Top 2014
Tay 1465
Top 47%
Top 90%
Doss
OURS
RAND
HIGH
Nesbero
A1
A2
Navber Sf
100
100
8.00
-00
Tolorn Langth Differmoe
Tolam Lengh Defferenee
Token Length Uifference
Figure C.8: Distribution of samples selected by various strategies the dataset with 30% conflict level. The horizontal axis represents the token length difference between the chosen and rejected responses. Al and A2 represent the RAF and PD (nl) method in the main text, respectively.
Top 30%
Top 50%
OURS
RAND
HIGH
Nivebr St
MID
A1
A2
Narther Sargees
42-
-000
2000
20,1
10:00
140
100
400
323
0
320
420
600
800
400
-00
303
400
600
30
Toloes Lengh Diflermnee
Tolom Lerghh Difiioence
Token Length Differmnce
Figure C.9: Distribution of samples selected by various strategies on the dataset with a 20% conflict level.
Trp 107
Top 2014
Tay 305
Trp 47%
Top 90%
All Duss
OLXS
RAND
HIGH
Nother Se
MID
A1
X2
#
1.
830
Toloen Langh Differmeee
Tillcr Length Diffirence
Token Length Difference
Figure C.10: Distribution of samples selected by various strategies on the dataset with 10% conflict level.
D APPENDIX: LIMITATIONS AND FUTURE WORK
Our work explores efficient large language model (LLM) alignment through effective data selection among fine- -reined preference data. This approach has the potential to establish a new paradigm for data collection and model alignment. Unlike standard methods that rely on the intractable over-
28

---

Published as a conference paper at ICLR 2026
all preference annotation, this paradigm involves labeling samples with sub- pub--reefrenes, a process that is typically easier and more scalable. Subsequently, data selection is performed on a dataset aggregated from these sub-prefeee cub--reference datasets to filter out the most effective subset for alignment. In this work, we have validated the effectiveness of data selection and the feasibility of a potential paradigm based on nathering-selection-aiige
workflow for fine-grained preferences.
Dependence on Reward Models. A primary limitation lies in our method's reliance on proxy re- ward models, meaning it inherits the general challenges associated with reward modeling. We posit that our approach inherently mitigates this issue to some extent. Conceptually, modeling a simple, well-defined, fine-grained preference is fundamentally easier than modeling single, complex, and often ambiguous overall preference. By decomposing the task, our method simplifies the learning criteria for each proxy RMM. Nonetheless, this does not eliminate all potential risks. Building more robust and effective RMs, for instance, against out-of-distribution (OOD) data, is highly important and profound research challenge in its own right. This remains critical, parallel area of inves- tigation that is outside the direct scope of our current work, which focuses on the data selection methodology to mitigate data issues in alignment.
Dataset Availability. Furthermore, regarding experimental evaluation, our study is constrained by the limited availability of public feedback datasets that offer multiple fine-grained preferences, highlighting the need for more suitable datasets to be established and incorporated in future studies. Moreover, this paradigm of data collection, selection, and alignment using fine eineigrieedd preferences could be extended to more specialized, real-world industrial and downstream applications.
Iterative and On-policy Extensions. Additionally, while our current framework establishes the foundation for data selection in the standard offline DPO pipeline, extending this strategy to iterative and on-policy paradigms is natural and promising next step. For instance, in Iterative DPO, the proxy reward models could be co-updated with the policy model to filter newly collected data in each round. And for on-policy RL, the multi-aspect evaluation could serve to filter weight high- quality experiences during exploration. While there are some technical and practical gaps to bridge, believe this is critical area for active exploration.
Alternative Optimization Objectives. Another promising direction for future work to move beyond the implicit assumption of equal weighting for all sub-preferences, which is inherent in our current average reward objective (EI (z y)]). While this serves as a direct and practical starting point, exploring alternative definitions of the objective is compelling area for investigation. This could include optimizing weighted reward (E]E WETK (I, H)I to reflect varying priorities, adopting robust optimization (Efming rx 3)) to maximize performance the worst- persf-perfoniing aspect, or formulating the problem by combining Pareto optimization with effective data selection to find set of models representing different trade-offs.
Integration with Distillation. Moreover, integrating techniques from other domains, such as LLM distillation, offers valuable potential for further enhancement, We discuss at least two feasi- ble pathways: a) Distilling Preference Information, where powerful teacher I.I.M could replace our proxy reward modeling by scoring all samples across all fine-grained aspects to compute PD terms; and b) Distilling Preference Data, where teacher LLM could be used to fix or rewrite low-quality samples rather than discarding them, thereby actively curating a better dataset. While this approach faces challenges regarding high costs and reliance on external models, it is promising for introducing powerful external knowledge. Importantly, these strategies are not mutually exclusive; in practice, data selection and distillation can be effectively combined to maximize alignment performance.
To summarize, exploring these potential, diverse, and other emerging research directions helps to facilitate more effective L.M alignment using fine-grained preferences.
E APPENDIX: THE USE OF LARGE LANGUAGE MODELS
During the preparation of this manuscript, we utilized a large language model as a general- purppsse writing assistant. Its role was strictly limited to improving the clarity, grammar, and expression of
29

---

Published as a conference paper at ICLR 2026
text. The LLM was not used for any part of the research process, including idcation, experimen-
tal design, data analysis, or drawing the conclusions presented this paper.
Mataed
Wizardl
Sinstmit
Vicuna
Kosla
Lima
Avenge
SFT
0.69
0.62
0.63
0.71
0.60
0.64,005
1.00
1,00
1,00,000
AVG.
L.15
1.16
1.17
1.08
1.19
1.15an04
FULL
ALL
L06
1.04
1.10
1.13
1.11
L.Ran
DMPO
1.21
1.30
1.35
1,18
1.27
1,25ap
PD (high)
0.63
0.48
0.61
0.63
0.46
0.54an
1.01
1.21
1.18
1.12,007
SELT
20%
RAF
RAND
1,06
1.01
0.90
1.07
1.03
1.16
1.37
1.24
1.14
1.11
1.38
1.03
1:22
1.12,00
1.22an00
1.14
1.34
1:77200
(ours)
1.15an
(righ)
0.53
0.56
to
PD (mii)
1.07
1.10
1.19
1.15
1.17
1.13ap0e
PD (nl)
1.16
1.11
1.31
1.18
1.24
L.9
SFLT
RAND
1.02
1.01
1.06
1.04
1.02
1,02.00
1.18a1
PD (nai.)
L.19
1.18
1.34
1.16
1.30
1,23,000
PD (ours)
L.16
1.24
1.40
1.23
1.25
1.24
PD (high)
0.75
0.69
0.76
0.86
0.76
0. 0.75gn06
PD (mid)
1.15
1,29
1.19
1.19
1.25
1.22,n0s
1,25
1,20
1.26gn00
RAND
1.05
1.08
1:18
40%
RAF
1,17
1,23
1:34
1.19
1.25
1,28an0s
PD (mii.)
1,15
1,19
1.36
1.21
1.28
1,235000
1.24
1,23,nns
PD (high)
0.90
0,78
0.98
0.95
0.36
0.872a00
1.22
1.27
1.32
1.22.00
SELT 50%
RAND
PD (al)
1.66
1.21
0.99
1.29
1.20
1.31
1.21
1.18
1.34
1.31
1,27
1.34
1,24,006
1.35
1.21
PD (ours)
1.20
1,23
1.30
1.17
1.24
1,22,000
Table E.6: Detailed win score across five test datasets of Llama model on 10% conflict level dataset.
WizardLM
AW
0.71
0.64.a00
OVA.
1.00
1.00,000
FULL
AVG.
ALL
0.95
1.15
0.91
1.16
1.17
1.11
1.08
1.06
1.09
1.19
1.0
L.Sanoe
DMPO
1.13
1.17,006
PD (high)
0.54
0.41
(mai)
1.12gu0
PD (nl)
1.15
1,05
1.15
1.16
L.15
1.13a00
SELT
20%
RAND
RAP
0.96
1.10
0.91
1.11
1.16
1.00
1.15g0e
(rati))
1.17
PD lours)
1.15
1.15
1.24
1.07
1.27
1.172000
PD (high)
0.74
0.59 mn
PD (midd
1.11
1.13
1.26
1.15
1.22
L.
PD (nl)
1.17
1.15
1.23
1.09
1.25
L.Baa00
SELT
1.05amm
RAF
1.13
1.19
1.27
1.14
1.23
1.19
PD (mii)
1.16
1.14
1.29
1.22
1.29
PD (wurs)
1.16
1.18
1.39
1.19
1.30
1.23
PD (high)
0.67
0.66
I.75
0.78
0.55
0.66ian
PD (uiid)
1.21
1.22g100
SELT 40%
RAND
1.01
1.01
1.16
1.10
1.11
RAF
1.13
1.19
1.30
1.17
1.26
1,20a
PD (eari))
1.222006
(ouns)
1.21an0s
PD (high)
0.83an
PD (mid)
1.15
1.15
1.33
1.16
1.26
1.20
PD (nl)
1.11
1.27
1.33
1.23
1.21
1.22an
SELT 50%
RAND
1.03
1.03
1.09
1.05
1.06
1.05ame
PD (raii)
1.19
1,30
1.28
1.18
1.31
PD (ouns)
1.30
1.22
1.28
1.18
1.30
1.24mn
Table E.7: Detailed win score across five test datasets of Illama model on 20% conflict level dataset.
30

---

Published as a conference paper at ICLR 2026
Method
WizniLM
Sinstruct
Vicuna
Kouls
Lima
AW
0.71
0.64anus
DVA
1.00
1.00
1.00
1.00
1.00
1.00,000
FULL
AVG.
1.15
1.16
1.17
1.08
1.19
1:15an0
0.91 a.91
1.09
PD (hight
0.45
PD (mud]
1.05
1,00
1.32
1.14
1.07
108ga0
SELT
PD (nl)
1.09
1,44
1.36
0.98
1.11
1.14
0.95
1.12an
1.12a000
PD frat.))
1.13
1.12
1.39
1.12
1.15
1.15a000
PD (ours)
1.39
1,04
1.34
1.16
1.23
1.17
PD (high)
0.54
0,43
0,43
0.59
0.39
0.47anm
PD (miid)
1.08
1.13
1.19
1.21
1.21
1.16nn
1,00
30%
RAF
1.08
1.17
1.33
1.14
1.19
1.17an
PD (mti)
1,17
1.22
1,31
1.17
1.28
1,23,005
(suns)
PD (high)
0.58
0.51
0.56
0.69
0.51
0.56
(mnd]
1.18,000
40%
SELT
PD ial)
RAND
0.99
1.19
0.95
1,20
1.15
1.34
1.13
1.03
1,07
1.29
1,20
1.22g000
1,02a0
(rati))
1.22g00
PD (ours)
1.17
1.24
1,38
1.23
1.24
1.25gn0e
PD (high
0.70
0.57
0.77
0.73
0.72
0.68.am
PD (midd
1.19
1.17
1.23
1.25
1.22
m.21
SELT
PD (nl)
1.15
1.24
1.30
1.20
1.06are
1.21an4
1,25,mm
PD (rati.)
1.20
1.26
1.41
1.11
1.27
L.25a00
PD (ouns)
1.16
1.27
1.29
1.19
1.25
1,23apm
Table E.8: Detailed win score across five test datasets of Llama model on 30% conflict level dataset.
Method
WizidLM
Sinstruct
Vieuna
Koala
Lima
AW
SPT
0.81
0.54
0.71
0.66
0.73
0.76,007
OVA.
1.00
1,00,00
0.91
1.02
1.0%2005
ALL
1.05
1.07
0.87
1.01
1.01
1 1.85p66
DMPO
1.08
1.22
1.02
0.98
1.09
1.09
RAND
0.93
1.02
0.96
0.89
0.85
0.92.no7
SELT
RAF
0.96
0.99
0.91
0.88
0.91
0.94-004
(mai.)
0.99.nce
PD lours)
1.01
1.14
0.96
0.92
1.03
1.00..007
0.91
0.34.004
SELT 50%
KAF
1.00
1,30
0.58
0.97
0.98
1.0)
PD (mii.)
1.06
1.12
1.08
0.88
0.98
1.0%,00
1.10ga00
RAND
0.95
0.98
0.96
0.91
0.89
0.94.ac4
1.11g004
40%
PD (rat.)
1.12
1.21
1,01
L.05
1.16
1.13g006
PD tours)
1.10
1:25
1.05
L.17
1.18
1.14406
RAND
0.98
1.07
0.81
1.00
0.95
0.98,por
SELTT
RAF
1.10
1.18
1.11
1.06
1.05
1.10,c5
1.15g00
(ours)
1.191004
Table E.9: Detailed win score across five test datasets of Llama model on HelpSteer.
31

---

Published as a conference paper at ICLR 2026
Sinstract
0.78
0.74
0,77:0
1.00
1.00
1.000
1.0000000
FULL
ALL
AVG.
1.13
1.06
1.13
1.19
1.37
1.24
1.12
1.10
1.31
1.25
1. 15
1.2000088
1.33+005
RAND
1.06
1:14
1.18
1.15
1.12:004
1.17.004
20%
PD (rati)
1.15
1.13
1:17
1.18
1.19
1.168002
PD (ours)
1.23
1.21
1.34
1.29
1,30
1.2680004
RAND
1.10
1.07
1:11
1.17
1,30
04
SELT
RAF
1.20
1.15
1.28
1,20
1.37
1,24+0109
0.20+0.05
(ouns)
1.24
1:29
1.23
1.27+0.05
1.15
SELT 40%
RAF
1.25
1.16
1.29
1.31
1:57
1. 1.2950008
PD (rati)
1.16
1.20
1.24
1.27
1.25
1.2250004
PD (ours)
1.22
1.43
1.32
1.36
1,300007
RAND
1.16
1.07
1.26
1,18
1,18
1.15
SELT
1.32+0055
(ruti)
1.23
PD (ouns)
1.21
1.23
1.35
1.42
1.45
1.33
Table 10: Detailed win score across five test datasets of Qwen model on UltraFeedback.
0.78
0:74
0.77+LL5
1.00
1.00
FULL
AYG.
1.00
1004
1.06
1.12
0.99
1.07+0.05
ALL
1.03
100
1.30
0.96
1.01
1.350004
DMPO
1.16
1.09
1.06
1.05000
RAND
0.97
0.97
0.34
0.86
0.89
0.920005
20%
(mai)
1.10
1.04
1.02
0.99
1.00.004
PD (ours)
0.97
1.11
0.96
0.97
1.00
1000
RAND
0.95
0.94
1.09
0.92
0.95
0.95
SELT
RAF
0.95
1.02
1.00
1.01
0.58
0.995003
(ours)
RAND
0.38
0.93
0.96:0.4
SELT 40%
RAP
1.02
1.15
0.93
1.01
1,00
1.020007
PD (raii)
1.07
1.25
1.17
1.11
1.17
1. 1.1600066
PD (ouns)
1.13
1.22
1.11
1.09
1.19
1680.05
RAND
0.99
0.98
0.90
1.02
0.97
08080.00
SELT
1.050004
(mxii
1.27
1.06
1.29
1.30
07
PD (surs)
1.11
1.35
1.16
1.15
1.36
1.22+0.00
Table E1 Detailed win score across five test datasets of Qwen model on HelpSteer.
32


Reviews:
[{"review_id": "TpDi818Eyi", "reviewer": "Reviewer_KGaJ", "summary": "The authors proposed a data-centric approach to align LLMs through the effective use of fine-grained preferences. They studied a data selection problem and propose an effective strategy to identify a subset of data corresponding to the most negative PD values for the training.", "questions": "To address the practical challenge in DFPO, i.e., the high computational cost and the risk of instability from unavailable/unreliable reward models, the authors filter the dataset to construct a high-quality subset for the training. Besides the data selection strategy, are there other strategies to address the challenge? Is it possible to apply LLMs’ distillation techniques?\n\nThe authors extend the principle of DPO to the fine-grained preference alignment setting by introducing an additional term (equation 4), i.e., the preference divergence one. Are there other way to define the objective for the purpose? Any discussions?\n\nThe authors consider K fine-grained criteria. What if these K fine-grained criteria are dependent to each other? \n\nThe authors construct the two fine-grained preference datasets from two existing datasets. And the details are provided in the appendix. Not sure if there are other ways to construct the multi objective dataset for the purpose of the experiments?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "A balanced sampling strategy is applied to mitigate the intrinsic bias towards longer responses that are favored regardless of quality.\n\nA penalty term is introduced into the reward model to discourage length bias as well.\n\nThe paper is well written and easy to follow.", "weaknesses": "See the below questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Mnr8qI4zUJ", "reviewer": "Reviewer_vgsY", "summary": "This paper addresses LLM alignment with human preferences, emphasizing the difficulties arising when using fine-grained, aspect-specific preferences rather than singular “overall” preference labels. The core proposal is a data-centric, preference divergence (PD)-based selection strategy that identifies and utilizes the most reliable samples from aggregated, potentially conflicting, fine-grained preference datasets. \nThe authors provide theoretical grounding for their selection method (establishing loss-bound optimality), introduce approaches to estimate PD in practice (including bias mitigation), and conduct comprehensive experiments demonstrating that selective alignment using just a fraction (e.g., 30\\%) of filtered data can outperform traditional full-dataset alignment in both performance and computational efficiency.", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors clearly demonstrate motivation for the problem, where aggregating fine-grained preferences introduces conflicts, redundancy, and noise that degrade LLM alignment.\n2. The development of loss bounds and the selection optimality result underpin the proposed data selection strategy with rigorous analysis, providing compelling mathematical justification for selecting samples by most-negative PD.\n3. Extensive evaluation: The method is thoroughly evaluated against full-data and alternative selection baselines, across datasets (UltraFeedback, HelpSteer), models (Llama, Qwen), and varying conflict levels. PD-based selection (especially “ours”) consistently outperforms others—even with substantial data reduction—while being robust to proxy model choice and ablation.", "weaknesses": "1. I am not familiar with this research scope, but the current evaluation focuses on UltraFeedback and HelpSteer, and their derived conflict settings are limited. The author should conduct experiments with more advanced benchmarks for a clear demonstration of their effectiveness. \n2.  The empirical studies do not report in-depth on the sensitivity of the method to hyperparameters (e.g., $\\lambda$, quantile level $\\gamma$, length penalty $\\rho$, sampling ratio $p_r$), aside from the generic selection budget. Although performance seems stable as shown in Figures 3 and 4, the rationale for particular choices and the potential for overfitting (especially when “tuning” for the best selection cut-off) are not thoroughly analyzed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZWcjDUbkq3", "reviewer": "Reviewer_qeXk", "summary": "The paper reframes preference learning under mixed and potentially conflicting preferences as the selection for a subset of training data whose implicit preference is closest to the ground-truth preference. A data-selection algorithm (DFPO) is accordingly proposed and evaluated extensively.", "questions": "1. The formalism treats (human) preference as an abstract concept, while the same conflict could arise when mixing code, math, etc.  How should the “ground-truth preference” be defined in such multi-task settings where no single human ranking exists?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "1. Detailed problem formulation and the novel transformation of the problem into data selection methods instead of algorithm development is interesting.\n\n2. Empirical coverage is thorough.", "weaknesses": "1. With ever-larger models and compute, scaling-laws may simply “wash out” moderate preference noise; the urgency of the problem is not demonstrated.\n\n2. The method operates on a fixed dataset and is demonstrated only with the now “classical” DPO pipeline.  Readers working with on-policy RL extensions are unlikely to see an immediate hook.  Extending DFPO to iterative regimes like iterative DPO would greatly widen its appeal.\n\n3. If a dataset contains several conflicting preferences, DFPO appears to return the largest conflict-free subset.  When preferences are equally strong and mutually incompatible, is the algorithm simply selecting the majority preference?  A crisp explanation that shows what is really done by DFPO is missing.\n\n4. Many experiments are presented, but each is analysed briefly. Spare more pages to careful analysis will be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YyC8B83cEw", "reviewer": "Reviewer_xH9J", "summary": "This paper addresses a critical challenge in aligning Large Language Models (LLMs): effectively utilizing datasets composed of multiple, fine-grained human preferences (e.g., helpfulness, honesty) which often contain inherent conflicts and noise. The authors identify that standard alignment methods like DPO struggle with such aggregated data. Their core contribution is a data-centric approach that reframes the problem as data selection. They first formulate a Direct Fine-Grained Preference Optimization (DFPO) objective, which introduces a Preference Divergence (PD) term that quantifies the conflict between a specific fine-grained preference and the consensus of all others. Instead of directly optimizing this complex objective, they theoretically and empirically demonstrate that selecting a subset of data with the most negative PD values (indicating high consensus across aspects) for standard DPO training leads to superior performance. The proposed method involves training small, de-biased proxy reward models for each sub-preference to estimate the PD terms. Extensive experiments on datasets derived from UltraFeedback and HelpSteer show that their method consistently outperforms full-data alignment and other baselines, achieving better results using only 30-50% of the data, while also being more robust to increasing levels of preference conflict.", "questions": "1. How would the method perform if the fine-grained aspects were not orthogonal or even negatively correlated? For instance, if \"conciseness\" and \"completeness\" were both aspects, the definition of \"conflict\" might need refinement.\n\n2. The length bias mitigation is a crucial component. Was any analysis done to determine the optimal value for the length penalty coefficient ρ? Is it robust across different datasets, or does it require tuning?\n\n3. The paper focuses on selecting data for a single round of DPO training. Could this PD-based selection be integrated into an iterative training loop, where the aligned model is used to re-estimate rewards and re-select data for further training?\n\n4. The \"PD (rati.)\" baseline, which uses ground-truth ratings, performs comparably well in some cases. Does this suggest that with perfectly reliable, continuous ratings (instead of binary preferences), the need for proxy models could be eliminated? What are the relative trade-offs between the annotation cost of ratings vs. the computational cost of training proxy models?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. he paper clearly identifies a significant and under-explored problem—handling noise and conflicts in aggregated fine-grained preference data. The motivation is strong, backed by a statistical analysis of a real-world dataset (UltraFeedback) showing a high rate of preference conflicts.\n\n2. The progression from the DFPO formulation to the PD term, and then to the data selection strategy, is elegant and well-justified. The theoretical analysis (Theorems 3.4 & 3.5) provides a solid foundation for the simple yet effective heuristic of selecting samples with the most negative PD.\n\n3. The experimental section is thorough and convincing. The authors demonstrate:", "weaknesses": "1. The entire selection process hinges on the quality of the proxy reward models. While the paper shows low sensitivity to the model's size and training data amount, it does not deeply explore scenarios where these proxy models might fail catastrophically (e.g., on out-of-distribution data or aspects that are inherently difficult to model). The method inherits all the challenges of reward modeling.\n\n2.  The method implicitly assumes all sub-preferences are equally important by summing their pseudo-reward gaps uniformly in the PD term. In reality, some aspects (e.g., \"truthfulness\") might be more critical than others (e.g., \"verbosity\"). The work does not explore weighted PD terms or user-specified preference hierarchies.\n\n3. The experiments are conducted on models up to 8B parameters. It is unclear how the method would scale to very large models (e.g., 70B+), where the computational savings would be even more dramatic but the dynamics of fine-tuning can differ. Furthermore, the evaluation, while comprehensive, is primarily based on general-purpose chat benchmarks; testing on more specific safety-critical or instruction-following benchmarks would strengthen the claims about robust alignment.\n\n4. The derivation of the DFPO objective in the main text (and Appendix A.1.1) is quite dense and could be challenging for a reader to follow. A more intuitive, step-by-step explanation of how the PD term emerges from the multi-aspect RLHF objective would improve accessibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)