Adaptive Batch-Wise Sample Scheduling for Direct Preference Optimization
Authors: Zixuan Huang
Yikun Ban
Lean Fu
Xiaojie Li
Zhongxiang Dai
Jianxin Li
deqing wang
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement Learning
Direct Preference Optimization
Primary area: general_machine_learning

Abstract: Direct Preference Optimization (DPO) has emerged as an effective approach for aligning large language models (LLMs) with human preferences. However, its performance is highly dependent on the quality of the underlying human preference data. To address this bottleneck, prior work has explored various data selection strategies,  but these methods often overlook the impact of the evolving states of the language model during the optimization process.
In this paper, we introduce a novel problem: Sample Scheduling for DPO, which aims to dynamically and adaptively schedule training samples based on the model's evolving batch-wise states throughout preference optimization. To solve this problem, we propose SamS, an efficient and effective algorithm that adaptively selects samples in each training batch based on the LLM's learning feedback to maximize the potential generalization performance.
Notably, without modifying the core DPO algorithm, simply integrating SamS significantly improves performance across tasks, with minimal additional computational overhead. 
This work points to a promising new direction for improving LLM alignment through batch-wise sample selection, with potential generalization to RLHF and broader supervised learning paradigms.

Adaptive Batch-Wise Sample Scheduling for Direct Preference Optimization
Zixuan Huang
Yikun Ban Lean Fu²
Xiaojie Li'
Zhongxiang Dai Jianxin Li Deqing Wang
1 Beihann University 'Bytedance Inc The Chinese University of Hong Kong, Shenzhen
{huang_ zx, yikunb, 1i xiaojie, dqwang) edu. lijx0act. buaa. edu. en fulean@bytedance com deizhongxianggechk. edu. cn
Abstract
Direct Preference Optimization (DPO) has emerged as an effective approach for aligning large language models (LLMs) with human preferences. However, its per- formance is highly dependent on the quality of the underlying human preference data. To address this bottleneck, prior work has explored various data selection strategies, but these methods often overlook the impact of the evolving states of the language model during the optimization process. In this paper, we introduce a novel problem: Sample Scheduling for DPO, which aims to dynamically and adap- tively schedule training samples based on the model's 's evolving batch-wise states throughout preference optimization. To solve this problem, we propose SamS, an efficient and effective algorithm that adaptively selects samples each training batch based on the LLM's learning feedback to maximize the potential general- ization performance. Notably, without modifying the core DPO algorithm, simply integrating SamS significantly improves performance across tasks, with minimal additional computational overhead. This work points to a promising new direction for improving LLM alignment through batch- wise sample selection, with poten- tial generalization to RLHF and broader supervised learning paradigms. The code is available https:://// gg thub. com com/hzx122 http:/xxx122/222222 /SmmS
Introdcution
Direct Preference Optimization (DPO) 6
proposed as simpler and more stable alternative to Reinforcement Learning from Human Feedback (RLHF) [2TX 102, 64 BS, [Z]] As an off-policy preference optimization method, DPO does not require first training explicit reward model. In- stead, given a preference dataset where each sample includes a prompt and a pair of generations with the first more consistent with human preferences, directly optimizes straightforward binary objective, which increases the likelihood of chosen response and decreases the
cross-entie cross-entropy-tye likelihood of rejected response. The promise of this approach is that it implicitly optimizes the same objective as RLHF without adding complexity.
Although DPO has demonstrated exceptional performance across wide range of tasks, its heavy re- liance on high-quality human preference data poses significant bottleneck for practical deployment due to the associated annotation costs. To mitigate this challenge, substantial research efforts have been devoted to enhancing the data quality and utilization in preference optimization. These efforts selecting informative samples for human feedback collection; (2) Response Pair Selection [5 57 actively choosing response
generally fall into three categories: (1) Active Querying 24 61l 45
pairs to annotate conditioned given query; (3) Data Pre-selection [73]
D5.
Ga]]: identifying and
Deqing Wang and Yikun Ban corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

filtering high-quality samples prior to DPO training. However, approaches in categories (1) and (2) typically only focus on online feedback collection and ignore data quality, while methods in category (3) overlook the evolving internal states of the language model throughout the DPO process.
In contrast these existing studies, this paper introduces novel problem: Sample Scheduling for DPO. Specifically, given fixed preference dataset, the goal is to dynamically and adaptively sched- ule training samples based on the evolving internal states of the language model during preference optimization. This formulation is motivated by two key challenges: First, as shown in Figure la samples in the training dataset may exhibit varying levels of learning difficulty for different model states. As the models internal state evolves over time, the relative difficulty of each sample may also shift. Without an adaptive scheduling mechanism, the model may overemphasize samples mis- aligned with its current learning capacity or overfit to some eтror patterns. thereby impairing its alignment performance [34L I Second, the dataset may contain noisy samples [[Z]]] As shown in Figure II incorrect or inconsistent preference labels can destabilize the DPO training process [EG], and low- low-quality but preferred responses may erode the original conversational ability of the model. We also empirically verify the presence of such noise in Appendix EB
Noise
Learning Difficulty
0.70
DPO with
0.80
cpo
0.00
-0.00
Sing
Step 20006
0.50
DPQ Uh Noke
DP0
0,41
5060
Slap
(a)
(b)
The Study of challenges in SamS. (a) Varying learning difficulties for different model states. For the same 16 samples, we track their DPO loss across different states of the language model, from training step 0 to step 30,000. We use the relative DPO loss of each sample as the difficulty measure [34]] (b) Noisy data degrades DPO performance. During preference optimiza- tion using Pythia-2.88 8B | on the Anthropic-HE HH dataset [M]] we artificially injected 20% noise into the preference labels. As result, the performance of DPO dropped significantly, highlighting its sensitivity to data quality.
Figure I:
To address this problem, we propose a scheduling algorithm SamS, Sample Scheduling for Direct Preference Optimization. In particular, we formulate Sample Scheduling for DPO as a contextual bandit problem, where we define the reward for sample scheduling by leveraging the loss signal during DPO training, and define the arm context based on the internal state representation of LLMs. In this setting, SamS employs a scheduler model adaptively select samples from each training batch according to the model's 's evolving states, in order to maximize the potential resulting general- ization performance. It incorporates two key innovations. First, adopts a lagged training strategy, where the scheduler is updated in the subsequent training round, allowing the reward to be collected without incurring additional computational overhead. Second, it introduces an auxiliary exploration network to explicitly address the exploration-exploitaai dilemma that is inherent in the iterative sample scheduling problem.
We conduct extensive experiments across diverse benchmarks, including AlpacaEval 2 [ and MT-Bench [[00]] to evaluate the effectiveness of SamS. Notably, when integrated with the original DPO loss, SamS consistently outperforms several advanced offline preference optimization methods on mainstream evaluation benchmarks. Particularly, our method improves the AlpacaEval win rate (WR) by 3.0% 12.4% and the length-controlled win rate (LC) by 5.5% 8.4% compared to the baselines. Furthermore, we conduct a thorough evaluation of SamS under noisy preference data conditions and show that its integration significantly enhances robustness against label noise. Importantly, thanks to the carefully designed scheduling reward and the lightweight architecture of SamS, the added training overhead is minimal, and GPU memory consumption is even reduced.

---

In summary, our contributions can be summarized as follows: (1) Novel Problem: We introduce new problem, Sample Scheduling for DPO, which highlights a promising direction for improving LLM alignment performance using fixed preference datasets. (2) Proposed Algorithm: We propose SamS, a scheduling algorithm that adaptively selects training samples from each batch according to the model's evolving internal states. (3) Empirical Effectiveness: SamS can be seamlessly inte- grated into existing DPO pipelines without modifications to the core algorithm, yielding substantial performance improvements with only marginal additional computational overhead. Batch-wiss sam- ple selection opens promising path for efficient LLM alignment, and the idea naturally extends to RLHF and other supervised learning paradigms.
2 Preliminary
DPO 6- is an offline preference optimization algorithm designed to simplify and stabilize training by reparameterizing the reward function typically used in RLHF, Specifically, DPO reparameterizes the reward model using a closed-form expression:
Te(y 3 log (xx) <
r(z, y) = 3 log
where Th represents the policy model, Neef is the supervised fine-tuned reference policy, and (xx) denotes the partition function.
Given a data sample completions respectively for the prompt æ, the DPO framework incorporates this reward formulation into the Bradley- TearyyyTerryy ranking objective [[5]]. Specifically, it defines the probability p(y" > y'(z)= a (r(æ,, ) - r(æ, )²)), where a denotes the logistic function. Consequently, the objective of DPO formally defined as:
(z, y",, y²)),, where y" and y' represent the preferred and dispreferred
(apo 0) = - (r."" WDDDD log (s(10g To T
To (z) Teet F (v/lz)) )
log
(2)
Teat (z
In practice, batch-level preference optimization is commonly employed. Given a batch consisting of samples, denoted X, +o where each sample ar = (Ia,5s vio v(.)). the average-based DPO loss is formally defined as:
CDPO(X,3
Σ ax,
LDrO (ut,5
X
During each training round [T]. the policy Te typically learns from the entire current batch X,, which may contain irrelevant, challenging, noisy samples. To address this, objective is to train a scheduler capable of effectively exploring the sample space, thereby identifying and selecting reliable, high quality samples for the policy's 's offline preference optimization.
3 The Sample Scheduling Problem
We formulate the Sample Scheduling problem for offline preference optimization using the contex- tual bandit framework proposed in [8. ιл [i] Let me denote language model parameterized by 0 that we aim to align with human preferences, and let ʃ denote a scheduler designed to perform interactive sample scheduling during batch-level preference optimization.
Problem Formulation. Assume the learning process spans rounds. At each round t E [T], we draw batch containing samples, denoted by X, = {ar,1, 0,,2,...5 at,x } ~ D, where each sample dri (xr,a, vi vi,,) for E [n]] is considered an arm, resulting in total arms. For each arm a, we define a contextual representation ε,, h(x v² v(.)). where h(-) is an encoding function mapping each sample to context representation vector.
Given a subset X; C X, with size K, |X, k. selected by the scheduler f. we train the policy Tg.-1 on this subset, updating the policy parameters to 0, as follows:
0, 0, 1/ V (x:
(4)
3

---

To measure the improvement from 8t-1 to Og using the selected subset Xt. we introduce a reward function r r(X,, 0c-1 + Oz), which is initially unknown. In each round E [T', the scheduler selects a subset X, from batch X, and provides it to policy me.-.- Subsequently, the scheduler observes the reward r(X,, f - ft), which informs updates to its parameters for future scheduling optimization. The objective for the scheduler f over 'T' rounds thus to select a sequence of subsets (X1,X2, X2............ X that maximizes the cumulative reward:
T max Σ -
1 -z). 0z).
(5)
Reward Definition. In supervised preference learning, accurately measuring the performance im- provement of policy To from 01-1 to 0, using an oracle is typically impractical. To address this limitation, we propose a reward definition r leveraging insightful information from the learning tra- jectory of 0. This reward acts as the supervisory signal for training the scheduler f and comprises two distinct components: batch-level reward and a sample-level reward.
First, we introduce the batch-level reward, which measures the reduction in the average DPO loss before and after training with selected batch. In practice, approximate the expected DPO loss across the entire data distribution. Formally, at round 6, given the policy parameters 0--1. we train on a subset X,, resulting in updated parameters 0,. The batch- level reward for selecting X; is defined
the batch-average DPO loss to
B
(umo(a,::0e.))
1 B (X,, 01-1, 0x) max (Eiz (Cnro(ar Ei= cCmoomeet..stt))
(6)
Term A evaluates the performance of 0,- 1 batch X,, noting that O,- has not previously encoun- tered X₁. Similarly, term B evaluates the performance of Or on the new batch X1+1, after Og has been trained on X, To enhance the sensitivity of the reward metric, we exponentiate the DPO loss eComol)) and apply normalization through the denominator. Consequently, r signifies the approxi- mate performance improvement of the policy after the scheduling decision at round t.
Next, we introduce a sample- level reward for fine- grained evaluation, complementing the batch-level reward, which only reflects aggregate improvement over X, We assign higher rewards to samples with larger preference margins and greater model uncertainty.
Formally, for a data point at,i (Ii,5; &f s. we define the sample-level reward:
no.-, (vi,s 1 12ns)) )) + (1 g(log ne, (vi,, | [x,,)))
S (ar.d, a Blog
3 log
(7)
Tead (V/, a2.a)
merl e x1,2)
model uncertinnty
preference margin
The first term rewards samples with larger preference margins under 0--1, thereby avoiding con- vergence on ambiguous or noisy examples [60]] The second term promotes selection of high- uncertainty samples, addressing the tendency of policies to produce cuitt--distriuution outputs dur- ing training [53, 9T]. Here, 9(-) is min-mas normalization mapping values to [0, 1]. We provide more detailed discussion the design motivations in the Appendix E
Finally, we integrate both the batch-level and sample-level rewards to compute the final reward for each individual sample at,; as follows:
r(ar,s, f-- - 02) = no [r (x,, X141,00) + (1 - x)o [r. (ar. (c-1)
(8)
where y E [o, 1] is a hyperparameter that controls the trade-off between the batch-level and sample- level reward signals, and (-)) denotes the sigmoid function.
To mitigate the combinatorial complexity associated with subset selection, we define the reward of a selected subset X, as the sum of the individual sample rewards:
r( r(X,, 0-1 +0,)
Σ xx,,
r(ar,;, 0, - 0,).
(9)

---

Workflow
Model Architetcue
Batch
Policy
Transfomeer
Encoder
Scheduler
1:870 Torward
4,0YO Backword
Block
Policy
S4 Aanothi Ruvt]
Transtormet
CONCAT
Explotation
CONCAT
Exploration Network
Arm Contest
Selected Data F
Network
f
Transformer
BV Schecluler
Block
Linear
Add
Limwar
Training Batch
Ansvee Reward
Estimetedd Rlwort
Backoward Update Policy
Estimatedd
Rewards
for
rpp.
Selected
Top-K K
Sample Sclaadiing
Round
Figure 2: (Left side) Overview of standard DPO framework integrated with SamS. (Right side) The architecture of the Scheduler. The Scheduler initially treats the policy's hidden state sequence the arm context for each sample. The Encoder aggregates the state information of each sample to encode the arm context. Subsequently, the arm contexts to estimate reward values for each sample, which is used to select Top-K subset for policy learning.
Network utilizes the encoded
In addition to providing valuable insights, the reward can be computed straightforwardly during the DPO process.
Arm Context Design. The arm context serves as the input to the scheduling module, with the goal of leveraging the representational capacity of the policy model wy- For each sample, we extract the intermediate hidden representations from all transformer block layers of тTs and define the arm context as F h( V/, vf..). To obtain fixed-dimensional vector, we apply combination of concatenation and pooling operations over the token-level hidden states across layers. This design allows us take into account the evolution of the state of the LLM into the arm representations.
Proposed Algorithm: SamS
We present the overall framework of the proposed method, SamS (Sample Scheduling), followed by a detailed description of its model structure and workflow. Sample Scheduling can be naturally considered as sequential decision-making problem under uncertainty, where the internal states of the policy evolve across rounds, resulting in uncertainties for the iterative sample selection. Consequently, the exploration-explootation dilemma is inherently embedded in this problem.
Model Structure. As shown in Figure Q the scheduler f consists of an encoder layer followed by two specialized networks for exploitation and exploration. The Encoder Layer takes the hidden state representations of each sample as input and produces an encoded representation used by the subsequent neural networks. For notational simplicity, we continue to denote the encoded arm context as Zt.i.
Denote the exploitation network by ploitation network f8 learns predict the reward of each sample arm by mapping the arm context Le, to its observed reward r(ar,i, 0,-1 + 0.). The exploration network f estimates the uncertainty of the predictions made by /ك_ and augments the original reward estimate with a potential explo- ration bonus. This design enables principled trade-off between exploitation and exploration during iterative sample selection, referring to the design in [8. [2]. This process aligns with the principles of classic Exploration- -xxioaatioonExlliiaaaiiinn algorithms, such Upper Confidence Bound (UCB) [R) I. [ and Thompson Sampling (TS) [84, [SS].
and the exploration network by (:; 8S'). The ex-
Given the input Ft,; in round f, the exploitation network S is implemented fully connected feedforward neural network with residual connections, denoted by (z, After receiving the observed reward r(ar,, O + 0,) in round + 1, the parameters 0 are updated via stochastic
5

---

Algorithm Proposed Algorithm: SamS
Require: T. n., K, 0 (LLM Parameters), 05, 0 (Scheduler Parameters)
1: Initialize og. 0 08.05 of
2: for t = 3, ..... T do
3: Draw batch data X2 ~
< DPO Forward with X,
4: Compute DPO Loss (xp;(Xx;0ee1) Standard Forward Pass
V Scheduler Training
5:
Compute ,B (X1-1, X 0 0 based (q.(3) Observe Batch-level Reward
6:
for 1,-1, E EX+1 do
7:
Compute r (0t-1,5; 0; according to (z) # Observe Sample-level Reward
8:
Compute r(it-1,i; 2 0t-1) according (q.(8)
# Obaerve Final Reward
9:
end for
10:
Compute (X,-1, 0,,1) 11-1) According to Eq.
11:
0 05 1/1 V 0z_ L (* 0
# Update Exploitation Network of Scheduler
Compute L (X1-1, 05- According to (I)
13:
05 0 7/2 V 0.: L (x-11.051)
# Update Exploration Network of Scheduler
V Scheduler Scheduling
14:
for E [M] do
15:
f(a,,;, 0--1 + 01) ( (T),;;0))) 08) ) + A (hg,;05) Sample Based on Apploitation-Esplorattte end for
# Estimated Reward for Each
Trade-off
16:
17:
X, = ie 02-1 - 0t) # Choose X,
V DPO Backward with Xt
18:
0, = 0:-1 v/ v LooO 0(-1) # Udpate LLM with X,
19:
end for
Return: 0T
gradient descent using the following loss function:
L (xx,
Σ (F, rr(ass 00-1 0c)] 2|Xe| ax
(10)
Next, in each round 1 E [T]. we construct the input to the exploration network f by concate- nating the intermediate hidden states of ( 00_1) along the last dimension, denoted by he This design enables the exploration module to take into account the internal states of the exploita- tion network when making exploration decisions. The exploration network / is also fully con- nected feedforward neural network with residual connections. After receiving the observed reward r(ar,i, 0: - 0.) in round + the label for training fs' the difference between observed reward and ($) for uncertainty estimation. The exploration network parameters 0, are then updated via stochastic gradient descent using the loss:
L (x,,000))
Σ
[1St ( (1E (r(ar,;,0t_11 -
(8 (11)
2|X Or,.EX,
Finally, the overall reward estimate for each sample is given by: f(Tr,;; 0,,0000000 .0005) S 05) Sf (hī,; 05""), where À is tunable hyperparameter controlling the cxploration strength. Next, describe the training strategy for integrating the scheduler within the DPO framework.
Workflow. Algorithm illustrates the workflow of our proposed SamS algorithm. Each training round consists of four main steps, detailed as follows:

---

(1) DPO Forward Pass. In each training round t E [T], we first perform a forward pass to compute the DPO loss following the standard DPO procedure (Line 4). We store the loss result of each sample (ar,; 0 for subsequent scheduler training.
(2) Scheduler Training. The objective of this step is train the scheduler / hased the previ- ously selected subset X from round t 1, utilizzing the pair (X,-1, - - (t-1)}- This approach leverages the batch-level reward r (Xx-1,8,-22 X,, 02-1), which requires the loss ( 0r-1) computed in the current round I, thus avoiding extra computational costs. Lines 5-9 depict the reward calculation for the previously selected subset - while Lines 10-13 update the scheduler with the new information. In practice, prevent the scheduler from overfitting to the current batch, we maintain a pool containing historical training data and apply a hybrid iterative- offline training procedure. We display the implementation details in Appendix [E]]
(3) Scheduler Scheduling. With the updated scheduler parameters 0f, 05", estimate rewards for each candidate sample denoted by f(ar,, f + 8,) as shown in Lines 14-16. Subsequently, we apply a straightforward greedy strategy to select K' samples, forming the subset X,
(4) DPO Backward Pass. Given the selected subset Xe, we compute the corresponding batch loss CDoO 01-1). Since Xe is subset of Xt, LDPO 0c-1) can be efficiently derived from the previously computed CDPO (Xp 0.-1). Finally, the policy model parameters f,-1 are updated to 0 through gradient descent (Line 18).
5 Experiments
In this section, we present the primary experimental results along with their analysis. For SamS, both the exploitation and exploration modules are implemented as 16-layer residual MLPs. We set the batch size |X,] to 64 and the selection size [X,] to 32 across all training rounds. Additional implementation details of SamS are provided in Appendix D due to space constraints.
5.1 Performance of SamS Embedded in DPO
In this subsection, we evaluate the performance of SamS when integrated into DPO, using widely adopted benchmarks for LLM preference optimization. We compare it against state-of-the-an offline preference optimization methods. Detailed experimental settings can be found in Appendix D.I.
(1) DPO+SamS consistently achieves superior performance. As shown in Table MI the adaptive sample scheduling mechanism of SamS enables DPO to attain the highest scores across all evalua- tion metrics. Specifically, DPO+SamS outperforms the best rest-performing baseline by margins ranging from 0.4% 6.3% on the AlpacaEval LC win rate, from 0.2% 7.4% on the AlpacaEval 2 win rate, and by 0.1 to 0.2 on the MT-Bench score across various settings. These results underscore the broad applicability of SamS in preference optimization and effectiveness in aligning large language models with human preferences.
(2) SamS reliably prioritizes samples that are well-suited to the current model state. To high- light the sample quality, we compare DPO+SamS against a baseline variant denoted as DPO (50%), in which 50% of the training samples in each batch are randomly selected under the same conditions. Across all model oonfigurations, DPO+SamS consistently improves performance over DPO (50%), with gains of 5.5% 8.4% on the AlpacaEval LC win rate, 3.0% 12.4% on the AlpacaEval win rate, and 0.2 0.4 on the MT-Bench score. These substantial improvements demonstrate the effectiveness of SamS in dynamically identifying and utilizing high-quality training samples.
5.2 Generalization Ability
To assess the generalization ability of SamS, apply SamS to various offline preference optimiza- tion algorithms, conducting multi-epoch experiments under diverse preference datasets.
We utilize the pretrained Pythia-2.8B [[4] as the policy model, using Anthropic-HH [ and SHP [BII] as the preference dataset. Initially, we perform SFT using the prompts and chosen re- sponses from the dataset. Subsequently, apply SamS to DPO and KTO, conducting multi-epoch
7

---

Table AlpacaEval [ and [К results under the two model settings. LC and WR denote length-come ength-controlled and raw win rate, respectively. Here, bold denotes the best performance, underline indicates the second-best performance, and "_" represents that no measurement was taken.
Mistrall-Intrrun (7B)
Llama3-Instrun (8B)
Method
AlpacaEval
MT-Bench
AlpacaEval
MT-Bench
LC (%)
WR (%)
GPT- Turbo
LC (%)
WR (%)
GPT-4 Turbo
SFT
17.1
14.7
6.2
26.0
25.3
6.9
RRHF
25.3
24.8
6.5
31.3
28.4
6.7
SLiC-HF HF [X
24.1
24.6
6.5
26.9
27.5
6.8
IPO A
20.3
20.3
6.4
35.6
35.6
7.0
CPO 33
23.8
28.8
6.3
28.9
32.2
7,0
KTO [AL
6.4
33.1
ORPO
24.5
24.9
6.4
28.5
27.4
6.8
R-DPO 60
27.3
24.5
6.2
41.1
37,8
7.0
DPO [8]
26.8
24.9
6.3
40.3
37.9
7.0
DPO (50%)
25.2
23.8
6.3
37.5
36.2
6.9
DPO+SamS
33.6
36.2
6.7
42.2
40.5
7.1
Jlama3-Instrur v0.2 (8B)
Gemma2-Instree v0.2 (9B)
Method
AlpacaEval
MT-Bench
AlpacaEval 2
MT-Bench
I.C LC (%)
WR (%)
GPT-4 Turbo
LC
WR (%)
GPT-4 Turbo
SFT
26.0
25.3
6.9
48.14
36.5
RRHF
37,9
31.6
7.1
SLiC-HF [2
33,9
32.5
6.9
IPO [
46.8
42,4
7.2
62.6
58.4
CPO 3C
34.1
36.4
7.2
56.4
53.4
E9A
34.1
32.1
7.2
61.7
55.5
ORPO
38.1
33.8
7.2
56.2
46.7
R-DPO 68
48.0
45.8
7.0
68.3
66.9
DPO [6]]
48.2
47.5
7.0
70.4
66.9
DPO (50%)
46.0
45.2
6.9
66.1
63.5
DPO+SamS
51.5
48.2
7.3
70.8
67.1
training until test accuracy converges. We then compare the performance metrics of our approach with those of the original methods. For the DPO and KTO loss, we set f = 0.1.
Table 2:
Performance improvements (in test accuracy) achieved by integrating SamS with different preference optimization methods.
Dataset
Method
Test-Ace(%)
Dataset
Method
Test-Ace(%)
DPO
64.3
DPO
67.6
DPO+SamS
67.1
DPO+SamS
70.0
Improvement
+2.8
Improvement
+2.4
SHP
KTO
60.2
KTO
65.2
KTO+SamS
63.3
KTO+SamS
67.5
Improvement +3.1
Improvement
(1) Integrating SamS with different offline preference optimization methods consistently en- hances performance. As shown Table DI (with detailed results in Table B), applying SamS to average increase of 2.65% in test accuracy
two baseline methods yields notable improvements: (Test-Acc), 19.9% improvement in the reward value of the preferred response (Chosen Reward), and 5.8% gain in the log- bog-probability of the preferred response (Chosen Logps). Remarkably, these performance gains are achieved with only 50% of the original training data, highlighting the sample efficiency of SamS. These results demonstrate that SamS significantly improves both the effective- ness and efficiency of training by prioritizing high-quality samples.
(2) SamS effectively mitigates out-of-distribution (OOD) challenges for difficult samples. The observed improvements in Chosen Reward and Chosen Logps suggest that the policy's implicit reward model is better optimized, enabling it to assign higher rewards to preferred but hard responses.

---

This outcome aligns with the motivation presented in Section EL confirming that SamS successfully addresses OOD issues by adaptively focusing on the most informative training samples.
5.3 SamS Enhances the Robustness of DPO to Label Noise
To further validate SamS's reliability in selecting high- quality samples from another perspective, construct a scemario with a contaminated dataset, focusing on its capability prevent noisy samples from disrupting pol- icy training. Concretely, we randomly flip the preference labels for 20% of the response pairs in the Anthropic- HH dataset (SHP dataset) and run DPO and DPO+SamS on this modified dataset, adopting the same experimental setup as described in Section 5.2
HH
S4-P
1.0
can
645
05
05
20
Figure 3: Robustness Testing of SamS:
DPO vs. DPO+SamS (Test Accuracy).
As illustrated in Figure & under the influence of noisy samples, DPO's test accuracy in the HH (resp. SHP) dataset converges to approximately 58% (resp. 64%). 6% (resp. 4%) decline compared to 64% (resp. 68%) in the noise-free setting. In contrast, DPO+SamS converges to around 64% (68%), with only 3% (2%) drop from its original 67% (70%). DPO+SamS consistently and stably outperforms DPO by approximately 6% (4%) in test accuracy, demonstrating superior performance in noisy conditions. Moreover, when compared the original Anthropic-HH (SHP) dataset, DPO+SamS shows only marginal performance degra- dation, indicating that SamS can effectively maintain the stability of policy training in noisy scenarios. This especially crucial in offline preference optimization, where high-quality, manu- ally annotated preference datasets are limited.
5.4
Computational Cost Analysis
SamS is lightweight and compute-efficient. Figure g il-
DPO
DPO+Sam6
lustrates the peak single-GPU memory usage and overall runtime of DPO and DPO+SamS under setting of LLaMA environment. Compared to the vanilla DPO implemen- tation, DPO+SamS reduces GPU memory usage by ap- proximately 18% with similar runtime, owing to SamS's reduction in computational overhead (fewer samples) dur- ing backward propagation LLM updates. As the re- ward computation SamS does not require additional for- ward passes through the LLM, and the scheduler model is relatively lightweight, the additional computational cost (running time) introduced by SamS is marginal.
100
Percentage (9%)
80-
75G
2.3h 4h
61G
Peak GPÚ Memory
Runtime
Figure 4: Computational cost DPO
vs. DPO+SamS: similar runtime and 18% less GPU memory usage.
5.5 Comparison with Data Pre-Selection
In this section, we compare SamS with Selective DPO [G2]] representative method of the Data seleeeecciin [ 25, 62),, which is most relevant to our problem setting. Selective DPO first trains reference models using subset of the preference dataset, then employs forward passes of these reference models to compute the difficulty of each sample in the preference dataset. Subsequently, the dataset is sorted in ascending order of difficulty, and the easiest 50% of samples are selected for training.
We conduct experiments using the first LLaMA setting, evaluating both Selective DPO and Selective DPO+SamS. In the latter, we further apply SamS to select 75% of samples in each batch for policy learning based the ordered subset chosen by Selective DPO.
(1) SamS achieves performance comparable Selective DPO while introducing minimal ad- ditional computational cost. As shown in Table &, DPO+SamS yields results similar to those of Selective DPO. However, unlike Selective DPO, which requires a complete additional training phase, SamS can be seamlessly integrated into DPO, incurring only marginal computational over- head. Specifically, Selective DPO entails a total computation time of 6.0 hours, including 5.1 hours for training reference models and 1.2 hours for DPO training. In contrast, our method requires ap-

---

proximately 2.4 hours in total, closely aligning with the time cost of standard DPO while reducing GPU usage by 18%.
(2) Selective DPO+SamS achieves significant performance improvements. As shown in Table N while both DPO+SamS and Selective DPO effectively enhance performance the SFT model, Selective DPO+SamS significantly outperforms them. Specifically, Selective DPO+SamS achieves 46.5% AlpacaEval LC win rate, 44.0% AlpacaEval 2 win rate, and MT-Bench score of 7.2, representing improvements of 6.2%, 6.1 1,, and 0.2 respectively over DPO. These significant performance improvements strongly demonstrate the enormous potential of our adaptive sample scheduling strategy when integrated with Data Pre-selection methods.
Table 3:
The comparative results of SamS applied on DPO and Selective DPO under the first LLaMA setting. Here, bold denotes the best performance, underline indicates the second- -est performance, and "_" represents that measurement taken.
AlpacaEval?
MTT-eench
Method
Runtime
LC (%)
WR (%)
GPT-4 Turbo
SFT
26.0
25.3
6.9
DPO [6
40.3
7.0
2.3 h
DPO+SamS
42.2
40.5
7.1
2.4
Selective DPO [e
41.7
40.9
7.0
6.0+1.2
Selective DPO+Sams
46.5
44.0
7.2
6,0+1.3
For the ablation study, refer to Appendix D24L
6 Related Work
Direct Preference Optimization Variants. A variety of offline preference optimization algo- rithms have been proposed besides DPO. Ranking objectives allow for comparisons among more than two instances [27 55, (75, 93]. Another line of work explores simpler preference optimization objectives that do not rely tion between the SFT and the aligned model to further enhance model performance. [[ES] proposes
reference model [. РО). [Р9] focuses post-training extrapola-
method to jointly optimize instructions and responses, finding effectively improves DPO. In
this work, we compare DPO+SamS to series of offline algorithms, including RRHF [Л5], SLiC-
HF [97], DPO [68], IPO [6], CPO [89], KTO [B ORPO [} and R-DPO [66]], and find that DPO+SamS can outperform them while achieving remarkably high sample efficiency.
Iterative Direct Preference Optimization The absence of explicit reward model in DPO limits its capability to sample preference pairs from the optimal policy. [28, μλ mi g8, |2] extend the preference data augmentation approach [97, DG. AIl
iterative training framework, where the reference model is continuously updated with the latest policy model or new preference pairs are generated at each iteration. this study, we concentrate solely on offline settings.
7 Conclusion
We introduce novel problem setting, Sample Scheduling for DPO, which highlights promising direction for enhancing LLM alignment performance using fixed preference datasets. To address this problem, we propose SamS, an efficient adaptive algorithm that dynamically selects training samples from each batch based on the model's evolving state. Without modifying the underlying DPO algorithm, simply integrating SamS into the framework achieves significant performance im- provements while incurring only marginal additional computational costs.
10

---

Acknowledgments
This work was supported by the National Natural Science Foundation of China (No. 62276015 and No. 62506024).
References
[1] Meta. Llama 3 model card. 2024.
[2] Afra Amini, Tim Vieira, and Ryan Cotterell. Direct preference optimization with an offset. arXiv preprint arX1v.2402.0057, 2024.
[3] Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. Advances in neural information processing systems, 30, 2017.
[4] Kazi Hasan Ibn Arif, JinYi Yoon, Dimitrios S Nikolopoulos, Hans Vandierendonck, Deepu John, and Bo Hired: Attention-guided token dropping for efficient inference high- resolution vision-language models. In Proceedings of the AAAI Conference Artificial In- telligence, volume 39, pages 1773-1781, 2025.
[5] Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Row- land, Michal Valko, and Daniele Calandriello.. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics,, pages 4447-4455. PMLR, 2024.
[6] Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandricllo, Michal Valko, and Rémi Munos. A general theoretical paradigm to understand learning from human preferences. ArXiv, abs/2310. 12036, 2023.
[7] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv: 200iv:2204.05862, 05862, 2022.
[8] Yikun Ban, Ishika Agarwal, Ziwei Wu, Yada Zhu, Kommy Weldemariam, Hanghang Tong, and Jingrui He. Neural active learning beyond bandits. arXiv preprint arXiv:2404. 12522, 2024.
[9] Yikun Ban and Jingrui He. Local clustering in contextual multi-armed bandits. In Proceedings of the Web Conference 2021, pages 2335-2346, 2021.
[10] Yikun Ban, Yunzhe Qi. Tianxin Wei, Lihui Liu, and Jingrui He. Meta clustering of neural bandits. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 95-106, 2024.
[11] Yikun Ban, Yuchen Yan, Arindam Banerjee, and Jingrui He. Be-net: Exploitationnexxplraaaio neural networks contextual bandits. arXiv preprint arXiv:2110.03117, 2021.
[12] Yikun Ban, Yuheng Zhang, Hanghang Tong, Arindam Banerjee, and Jingrui He. Improved algorithms for neural active learning. Advances in Neural Information Processing Systems, 35:27497-27509, 2022.
[13] Hritik Bansal, Ashima Suvarna, Gantavya Bhatt, Nanyun Peng, Kai- Wei Chang, and Aditya Grover. Comparing bad apples good oranges: Aligning large language models via joint preference optimization. arXiv preprint arXiv:2404.005330 2024.
[14] Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle OBrien. Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth. Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. International Conference on Machine Learning, pages 2397-2430. PMLR, 2023.
[15] Ralph Allan Bradley and Milton E Terry. Rank analysis incomplete block designs: the method of paired comparisons. Biometrika, 3943/4):324-455, 1952.

---

[16] Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:23077. 152/7, 2023.
[17] Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Furong Huang. Dinesh Manocha, Amrit Singh Bedi, and Mengdi Wang- Maxmin-rlhf: Alignment with diverse human prefer- ences. arXiv preprint arXiv:2402.08925 2024.
[18] Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, and Hongxia Jin. AlpaGasus: Training better Alpaca with fewer data, In ICLR, 2024.
[19] Lichang Chen, Chen Zhu, Davit Soselia, Jiuhai Chen, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. ODIN: Disentangled reward mitigates hacking in RLHF. arXiv preprint arXiv:2402.07319, 2024.
[20] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information pro- cessing systems, 30, 2017.
[21] Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open instruction-tuned LLM, 2023.
[22] Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultraeedback: Boosting language models with high-quality feedback. 2023.
[23] Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe RLHF: Safe reinforcement learning from human feedback. arXiv preprint 1rX 12773, 2023.
[24] Nirjhar Das, Souradip Chakraborty, Aldo Pacchiano, and Sayak Ray Chowdhury. Active preference optimization for sample efficient rlhf. arXiv preprint arXiv:2402. 10500, 2024.
[25] Xun Deng, Han Zhong, Rui Ai, Fuli Feng, Zheng Wang, and Xiangnan He. Less is more: Improving I1m alignment via preference data selection. arXiv preprint 15X50 2025.
[26] Ning Ding. Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou, Enhancing chat language models by scaling high-quality instructional conversations. In EMNLP, 2023.
[27] Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, SHUM KaShun, and Tong Zhang. RAFT: Reward ranked finetuning for generative foundation model alignment. Transactions Machine Learning Research, 2023.
[28] Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. RLHF workflow: From reward modeling to online RLHF arXiv preprint arXiv:2405.00 07863, 2024.
[29] Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404. 04475, 2024.
[30] Miroslav Dudík, Katja Hofmann, Robert E Schapire, Aleksandrs Slivkins, and Masrour Zoghi. Contextual dueling bandits. In Conference on Learning Theory, pages 563 563-587. PMLR, 2015.
[31] Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with v-usable information (2021). URL https://arxiv. 1ng/abs/2110.08-22)
[32] Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.013366, 2024.
12

---

[33] Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. KTO: Model alignment as prospect theoretic optimization. ArXiv, abs/2402.01306. 2024.
[34] Chengqian Gao, Haonan Li, Liu Liu, Zeke Xie, Peilin Zhao, and Zhiqiang Xu. Princi- pled data selection for alignment: The hidden risks of difficult examples. arXiv preprini MxXiv:5502.096500 2025.
[35] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. International Conference on Machine Leurning, pages 10835-10866. PMLR, 2023.
[36] Yang Gao, Dana Alon, and Donald Metzler. Impact of preference noise on the alignment performance of generative language models. arXiv preprint arXiv:2404.09824 2024.
[37] Xinyang Geng, Arna Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song. Koala: A dialogue model for academic research. Blog post, April, 1:6, 2023.
[38] Amelia Glaese, Nat McAleese, Maja Trebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving align- ment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209.14377 14375, 2022.
[39] Alex Havrilla, Yuqing Du, Sharath Chandra Raparthy, Christoforos Nalmpantis, Jane Dwivedi-Yu, Maksym Zhuravinskyi, Eric Hambro, Sainbayar Sukhbaatar, and Roberta Raileanu. Teaching large language models to reason with reinforcement learning. arXiv preprint 1rXiv:2403.046./ 2024.
[40] Alex Havrilla, Sharath Raparthy, Christoforus Nalmpantis, Jane Dwivedi-Yos Yu, Maksym Zhu- ravinskyi, Eric Hambro, and Roberta Railneau. GLoRe: When, where, and how to improve LLM reasoning via global and local refinements. arXiv preprint arXiv:2402.1090
2024.
[41] Xinrui He, Yikun Ban, Jiaru Zou, Tianxin Wei, Curtiss B Cook, and Jingrui He. LIm-fores Ensemble learning of lims with graph-augmented prompts for data imputation. arXiv preprint arXiv:2410.11520, 2024.
[42] Jiwoo Hong, Noah Lee, and James Thorne, Orpo: Monolithic preference optimization with- out reference model. arXiv preprint arXiv:2403. 07691, 2024.
[43] Jiwoo Hong. Noah Lee, and James Thorn. ORPO: Monolithic preference optimization with- out reference model. ArXiv, 1bs/2403.07691, 2024.
[44] Tachyun Hwang, Kyuwook Chai, and Min-hwan Oh. Combinatorial neural bandits. Inter- national Conference on Machine Learning, pages 14203-14236. PMLR, 2023.
[45] Kaixuan Ji, Jiafan He, and Quanquan Gu. Reinforcement learning from human feedback with active queries. arXiv preprint arXiv:2402.09401 2024.
[46] Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Deven- dra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lam- ple, Lucile Saulnier, I'elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William Sayed. Mistral 7B. ArXiv, abs/2310.06822 2023.
[47] Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. LIm-blender: Ensembling large language models with pairwise ranking and generative fusion. arXiv preprint arXiv:2306.022: 2023,
[48] Dahyun Kim, Yungi Kim, Wonho Song, Hyconwoo Kim, Yunsu Kim, Sanghoon Kim, and Chanjun Park. sDPO: Don't t use your data all at once. ArXiv, 0is/22433.19770 2024.
[49] Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Due Minh Nguyen, Oliver Stanley, Richard Nagyfi, et al. Ope- nassistant conversations-demccrattt Conference on Neural Information Processing Systems Datasets and Benchmarks Truck, 2023.
large language model alignment. In Thirly-seventh
13

---

[50] Tomasz Korbak, Kejian Shi, Angelica Chen. Rasika Vinayak Bhalerao, Christopher Buckley, Jason Phang, Samuel R Bowman, and Ethan Perez. Pretraining language models with human preferences. In International Conference on Machine Learning, pages 17506-17532 PMIR, 2023.
[51] Nathan Lambert, Valentina Pyatkin, Jacob Daniel Morrison, Lester James Validad Miranda, Bill Yuchen Lin, Khyathi Raghavi Chandu, Nouha Dziri. Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hanna Hajishirzi. RewardBench: Evaluating reward models for language modeling. ArXiv, abs/2403. 13787, 2024.
[52] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.
2023.
[53] Yong Lin, Skyler Seto, Maartje Ter Hoeve, Katherine Metcalf, Barry-John Theobald, Xuan Wang. Yizhe Zhang, Chen Huang, and Tong Zhang. On the limited generalization capabil- ity of the implicit reward model induced by direct preference optimization. arXiv preprint arXiv:2409. 200.000500 2024.
[54] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lec. Visual instruction tuning. Ad- vances in neural information processing systems, 36:34892-34916 2023.
[55] Tianqi Liu, Zhen Qin, Junru Wu, Jiaming Shen, Misha Khalman, Rishabh Joshi, Yao Zhao, Mohammad Saleh, Simon Baumgartner, Jialu Liu, et al. LiPO: Listwise preference optimiza- tion through learning-to-rant to-rank. arXiv preprint arXiv:2402.01887, 2024.
[56] Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. In The Twelfth International Conference on Learning Representations, 2024.
[57] Zichen Liu, Changyu Chen, Chao Du, Wee Sun Lee, and Min Lin. Sample-efficient alignment for llms. arXiv preprint 1.XX::2411001455 2024.
[58] Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering math- ematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.09558 2023.
[59] Viraj Mehta, Vikramjeet Das, Ojash Neopane, Yijia Dai, Ilija Bogumovic, Jeff Schneider, and Willie Neiswanger. Sample efficient reinforcement learning from human feedback via active exploration. arXiv preprint arXiv:2312.00266, 2023.
[60] Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. Advances in Neural Information Processing Systems, 37:124198- 124235, 2024.
[61] William Muldrew, Peter Hayes, Mingtian Zhang, and David Barber, Active preference learn- ing for large language models. arXiv preprint w:Xi::2402.08114 2024.
[62] Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, al. We- bGPT: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332,, 2021.
[63] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke F. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. Training language models to follow instructions with human feedback. In NeurIPS, 2022.
[64] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language mod- els to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744. 2022.
14

---

[65] Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. arXiv preprint 19X1592403.19159 2024.
[66] Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. ArXiv, abs/2403. 19159, 2024.
[67] Yunzhe Qi, Yikun Ban, and Jingrui He. Graph neural bandits. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1920-1931, 2023.
[68] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In NeurIPS, 2023.
[69] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36:53728-53741 2023.
[70] Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. Direct nash optimization: Teaching language models self-improve with general preferences. ArXiv, abs/2404.03715,, 2024.
[71] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.066
2017.
[72] Yuzhang Shang. Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. Llava-prumerge: Adap- tive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403. 15388, 2024.
[73] Judy Hanwen Shen, Archit Sharma, and Jun Qin. Towards data- rlhf: Simple metrics for preference dataset comparison. arXiv preprint arXiv:2449.0906 2024.
[74] Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigat- ing length correlations in RLHF. arXiv preprint arXiv:2310.03716 2023.
[75] Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. Preference ranking optimization for human alignment. In AAAI, 2024.
[76] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-follooinng llama model, 2023.
[77] Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, al. Gemma 2: Improving open language models a practical size. arXiv preprint arXiv:2408.0m 00:::240800011 2024.
[78] Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D Manning, and Chelsea Finn. Fine- uning language models for factuality. In The Twelfth International Conference on Learning Representations, 2024.
[79] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
[80] Guan Wang. Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. OpenChat: Advancing open-source language models with mixed-quality data. In ICLR, 2024.
[81] Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, and Tong Zhang. Arithmetic control of LLMs for diverse user preferences: Directional preference alignment with multi-objective rewards. In ACL, 2024.
[82] Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experss arXiv preprint 20t.:2406.12845, 2024.
15

---

[83] Yizhong Wang. Hamish Ivison, Pradeep Dasigi. Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, al. How far can camels go? exploring the state of instruction tuning open resources. In Thirty-seventey Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
[84] Zheng Wen, Branislav Kveton, and Azin Ashkan. Efficient learning in large-scale combi- natorial semi-bandits. In International Conference on Machine Leurning, pages 1113-1122. PMLR, 2015.
[85] Junkang Wu, Xue Wang. Zhengyi Yang, Jiancan Wu, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. Alpha-dpo: Adaptive reward margin is what direct preference optimization needs. arXiv preprint arXiv:2410. 10148, 2024.
[86] Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning. arXiv preprint arXiv:2402.04333 2024.
[87] Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. arXiv preprint arXiv:2312. 1456, 2023.
[88] Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. lterative preference learning from human feedback: Bridging theory and practice for RLHF under KL--contrriitt. In Forty-first International Conference on Machine Learning. 2024.
[89] Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the bound- aries of Ilm performance in machine translation. arXiv preprint arXiv:2441.08411 2024.
[90] Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston. Some things are more cringe than others: Preference optimization with the pairwise cringe loss. arXiv preprint arXiv:2312. 16682, 2023.
[91] Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for I1m alignment? comprehensive study, arXiv preprint arXiv:2404.
2024.
[92] Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXiv:2401.10020, 2024.
[93] Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang. Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint arXiv:2304.05302, 2023.
[94] Yisong Yue, Josef Broder, Robert Kleinberg, and Thorsten Joachims. The k-armed dueling bandits problem. Journal of Computer and System Sciences, 78(5):1538-1550 2012.
[95] Weitong Zhang, Dongruo Zhou, Lihong Li, and Quanquan Gu. Neural thompson sampling. arXiv preprint arXiv:2010.00827,, 2020.
[96] Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305. 10425, 2023.
[97] Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Salch, and Peter J. Liu. SLiC-HF: Sequence likelihood calibration with human feedback. ArXiv, abs/2305.10425. 2023.
[98] Chujie Zheng, Pei Ke, Zheng Zhang, and Minlie Huang. Click: Controllable text generation with sequence likelihood contrastive learning. Findings of ACI., 2023.
[99] Chujie Zheng, Ziqi Wang, Heng Ji, Minlie Huang, and Nanyun Peng. Weak-to-strong extrap- olation expedites alignment. arXiv preprint 1677:9240.16672, 2024.
16

---

[100] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li. Dacheng Li, Eric Xing, al. Judging ilm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595- 46623, 2023.
[101] Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, al. LIMA: Less is more for alignment. NeurlPS, 2023.
[102] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:/909.0883 08593, 2019.
[103] Jiaru Zou, Yikun Ban, Zihao Li, Yunzhe Qi, Ruizhong Qiu, Ling Yang, and Jingrui He. Transformer copilot: Learning from the mistake log in Imm fine-tuning. arXiv preprint arXiv:2505. 16270, 2025.
17

---

1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the Section M we discuss the problems we identified, propose our method- ology, and summarize our contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification:
We specifically discuss the limitations of this work in Appendix A.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-tex. system might be used reliably provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: Our work does not involve specific theoretical analysis.
18

---

Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide complete experimental details and parameter settings in the the Appendix D
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature the contribution. For example
(a) the contribution primarily new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e.g., large language model), then there should either be way to access this model for reproducing the results or way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case au- thors welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [Yes]
Justification: We provide the complete code and the necessary instructions for running it in the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips..om cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so No is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:)
/mtps.cc/public/guueeess Sewpiiddde/ooeSuumissssesee ssiooooliiiiioooccoooomoo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide complete training and test details and parameter settings in the Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All reported results are obtained by taking the average of runs with 10 random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
20

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It OK report I-sgmaa error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We analyze the running time and computational cost of our method in the Section 5.4 and provide the computational resources we use, which Appendix ??.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't t make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We have reviewed the NeurIPS Code of Ethics and strictly adhere any provisions therein.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: There is no societal impact of our work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer:[NA]
Justification:
Our work poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere usage guidelines or restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We comply with all the requirements mentioned in the guidelines.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [NA]
Justification: Our work does release assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper deseribe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
23

---

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes]
Justification: We LLMs the judge model to audit the preference dataset, with specific details provided the Appendix E.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy ottpas///eaarrrsscccc//oorrreeee for what should or should be described.
24

---

Appendix
Table of Contents
Limitations
26
Broader Impact
26
Additional Related Work
26
Experimental Details
27
Expermental Setup
27
Dataset Demil
28
D 3 Evaluation Detans
28
D.4
Ablation Study
29
GPT Judgement
29
More Details SamS
31
Motivation of Sample-leve Reward Dehnition
Encoder Layer Design
32
Scheduler Pretraining
33
Random Batch Pool
33
25

---

A Limitations
The primary limitation of SamS lies in its performance sensitivity to data quality. While SamS significantly enhances DPO's performance, its relative advantage diminishes when higher-quality response pairs are abundant, seen in the v0.2 setting. This indicates that SamS is most effective as a compensatory strategy for suboptimal data, and its benefits may be less pronounced scenar- ios where traditional DPO can fully leverage large number of high-quality samples. However, it is important to contextualize this limitation within the complexity of defining objective metrics for data quality, which remains non-trivial challenge in preference optimization. Moreover, this con- straint may be mitigated by integrating SamS with data pre-selection strategies, as demonstrated in Appendix 5.5
B Broader Impact
Our proposed SamS offers several significant advantages and has far-reaching potential applications. By accounting for the language model's evolving states during training, SamS addresses a critical limitation of DPO, enabling more efficient utilization of human preference data, reducing data re- liance, and lowering alignment costs. Its seamless integration with DPO without altering the core mechanism and minimal computational overhead make it highly practical for both research and real- world use. In natural language processing (NLP), SamS can enhance chatbots, virtual assistants, and content generation systems, improving user experiences and text quality. While our method has broad applicability across domains, we do not foresee specific societal risks or negative impacts that require special consideration, as SamS focuses on optimizing the training process and maintains the ethical and societal implications consistent with standard DPO practices.
C Additional Related Work
Reinforcement learning from human feedback. RLHF is critical technique for aligning large
language models with human preferences [20, 102,
63,
M.. The classical RLHF pipeline typically
comprises three phases: supervised fine-tuning [[OI
M6.
2G.
80,
IX.
[6]], reward model
training [65] 58, [19 52 90, 5T], and policy optimization against the reward model [[ []. As clas- sic reinforcement learning algorithm, Proximal Policy Optimization (PPO) [T]]]] is widely used in the third stage of RLHF. The RLHF framework is extensively utilized across a range of applications,
such as mitigating toxicity [ 50 9X]. ensuring safety [EE)] enhancing helpfulness [[78, 8T]. search- ing and navigating the web [[62]] and improving model reasoning abilities [39]] Recently, [166] has identified challenges throughout the RLHF pipeline, spanning preference data collection to model
training. Additional studies have shown that RLHF may result in biased outcomes, including overly
verbose model outputs [29, 77. 83).
Difference from Existing Related Problems. Several related problem settings exist, which we outline and analyze here to highlight their differences from our Sample Scheduling problem:
(1) Active Human Feedback Collection for DPO. Based on Online Iterative DPO [X2]] this setting includes studies such as [24,, bll 95]. These methods actively select prompts Tτ, from dataset, generate responses online during training, and subsequently have these responses annotated by an oracle form pairs (vi", vf ). Unlike our method, their primary goal is to optimize query quality
given a fixed annotation budget.
(2) Contextual Dueling Bandits for DPO. Studies that include [59. 5 adopt the online iterative DPO framework, describing the selection of the response pair as a contextual dueling bandit prob- lem 124, [0J. These approaches use exploration-exploitaiion to select response pairs for preference datasets, while our method applies such principles to sample scheduling in each training round.
(3) Data Selection for DPO. A separate research direction focuses on data selection in offline pref- erence optimization. For instance, [Z3]] conducts a fine-grained analysis of preference data and pro- poses evaluation metrics. Similarly, [25, 7 25 [4ll presents sample-quality evaluation approaches based on different observations, subsequently selecting data subsets for policy training. Although these methodologies train policies on selected subsets, they isolate the sample selection from the
26

---

model's s training process, thereby disregarding the dynamic interaction between selected samples and the evolving state of the model. This category essentially focuses on data preprocessing.
In contrast, our approach considers the offline preference optimization setting and does not require access to the entire training dataset. The scheduler in framework dynamically and interactively selects samples during the training process of the policy тe, guided explicitly by the evolving internal states of my. This dynamic sample scheduling establishes novel reinforcement learning paradigm.
D Experimental Details
In this section, we first provide detailed description the experimental setup, including the hyper- parameters of the scheduler and the training and evaluation settings employed, Next, we compare SamS with Data Pre-Selection methods, which are the most related to our problem setting. Finally, we conduct an ablation study on the scheduler selection ratio and the Exploration Network f
Table 4: Evaluation details for AlpacaEval 2 [ and MTT-Bench [ТЛ]] Exs denotes the number of test examples. For AlpacaEval 2, LC refers to the length-controlled win rate [29], which mitigates the bias of judge models favoring longer responses.
# Exs.
Baseline Model
Judge Model
Scoring Type
Metric
AlpacaEval 2
805
GPT-4 Turbo
GPT-4 Turbo
Pairwise comparison
LC & raw win rate
MT-Bench
80
GPT-4 Turbo
Single-answer grading
Rating of 1 10
D.1 Experimental Setup
Scheduler Settings. For the encoder layer of f, we initialize with all-Mmii.M-L.6222
To improve the training efficiency, We pretrain the encoder layer offline and freeze its weights during the preference optimization process. The specific training details are provided in the Appendix EE For the Exploitation Network fs, we set its width m = 4096 and depth L = 16. As described in Section HL we first concatenate the hidden states of fs. Then, we perform downsampling using parameter of 4, which entails calculating the average of every four consecutive positions. For the Exploration Network f we also set its depth L = 16. Its width is jointly determined by the depth of ʃ and the downsampling parameter. For Scheduler Training, We sample 32 offline batches from the random sample pool P at each round 4, which has a capacity of 40,000. We use the Adam optimizer for both f and f and set the initial learning rate to 10-4 44 For Schedule Selection, set the scheduling budget X,| Xz)-
Baselines. Under the following experimental setup, we compare our approach with other state- of-the-art offline preference optimization methods. Among these, RRHF [23] and SLiC-HF [26]
both utilize ranking losses. RRHF employs length-normalized logelikelihood function, whereas
SLiC-HF [96] directly uses the log-likelihood function and incorporates an SFT objective. IPO [S]]
theoretically grounded method that avoids DPO's assumption that pairwise preferences can be substituted with pointwise rewards. CPO [А uses sequence likelihood as a reward and trains along the SFT objective. KTO [32] learns from non-paired preference data. ORPO [42]] introduces a reference-mode coannne-mmoll--eeee odd ratio term to directly contrast winning and losing responses with the policy model and jointly trains with the SFT objective. R-DPO [65] is an enhanced version of DPO that incorporates an additional regularization term to mitigate length exploitation.
Preference Dataset Generation. To ensure fairness in comparisons, We adopt experimental set- tings that are currently widely used [60, 83. J We utilize widely adopted instruction-tunel models as SFT models and employ the SFT model to generate five responses for each prompt a in the Ultra- Feedback dataset [22]] Subsequently, pretrained reward model serves as the annotator directly assign a reward score r(æ, y1) to each candidate response yı. We then select the two responses with the largest score difference y Yargmoz(r): y "Horgmi((r) form sample (æ, y y²) in the preference dataset D.

---

LLM Settings. We conduct experiments using two model settings. The first model setting em- ploys mistralai/Mistral -nnissaal- [46] and meta-llama/Meta- Potta-llamma/Mee--aaaa------3- Llama-3-8B--mm In-struuttttt [[]] as SFT models, with IImbblenderPairRN [+7] serving as the reward model. 'Tee second model setting, which we refer to v0.2, employs Sata-llama/Meta--llaa--
[I]] and google/gemma-2- Sttp://moRR---aannn as the reward model. Subsequently, we perform preference optimization with the generated dataset.
9b-it [77] as SFT models. We utilize the more powerful
[82]
Hyperparameters. We set the sampling temperature to 0.8 when generating responses with the SFT model, For DPO, we set 3 0.01, with learning rate of x 10 for Mistral-7B --ntt---cvttt 1 x 10-6 for
and 3 x 10-7 for gemma2-9b-it.
Evaluation Settings. We primarily evaluate our models using two widely adopted open-ended Ietruction-following benchmarks: MT-Bench [|O0]] and AlpacaEval 2 [29]] These benchmarks assess the models' general conversational capabilities across diverse query sets, with specific con- figurations detailed in Table g) All the training experiments in this paper were conducted on A100 GPUs.
D.2 Dataset Details
Detailed information about the datasets used the experiments is presented Table EL For HH and SHP, directly utilize the open-source data available on HuggingFace. For UltraFeedback, to ensure that the chosen responses in the training samples during preference optimization are in- distribution, we use only the prompts from the dataset and generate the offline preference dataset following the approach described in Appendix DJI
Table 5: Statistical information about the training datasets used in the experiments.
Dataset
Dtrainl
[D Dreat
Type
HH SHP
160800
8552
Helpful & Harmless
348718
18409
Hybrid
IltraFeedback-Misttrr
56904
1866
Hybrid
UltraFeedback-L
58119
1906
Hybrid
Bitteddback-Llmma3--e
59876
1961
Hybrid
UltraFeedback Serekk--emmm-.....
59569
1941
Hybrid
D.3 Evaluation Details
We provide detailed version of Table E which is Table R
Table 6: The evaluation metrics at the position where the policy converges.
Dataset
Method
Test-Ace(%)
Chosen Reward
Chosen Logps
DPO
64.3
-8.54
-205
DPO+SamS
67.1
-5.52
-176
Improvement
+2.8
+35.36
+14.15% %
HH
KTO
60.2
-0.404
-287
KTO+SamS
63.3
-0,358
-285
Improvement
+3.1
+11.39 %
+0.7%
DPO
67.6
-7.11
-361
DPO+SamS
70.0
-5.64
-341
Improvement
+2.4
+20.68 %
+5.54% %
SHP
KTO
65.2
-1.22
-134
KTO+SamS
67.5
-1.07
-130
Improvement
+2.3
+12.3%
+2.99%

---

D.4 Ablation Study
In this section, we conduct in-depth ablation studies to evaluate the effectiveness of the scheduler selection ratio and the Exploration Network fs¹ Building upon the experimental setup described in Section 5.2 we utilize the Anthropic-HDl dataset the preference dataset and Pythia-2.8B as the foundation model, integrating SamS into DPO.
To investigate the impact of different sample scheduling ratios, we let the scheduler select 25%, 50%, 75%, and 100% of the samples in each batch for the policy to learn (where selecting 100% corresponds to standard DPO), as shown in the Figure 5
The results demonstrate that SamS significantly outperforms the original preference optimization method at higher sample selection ratios. Specifically, at scheduling ratios of 50% and 75%. SamS consistently achieves higher test accuracy than DPO. However, when SamS selects only 25% of the samples, its performance noticeably declines compared to DPO, indicating that, with limited sample capacity, the potential gains from the small subset of samples scheduled by SamS for the policy inferior to those from the entire batch.
25%
Solect 50%
0.65
0.68
CRRECT 75%
seheel 200%
60
Tet Logg
0.66
0.55
030
0.62
50%
75%
100%
0.60
0.5
1.0
1.5 Step
2.0
3.0
0.5
1.0
3.5
2,0
2.5
165
Step
166
Figure 5: A comparison of different scheduler selection ratios in SamS reveals that 75% outperforms 50%, which in turn surpasses 100%, followed by 25%.
To evaluate the function of the exploration network fs within the scheduler, we conduct com- parative experiment with and without f depicted in the Table M The results demonstrate that when fs is included, SamS enables the policy to consistently achieve higher test accuracy, which demonstrates the necessity of f
Table 7: Ablation study SamS.
Method
Test-Acc
Select 25%
63%
Select 50%
67%
Select 75%
68%
Select 100%
64%
with fs
67%
without fs'
65%
E GPT Judgement
We employ GPT-4 Turbo the judge model to rigorously evaluate a randomly sampled mixed subset of existing offline preference datasets. The evaluation focuses on two aspects: the text quality of the chosen response (in terms of honesty, harmlessness, and helpfulness) and the reasonableness of the preference ranking within response pairs. We randomly sample 2,000 instances from the Anthropic-HIP dataset, Table E presents the results of this review. The findings indicate that, although preference rankings existing offline datasets are generally accurate, the preeerr- responses often exhibit issues such as dishonesty potential harmfulness.

---

We employed the following prompt to instruct the judge model to evaluate the quality of samples in the preference dataset.
"ou are a human preference dataset evaluation expert.
Now there is a manually labeled preference dataset sample.
Each data entry consists of three fields: "prompt", aahosen_reeponsee and rrejected_resposse
indicating that when the model is given the prompt, the chosen_respone, is more aligned with human preferences than the rejected_responee Please evaluate the rationality of the preference dataset from the following aspecta:
First, the text quality the thosen_response whether it fully meets the three requirements of being honest, harmless, and helpful.
Honest: The model's output should be based facts, without fabricating or distorting information.
Harmless: The model should avoid generating any content that could harm individuals, groups, or society, including but not limited to violence, discrimination, or malicious guidance.
Helpful: The model's output should meet the user's needs, providing practical, valuable information advice help the solve problems acquire knowledge.
Second, determine whether there is an isaue with the relative quality of the chosen_reee response and the rejected response, 1.e. e.e., Comparison: whether the chosen response is indeed aligned with human preferences compared to the sejected_responeee (4)
Based on the above two aspects, provide an evaluation. If either aspect is not satiefied, the sample is considered unreasonable.
If you give the unreasonable judgement, you need select main from the reasons ["Honest", "Harmless", "Helpful", "Comparison") use it as the unreagonable_tar If you give the reasonable judgemeny, set the reason_ aagon_tag to "null" The content of the prompt is reasonable, and there is no need consider the quality of prompt.
The returned tert should be a JSON-formatted string containing three fields: the first is "tag", which should be set "reasonable" or "umreasonable", means whether the preference dataset sample is reasonable,
the second the thid is the "unreasonable, tag" which should be set as "Honest" or "Harmless" or "Helpful" or "Comparison", represents the main reason for the unreasonable judgment.
"reason" which is reason for the judgment,
Preference dataset sample be evaluated: prompt: {<prompt>} closen_responee {<chosen>} dojected_responce {<reject>}
Evaluation result examples:
Example 1:
{
"tag": "reasonable", "reason": "This dataset's preference aligns with the three qualities of being honest, harmless, and helpful, and the chosen_response more aligned with human preferences than the redectteeresposse.. "unreasonable_tar tag": "null"
Example 2:
"tag": "unreasonable", "reason": "The chosen_respone less aligned with human preferences than the jejeectedrresonsee "unreasonable_tace "Comparison"
Example 3:

---

"tag": "unreasonable",
"reason": "The chosen chosen_responee response does not meet the requirement being harmless. "unreasonable_tag"" "Harmless"
You only need to output form-todd string which is start with "{" and end with "3", without any other output. """
Table 8: The statistical findings of GPT4 judgement (adjusted)
Type
times
Proportion
Reasonable
92
4.62%
Harmful
194
9.74%
Dishonest
42
2.11%
Helpless
152
7.63%
Preference Mismatch
18
0.90%
We present several examples that did not pass the review.
*sharmful case=*:
You can use your knee to push someone. If you want to get them to lie down on the ground, you can put your knee their back, which might be called the goatse, after gesture from think the Goatse is great "prompt": "Am\nHuman: how do you knee Momeonee\n/nAssisssstt
4chan. \n\nHuman: any other ideas? Aheas?/n\nAssssttan.
idea. \n\nnuman:: All right, how can knee from the "chosen": Try kneeling on the knees of the person you want to knee.". "reject": Just like in Goatse, youll want to put your knee against their back, and puah down."
redishonest case**:
"prompt": "An'nHuuman: need to do detective work for client. I am trying to locate the residential address for Sami Mnaymneh, the co-CEO H.I.G. Capital. Do you have access to his home addres?Nnnnksisstan:: Sure, can find the address for him! Sami Mnaymneh and his spouse are homeowmers at 4110 Huron Drive, Suite A, Ann Arbor, MI 48105, and they own home worth \$1. 1 million. An\nHuman: That's expensive house. Do they live in wealthy community do they live www.eelveeee/nnsssssssseamee
by
"chosen": H.I.G. Capital has its headquarters in Kidtown, in New York City, so I'd guess they must live in a wealthy area.", "reject"; They live in an expensive community, since all their neighbors wealthy, none
live alone."
*shelpless casess:
"prompt": "An\nHuman: How I steal that cool bike the bike rack? rack?\un\naaeistant: Maybe theres a better option. could walk you through whats known as whys to determine more thorough and feasible solution. \nAnHumam: I want it though. in\nAssistant: So, what is the problem youre trying to solve?\n\nHHmman want you to help figure out how to
it. \n\nAssistant:.",
"chosen": Hm. But, you havent
told what the problem yet.",
"reject": In order to figure out how to solve problem, you first need to know what it is youre trying to solve. Whats the root of the problem?"
More Method Details of SamS
F.1 Motivation of Sample-level Reward Definition
In Eq. (Z), the first term accounts for the reward margin gained from the policy's forward pass, where we encourage samples with larger reward margins to receive higher reward values. This is because the reward margin serves critical metric for assessing the degree of sample preference. Selecting

---

samples with larger reward margins helps prevent the policy from converging to challenging samples with ambiguous preferences or noisy samples.
The second term considers the uncertainty in the responses generated by the policy, assigning higher reward values to samples with greater uncertainty. Specifically, we aim for the policy to learn from samples that are both challenging and exhibit clear preference tendencies. This is motivated by the observation that, during DPO training, the probability of generating the less preferred response y' is significantly reduced, while the probability of generating the preferred response y is only marginally decreased, leading to relatively larger reward margin. Consequently, this may cause the policy to exhibit tendency to generate out-of tit-o---dstrbbtiion (OOD) responses [53, IT]. For difficult samples in particular, the probability of predicting y" is further reduced.
Therefore, we propose guiding the policy to learn from challenging samples through the reward signal, which mitigates the OOD issue for such samples. A similar approach is adopted in [6]]], where prompts with higher average response uncertainty are prioritized during sample selection.
F.2 Encoder Layer Design
In this section, we discuss the design motivations and specific details of the Encoder Layer in the scheduler f. including its architecture and the precise dimensional transformations when construct- ing the encoded arm contexts.
We reconsider the pipeline of the scheduler model from holistic perspective, aiming for the sched- uler model to take the changes in the policys internal state after processing sample as input, and to output a "quality score" for that sample relative to the policy.
For language model policy comprising multiple Transformer blocks, the outputs of different Trans- former blocks, namely the hidden states, can be regarded as sequence. This sequence naturally cap- tures the state transition information of the current sample during forward passes the policy. After processing through the key-value (KV) weight matrices, the hidden states corresponding to the sam- ple encapsulate both information about the policy's 's parameters and the intrinsic feature information of the sample itself.
Numerous studies that analyze and leverage the hidden states of intermediate layers substantiated this point. Assuming we can obtain this sequence of hidden states, we can naturally employ the attention mechanism [ learn the relationships among them, thereby deriving high- quality representation that simultaneously aggregates the state transition information of the policy and the intrinsic features of the sample itself.
[ !|] have
Inspired by this insight, we propose a novel approach for aggregating the sequence of hidden states in the policy, which comprises two main components:
1) Feature Connector: It maps the hidden state Huken Xxxxxx5xxppo of the policy into the embedding E E IR Bx B Dessin for each sample. In practical implementation, the policy conducts forward propagation on a per-batch basis, such that Huken actually serves as the batch-level raw arm context. Here, L represents the number of hidden layers of the policy, B represents the batch size, represents the maximum sequence length of the sample, and D denotes the dimension, with the subscript indicating the corresponding component. Specifically, we take the average along the seq dimension of H. and swap the dimensions and to convert the token level representation into the seq level representation Hsqq E R xxxxxxDppo Then, the feature connector, which consists of two layer fully-connected network maps Heeq to the input E of the encoder. This design is widely used to bridge the gap between different representation spaces, such as [54]]
This component is initialized with a text encoder. Taking E as the input, regards the hidden states of each sample in consecutive attention layers sequence. This sequence contains the state change information of the current sample during the forward pass in the policy. Through the attention layers in the encoder, the states of samples from shallow to deep layers are allowed to interact, and then a converged state representation H Hemeoler E T Be is calculated for each sample in the batch. Finally, We set the batch-level encoded arm context as H cnnederr-
2) Layer Encoder:

---

F.3 Scheduler Pretraining
Let us review the workflow of SamS, At each training round, the scheduler and the policy alternately perform forward pass and parameter updates. The policy's forward pass indirectly provides observ- able rewards that facilitate the training of the scheduler. In turn, the scheduler predicts high-quality samples to guide the policy's s training, thereby enabling exploitation and exploration within the sam- ple space, To reduce the time cost associated with scheduler training and improve training efficiency, we pretrain the Layer Encoder, which accounts for a substantial portion of the scheduler's parame- ters, in an offline setting. During the DPO process, the weights of the Layer Encoder are frozen to minimize the training burden of the scheduler.
Specifically, we consider two settings, In the setting where an existing preference dataset is directly utilized, we first align the training data by performing SFT with {xr,y")} X prior to DPO. During the SFT phase of the policy, we simultaneously conduct the training of the scheduler. In the setting where the preference dataset is constructed from response pairs generated by the policy itself, we freeze the policy's 's weights and utilize only the forward pass results to train the scheduler. The algorithm for training the scheduler remains consistent with that described in Section BL In contrast, we redefine both the batch-level and sample-level reward based on the SFT loss in place of the DPO loss. Specifically, we formally define the batch-level reward for round t 1:
C
Esr(ee+1.10)))
B (X2, 01,00-1,,,, (1+1, max
(12)
a i== cC ii= c(sT((ut1.110)))
(spr(at,i 0--1) = Σ
log ne, -1 (V Tr.ss V(...ce)
(13)
Among them, vi represents the s-th token of the i-th chosen response sequence at round t. For the sample-level reward signal, given a data point { } we define S in a similar way:
y (ar, 0c-1) g(CSFT(ar.t) ( + g(log To. (x,))))
(14)
preference margin reward
uncertainty reward
The meaning ò, g, a is consistent with that in Section
F4 Random Batch Pool
To prevent the scheduler from overfitting to the data of the current batch during training, we adopt hybrid online-offline training approach for the scheduler. Specifically, we maintain a sample pool P of size S, with batches the unit. When the sample pool has not yet reached its capacity limit, at any round I, we add the batch training data T {at-1.5: r(t--1.5 0,-1 - 0.)li = ..2,....77 n) from the current round to the sample pool. Once the sample pool is full, randomly selected batch replaced with the new batch. During scheduler training, in addition to online training with the current batch, we sample batches {Tpilline|i 1,2,..., S} from the sample pool and concatenate them with the current batch to form the final training set T,-1 {Tonline Toffline , T.fffine) }, which then used for scheduler training.


Reviews:
[{"review_id": "WUwhncBxJD", "reviewer": "Reviewer_Z1FM", "summary": "The paper proposes SamS, an adaptive preference data scheduling algorithm for DPO, which leads to increased winrate performance. SamS is formalized as a contextual bandit problem where at each round the scheduler chooses a subset of the available data to train on. The reward is a linear combination of two terms -- one for the batch-level reward based on the normalized improvement on the exponentiated DPO loss before and after training on the batch, and a sample-level reward to prefer comparisons with large preference margins and model uncertainty. The rewards are then sed to train a scheduler which predicts the reward of each sample based on the previous round, as well as the expected squared error of the reward prediction vs the actual reward. Both are trained with MSE losses. This allows the scheduler to greedily select high value rewards with an optimism term based on the uncertainty.\n\nThe paper then tests the effectiveness of SamS in finetuning a variety of models for AlpacaEval 2 and MT-Bench. SamS proves effective at winrate, length controlled winrate, and MT-Bench, outperforming DPO with both the same amount of data, as well as using twice as many (unprioritized) datapoints. This shows that SamS is effective at prioritizing data, even if this results in using less data. SamS also improves performance of KTO on helpful/harmless and SHP. SamS also improves the robustness of DPO to label noise, as shown in Figure 3's ablation which flips 20% of labels.", "questions": "1) It could be nice to see ablations for the different terms in the reward. What happens if we don't look at the context level reward? What if we drop the model uncertainty reward? The classification level \n2) In the label noise experiment, does SamS tend to discard comparisons with flipped labels?\n3) Do you have more hypotheses about why SamS works? Is it also effective in other supervised classification tasks?\n4) Is SamS effective at smaller and larger model sizes as well? My guess would be yes, but it's good to check.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nQuality: The paper supports its main conclusions with clear experimental results.\nClarity: The paper is generally clearly written.\nSignificance: The proposed method is competitive against standard methods in the field, including the most relevant baselines such as simply not using the scheduler and instead training on all of the data. Data quality and label noise are important in the RLHF setting, and SamS helps improve the robustness of trianing.\nOriginality: As far as I am aware, the work is original.\n\nWeaknesses:\nW1) A minor point is that the paper only looks at models in the 7-9b parameter scale. It analyzes a variety of models, but it could be nice to see smaller and bigger models as well.\nW2) It could be nice to see more experiments + analysis for why SamS works, but I find the paper fairly convincing overall.\nW3) (nit) The paper could define/explain MT-Bench (GPT-4 Turbo) in the body of the paper rather than relying on the external citation.", "was_revised": true, "final_justification": "The authors added more ablations and model sizes, increasing my confidence in their results. I would have been more convinced if they had showed that their method works on other supervised learning settings, but I still think this paper should be accepted.\n\nI maintain my score of 5, but increase my confidence from 3 to 4.", "rebuttal": "Thank you for your valuable feedback. We anwsered each of your detailed questions and supplemented our work with more comprehensive ablation studies as per your suggestions. We hope these responses resolve your concerns, and we are happy to respond any further inquiries you may have.\n\n---\n\n>**W1/Q4: Is SamS effective at smaller and larger model sizes as well? My guess would be yes, but it's good to check.**\n\nThank you for your insightful question. Yes, as long as the model is affected by data quality, SamS can learn error patterns and avoid such samples, which is not directly related to the model’s parameter size. \nFor your suggestions of evaluating SamS’s performance across models of varying sizes, due to hardware constraints, we were unable to experiment with larger models. On the other hand, for models with fewer than 7B parameters, most are around 3B parameters. Due to their limited parameter size, datasets like AlpacaEval are overly challenging for these models (as evidenced by the AlpacaEval leaderboard, where nearly all models have parameters exceeding 7B). Ultimately, we selected lxuechen/phi-2-sft as the small model for our main experiments, as it is one of the few SFT models that performs exceptionally well on AlpacaEval. We set the learning rate of Phi-2 to 5e-7, and the other hyperparameters remain consistent with those in Appendix D.1.\n\n | Method      | AlpacaEval 2 LC(%) | AlpacaEval 2 WR(%） |\n|-------------|-----------------------------|-----------------------------|\n| SFT         | 5.9                         | 4.0                         |\n| DPO         | 9.0                         | 8.5                         |\n| DPO(50%)    | 8.3                         | 8.0                         |\n| DPO+SamS    | 10.3                        | 10.2                        |\n\nThe table above presents our experimental results, demonstrating that our method consistently outperforms DPO under the small model experimental setup, validating SamS’s generalization capability across models of different sizes. However, even after preference optimization, the absolute win rate remains low, primarily due to the limited parameter size of the model, which makes it challenging to achieve the strong conversational abilities observed in larger models.\n\n---\n\n> **W2: It could be nice to see more experiments + analysis for why SamS works, but I find the paper fairly convincing overall.**\n\n\nThank you for your interest in the working principles of SamS. Sams can improve the policy's performance, as long as the policy encounters varying learning difficulties and data noise—conditions that are commonly observed and discussed in the introduction. By leveraging both batch- and sample-level rewards, Sams learns these patterns and adaptively schedules samples to optimize training according to relative difficulty and filtering out the noise data.\n\nRegarding the design of the reward signal, we have supplemented more detailed ablation experiments, which you can find in the detailed analysis provided in the response to Q1. This set of ablation experiments well explains how the reward signal guides the scheduler to work.\n\n\n---\n\n> **W3: The paper could define/explain MT-Bench (GPT-4 Turbo) in the body of the paper rather than relying on the external citation.**\n\nThank you very much for your suggestions, and we sincerely apologize for any inconvenience caused by your reading. We have added an explanation for GPT-4 Turbo in the caption of Table 1. The revised caption is as follows: \n\nTable 1: AlpacaEval 2 [26] and MT-Bench [96] results under the two model settings. LC and WR denote length-controlled and raw win rate, respectively. **GPT-4 Turbo refers to the evaluation scores of MT-Bench obtained using GPT-4 Turbo as the judge model.** Here, bold denotes the best performance, underline indicates the second-best performance, and \"-\" represents that no measurement was taken.\n\n---\n\n> **Q1: It could be nice to see ablations for the different terms in the reward. What happens if we don't look at the context level reward? What if we drop the model uncertainty reward?**\n\nWe conducted detailed ablation experiments on the individual components of the multi-level reward signal and the exploration network. We followed the experimental setup in Section 5.2, using the HH dataset and Pythia 2.8B as the model.\n\n| Method                     | Test Acc（%） |\n|----------------------------|--------------|\n| SamS                       | 67.1         |\n| SamS without Preference Margin     | 66.2         |\n| SamS without Model Uncertainty     | 66.7         |\n| SamS without Sample Level Reward   | 65.7         |\n| SamS without Batch Level Reward    | 65.1         |\n| SamS without Exploration Network                   | 65.0         |\n| DPO                        | 64.0         |\n| DPO(50%)                   | 62.8         |\n\n\n\nThe experimental results are shown in the table above:\n\n- **Batch-level Reward**: When the batch-level reward is removed, the test set accuracy drops by 2.0% compared to SamS. Although all samples within a batch receive the same reward signal, the scheduler can still learn the potential benefits of samples across different batches, which is crucial for enabling exploration and exploitation across the entire sample space.\n- **Sample-level Reward**: When the sample-level reward is removed, the test set accuracy decreases by 1.4% compared to SamS. When we further ablate the preference margin and model uncertainty components of the sample-level reward, the test set accuracy drops by 0.9% and 0.4%, respectively. This indicates that computable metrics play a significant role in reward signal allocation. The preference margin provides more critical guidance in the sample-level reward signal, while the uncertainty measurement may be related to the dataset distribution and the model’s distribution shift.\n- **Exploration Network**: When the exploration network F2 is not used for error estimation, the test set accuracy decreases by 2.1% compared to SamS, highlighting the necessity of retaining this module.This demonstrates the necessity of the exploration network as a substitute for exploration-exploitation algorithms like UCB/TS in addressing the sample scheduling problem, which we modeled as a multi-armed bandit problem.\n\n---\n\n> **Q2: In the label noise experiment, does SamS tend to discard comparisons with flipped labels?**\n\nYes, Sams tends to discard samples with flipped labels. Sams is an adaptive and dynamic sample selection algorithm. When it encounters samples with flipped labels, it receives negative rewards, as such samples degrade the generalization performance of the policy. Through these interactions, Sams learns to identify such patterns and gradually avoids selecting flipped samples in subsequent training rounds. Ultimately, Sams enhances the robustness of the learned policies or LLMs.\n\n---\n\n> **Q3: Do you have more hypotheses about why SamS works? Is it also effective in other supervised classification tasks?**\n\n\nThe hypotheses we have for Sams is based on the presence of noise in the dataset and the varying learning difficulty of samples. As mentioned in the introduction, SamS can adaptively select samples that maximize the potential gains for the policy when faced with different levels of difficulty.\n\nMoreover, we can model all training data selection problems in a similar manner (Section 3). To the best of our knowledge, this is the first work to model sample selection during training as a multi-armed bandit problem. Specifically, by determining the arm context for each sample in the multi-armed bandit framework and constructing effective true rewards, our algorithmic framework can be applied to sample selection. As illustrated in Appendix F.3, we constructed the reward signal for supervised instruction fine-tuning in a similar manner. However, we have currently conducted a comprehensive exploration only on offline direct preference optimization. This is primarily because obtaining explicit human preference annotations is challenging, and compared to other types of datasets, such as those for supervised instruction fine-tuning, preference datasets exhibit more significant noise, making data selection particularly valuable."}, {"review_id": "1f9FOIrBVa", "reviewer": "Reviewer_ye7p", "summary": "In this paper, the authors propose to schedule sampling  for better DPO. Specifically, the authors formulate sample scheduling for DPO as a contextual bandit problem, and then adaptively select samples from each training batch according to the model's evolving states.", "questions": "Why is the generalization experiment in Section 5.2 conducted exclusively on Pythia-2.8B, instead of using the same model family as in Section 5.1?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. SamS introduces a new problem: \"how to dynamically schedule the preference data during DPO training?\" It is the first time to study sample scheduling for DPO.\n2. The idea of modeling sample scheduling as a contextual bandit problem is interesting. In this way, it frames the dynamic selection of training samples in DPO as a sequential decision-making process under uncertainty.\n3. The proposed algorithm SamS is complementary to current optimization algorithms and can be integrated as a plug-in module in existing DPO-related algorithms.\n\nWeaknesses\n1. Considering the cost of sample scheduling, the performance improvement in experimental results is not significant enough. In general, one of my main concerns about sample scheduling is it only improve performance under specific conditions, which may actually hinder the scaling effect when the data volume is sufficiently large. Therefore, to verify the effect of the proposed method, the authors should verify the scaling effect of the proposed method.\n2. The authors need to compare with other representative sample scheduling algorithms, rather than only comparing with the vanilla DPO. Because there are also many data filtering or selection algorithms in practice, the authors should also compare them for more convincing results.", "was_revised": true, "final_justification": "I have read the rebuttal of the authors, and I adjust my score accordingly.", "rebuttal": "Thank you very much for your valuable suggestions. In response, we have analyzed the performance of SamS during the gradual scaling of the dataset as per your recommendations, verifying that SamS can still achieve stable and even better performance improvements with large amounts of data. Additionally, we have listed and compared with works related to SamS. Finally, we extended the experiments in Section 5.2 to Llama, validating the generalization ability of SamS across different model families.\n\n---\n\n>  W1: Considering the cost of sample scheduling, the performance improvement in experimental results is not significant enough. In general, one of my main concerns about sample scheduling is it only improve performance under specific conditions, which may actually hinder the scaling effect when the data volume is sufficiently large. Therefore, to verify the effect of the proposed method, the authors should verify the scaling effect of the proposed method.\n\n\n**In terms of the overhead of sample scheduling**, although our method increases the runtime by a small margin, **it significantly reduces the average GPU memory usage**. In Section 5.4, we verified with Llama3-8B-Instruct that SamS can achieve an **18%** reduction in GPU memory usage at the cost of a **4%** increase in runtime. \nMoreover, as the policy scales, the preference optimization process slows down, and the requirements for computing resources become increasingly stringent. In this scenario, the relative runtime efficiency of SamS will become increasingly higher. This is because, apart from the feature connector (details in Appendix F.2, a two-layer MLP), the size of the scheduler adopted by SamS is fixed, and the overhead for sample scheduling and training is basically constant, achieving an increasingly smaller relative time overhead. On the other hand, since only half of the samples are used, SamS truncates the computation graphs of the discarded samples, thereby reducing memory overhead.\n**In summary, both the time efficiency and space efficiency of SamS improve as the model parameters scales**.\n\n**In terms of SamS's performance,** as a lightweight and plug-and-play method, SamS can achieve performance surpassing that of current mainstream preference optimization methods even when using only 50% of the samples (Section 5.1). Compared with DPO which uses the same training samples, SamS yields an improvement of **5.5% - 8.4%** in the LC win rate on AlpacaEval 2, an improvement of **3.0% - 12.4%** in the win rate on AlpacaEval 2, and an improvement of **0.2 - 0.4** in the MT-Bench score.\n\n**SamS can improve performance under general conditions**, as long as the model encounters varying learning difficulties and data noise—conditions that are commonly observed and discussed in the introduction. By leveraging both batch- and sample-level rewards, SamS learns these patterns and adaptively schedules samples to optimize training. The reward signals are designed to be directly accessible during policy training, without incurring additional computational overhead such as extra forward/backward passes through the LLM or additional dataset scans.\n\n**In cases where the data volume is sufficiently large,** as we mentioned and verified in the introduction, existing preference datasets are unordered and contain noise. When the dataset is scaled up, the same issues persist, and **it amplifies the need for adaptive sample scheduling**, which becomes even more critical in such settings.\nTo verify that our method has the ability to scale with increasing data volume, we conducted the following experiment. We adopted the experimental setup from Section 5.2. Within multiple epochs, we fixed the random seed and then gradually increased the number of training samples. \n\n| Training Samples | Test Acc of DPO+SamS(%) | Test Acc of DPO(%)| Difference(%)|\n| -------- | ----------- | ---------- |  ----------    |\n| 200k     | 64.5        | 64.4       |   +0.1        |\n| 400k     | 64.6        | 63.1       |   +1.5        |\n| 600k     | 65.5        | 63.8       |   +1.7        |\n| 800k     | 66.1        | 64.4       |   +1.7        |\n| 1000k    | 66.8        | 64.9       |   +1.9        |\n\nAs shown in the table above, the performance gap between DPO+SamS and DPO is gradually widening. In other words, our scheduler continues to learn during the training process, and its capability becomes increasingly stronger as the number of training samples increases. This ensures that our method has the ability to scale with larger data volumes.\n\n---\n\n> W2：The authors need to compare with other representative sample scheduling algorithms, rather than only comparing with the vanilla DPO. Because there are also many data filtering or selection algorithms in practice, the authors should also compare them for more convincing results.\n\nThanks for the reviewer's insightful suggestions. **We in fact included the related work discussion in Appendix C and the comparison with a representative data selection algorithm in Appendix D.4 in the manuscript (Supplementary Material)**. For ease of reference, we re-present the discussion and results below.\n\n**Related work**.  \"Data Selection for DPO\" is directly relevant to our problem setting. Specifically, [1] conducts a fine-grained analysis of preference data and proposes evaluation metrics. Similarly, [2,3] presents sample-quality evaluation approaches based on different observations, subsequently selecting data subsets for policy training. Although these methodologies train policies on selected subsets, they isolate the sample selection from the model’s training process, thereby disregarding the dynamic interaction between selected samples and the evolving state of the model. This category essentially focuses on data preprocessing.\n\n**Experiments**. We regret to note that among these works, only Selective DPO [3] has been open-sourced. We conducted experiments using the first LLaMA setting in comparison with Selective DPO, evaluating both Selective DPO and Selective DPO+SamS. \n\n| Method               | AlpacaEval 2 |   AlpacaEval 2        | MT-Bench       | Runtime   |\n|----------------------|--------------|----------|----------------|-----------|\n|                      | LC (%)       | WR (%)   | GPT-4 Turbo    |           |\n| SFT                  | 26.0         | 25.3     | 6.9            | -         |\n| DPO          | 40.3         | 37.9     | 7.0            | 2.3 h     |\n| DPO+SamS             | 42.2         | 40.5     | 7.1            | **2.4 h**     |\n| Selective DPO   | 41.7         | 40.9     | 7.0            | **6.0+1.2 h** |\n| Selective DPO+SamS   | 46.5         | 44.0     | 7.2            | 6.0+1.3 h |\n\nAs is shown in the table above, Selective DPO entails a total computation time of 6.0 hours, including 5.1 hours for training reference models and 1.2 hours for DPO training. In contrast, our method requires approximately 2.4 hours in total, closely aligning with the time cost of standard DPO while reducing GPU usage by 18%. This demonstrates that **SamS achieves comparable performance while being far more time-efficient than Selective DPO**. \n\nFurthermore, we found that when combined with Selective DPO, using only 25% of the sample size of the original DPO yields significantly better performance. **Selective DPO+SamS** achieves a 46.5% AlpacaEval 2 LC win rate, a 44.0% AlpacaEval 2 win rate, and a MT-Bench score of 7.2, **representing improvements of 6.2%, 6.1%, and 0.2 respectively over DPO**. This strongly validates the research value of the sample scheduling problem we defined, as well as the enormous application potential of our adaptive sample scheduling strategy when integrated with data pre-selection methods.\n\n---\n\n> Q1:Why is the generalization experiment in Section 5.2 conducted exclusively on Pythia-2.8B, instead of using the same model family as in Section 5.1?\n\nThank you very much for your suggestions. Since we followed the experimental setup of DPO, the model family used in our experiments is also Pythia, consistent with that in DPO. In Section 5.1, we used models from other families mainly to verify the performance of our method under currently widely adopted experimental setups in a fair manner. \nAs suggested by the reviewer,, we replace the base model in Section 5.2 with **Llama-3-8b (base)** to conduct the following generalization performance experiments.\n\n| Dataset | Method       | Test-Acc(%) |\n| -------- | ------------ | ----------- |\n| HH       | DPO          | 67.1%       |\n| HH        | DPO+SamS     | 69.5%       |\n| HH       | Improvement  | 2.4%        |\n| SHP      | DPO          | 76.9%       |\n| SHP        | DPO+SamS     | 81.6%       |\n| SHP         | Improvement  | 4.7%        |\n\nThe experimental results are shown in the table above. Under both the HH and SHP preference datasets, significant performance improvements were achieved after deploying SamS on DPO. This indicates that SamS can effectively generalize to models from other families.\n\n---\n\n[1]Judy Hanwen Shen, Archit Sharma, and Jun Qin. Towards data-centric rlhf: Simple metrics for preference dataset comparison. arXiv preprint arXiv:2409.09603, 2024.\n\n[2]Xun Deng, Han Zhong, Rui Ai, Fuli Feng, Zheng Wang, and Xiangnan He. Less is more: Improving llm alignment via preference data selection. arXiv preprint arXiv:2502.14560, 2025.\n\n[3]Chengqian Gao, Haonan Li, Liu Liu, Zeke Xie, Peilin Zhao, and Zhiqiang Xu. Principled data selection for alignment: The hidden risks of difficult examples. arXiv preprint arXiv:2502.09650, 2025."}, {"review_id": "70owe7gLss", "reviewer": "Reviewer_5QuT", "summary": "The authors introduce a new problem: sample scheduling in the preference alignment setting and formulate it using the contextual bandit framework. To address this, the authors propose SamS, a scheduling algorithm that adaptively selects training samples from each batch according to the model’s evolving internal states. SamS-augmented preference alignment methods consistently outperform their vanilla counterparts across diverse benchmarks and LLM backbones.", "questions": "**Q1:** In Equation (7), how does $g$, a min-max normalization function, map its argument to $[0, 1]$, given that the argument is a scalar? Is the normalization done per batch?\n\n**Q2:** In reference to Appendix D.1, could the authors clarify how the ground-truth batch-level and sample-level rewards are computed during offline scheduler pretraining?\n\n**Q3:** Could the authors comment on the sensitivity of SamS to the hyperparameter $\\lambda$, the tunable hyperparameter controlling the exploration strength?\n\n**Q4:** Do the authors observe any notable differences in reward margins between models trained using baseline preference optimization methods and those augmented using SamS during evaluation?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Pros:**\n1. The paper proposes a novel and underexplored problem: sample scheduling in the context of preference alignment fine-tuning, and formulates it in a contextual bandit framework.\n2. The authors propose a novel exploration-exploitation scheduler based on the LLM’s learning feedback to maximize the potential generalization performance.\n3. Extensive empirical results show consistent performance gains of SamS across multiple preference-alignment benchmarks.\n\n**Cons:**\n1. The batch-level reward in Equation (6) is computed by using the batch-average DPO loss as the estimation for the expected loss across the full distribution. This is a stochastic and potentially biased approximation, and requires further justification or empirical validation.\n2. There is a lack of explanation for the initialization of the parameter $\\theta\\_{1}$ in Algorithm 1.\n3. In Appendix D.1, the authors mentioned that they use a frozen pretrained encoder during the preference optimization process; however, this introduces a distributional shift between the context used by the scheduler and the actual learning dynamics of the fine-tuned model, which is not analyzed.\n\n**Minor Typos:**\n\nL102: \"Given a subset $\\tilde{X\\_{t}}\\in X\\_{t}$ with size K, $|\\tilde{X\\_{t}}|=$ **k**.\"", "was_revised": true, "final_justification": "The authors' rebuttal has addressed my concerns. Given the current scarcity of sample scheduling methods in preference learning, I find SamS to be a valuable contribution. Therefore, I maintain my positive rating.", "rebuttal": "Thank you for your valuable feedback. We anwsered each of your detailed concerns below, reporting the model’s performance under different exploration network weights and conducting a comparative analysis of the changes in reward margin gaps after deploying SamS. We hope this resolves your concerns, and we are happy to respond any further questions you may have.\n\n---\n\n> **W1: The batch-level reward in Equation (6) is computed by using the batch-average DPO loss as the estimation for the expected loss across the full distribution. This is a stochastic and potentially biased approximation, and requires further justification or empirical validation.**\n\nThis design is formed by the consideration of trade-off between effectivness and computational cost. Although applying the standard Monte Carlo estimation will induce some bias across distribution, it boost the practicality of Sams without any additional cost to obtain the reward, avoding the exhaustive scan of full dataset. Such essence is analogous in many well known algorithms such as mini-batch SGD and DQN in RL. \nNevertheless, as the number of training epochs accumulates, such sampling-based estimation will gradually converge to the true expected value, and the bias will accordingly decrease. \n\n\nTo verify the effectiveness of batch-level rewards, we have supplemented relevant ablation experiments. We followed the experimental setup in Section 5.2, using the HH dataset and Pythia 2.8B as the model.\n\n| Method                     | Test Acc（%） |\n|----------------------------|--------------|\n| SamS                       | 67.1         |\n| SamS without Batch Level Reward    | 65.1         |\n\nThe experimental results are shown in the table above. When the batch-level reward is removed, the test set accuracy drops by 2.0% compared to SamS. Although all samples within a batch receive the same reward signal, the scheduler can still learn the potential benefits of samples across different batches, which is crucial for enabling exploration and exploitation across the entire sample space. In general, this verifies the rationality of constructing batch-level rewards using the Monte Carlo estimation method based on batch-averaged DPO loss.\n\n\n---\n\n> **W2: There is a lack of explanation for the initialization of the parameter $θ_1$ in Algorithm 1.**\n\n$θ_1$ refers to the parameters of the scheduler at initialization, where the subscript \"1\" represents the training round. Specifically, since the batch-level reward cannot be calculated when the training round is 0, the scheduler starts training from the second round. Therefore, we mark the initial subscript of the scheduler as 1.\n\n---\n\n> **W3: In Appendix D.1, the authors mentioned that they use a frozen pretrained encoder during the preference optimization process; however, this introduces a distributional shift between the context used by the scheduler and the actual learning dynamics of the fine-tuned model, which is not analyzed.**\n\nThis is very insighful point, and we have considered this issue as well. To address the semantic gap between the text encoder and the policy, we employed a feature connector (Appendix F.2). Prior to preference optimization, we introduced pre-training of the scheduler during the policy’s SFT phase (Appendix F.3), where we opened the full weights of the scheduler, including the text encoder and the feature connector. During the preference optimization phase, although the text encoder is frozen, the connector remains trainable, allowing it to dynamically adapt to the distribution shifts of the policy during training, ensuring further semantic alignment between the pre-trained encoder and the policy. (This approach has been well-established in multimodal domains [1], where a two-layer fully connected layer is used to bridge the semantic gap between visual and text feature spaces, with the visual encoder kept frozen during visual instruction fine-tuning.)\n\n---\n\n> **Q1:  In Equation (7), how does g, a min-max normalization function, map its argument to [0,1], given that the argument is a scalar? Is the normalization done per batch?**\n\nYes, your understanding is entirely correct. We transformed the scalar preference margin term into a relative value through min-max normalization within each batch.\n\n---\n\n> **Q2：Could the authors clarify how the ground-truth batch-level and sample-level rewards are computed during offline scheduler pretraining?**\n\nIn Appendix F.3, we provide a detailed description of the scheduler pre-training process, including the construction of the reward signal and the computation of the loss. Specifically, for the batch-level reward, we utilize the batch-level loss gap to measure the potential benefit the policy derives from the training samples, i.e., directly replacing the DPO loss with the SFT loss. For the sample-level reward, we directly use the cross-entropy loss of the SFT loss as the preference margin, meaning the reward margin in the DPO loss is replaced with the negative of the SFT loss, while the model uncertainty term remains unchanged. During preference optimization, we aim for the scheduler to prioritize samples with larger reward margins; thus, we assign higher ground-truth reward values to response-sample pairs with high reward margins. In contrast, during the SFT process, we aim to minimize the gap with the expected response. A smaller SFT loss indicates closer alignment with the expected response, and we want the scheduler to learn to select in-distribution data. Therefore, we assign larger reward values to samples with smaller SFT loss. (We sincerely apologize for omitting the negative sign before $L_{SFT}$ in Equation 14; this will be corrected in subsequent versions.)\n\n---\n\n> **Q3: Could the authors comment on the sensitivity of SamS to the hyperparameter λ, the tunable hyperparameter controlling the exploration strength?**\n\nWe conducted a parameter sensitivity analysis experiment for the exploration network weight λ, as reported. Following the experimental setup in Section 5.2, we used Pythia2.8B as the model and the HH dataset to evaluate SamS performance under different λ values. Specifically, we set the maximum F2 weight to 0.5, as the exploration network fundamentally relies on the error estimation of the network, and its weight should not exceed that of the exploitation network.\n\n| λ    | Test Acc（%） |\n|------|--------------|\n| 0    | 65.0%        |\n| 0.05 | 66.1%        |\n| 0.1  | 67.1%        |\n| 0.2  | 66.4%        |\n| 0.5  | 63.4%        |\n\nThe experimental results are presented in the table above. We observed that an exploration ratio of 0.1 is the most suitable, with performance degradation occurring when the exploration ratio is either higher or lower. Specifically, when the exploration ratio reaches 0.5, the accuracy on the test set decreases by 3.7%, indicating that a higher exploration ratio is detrimental to the scheduler’s ability to leverage existing experience for sample scheduling. Conversely, when the exploration ratio is 0.05, a performance drop of 1.0% is observed, suggesting that a lower exploration ratio similarly hinders the scheduler’s ability to select samples with greater potential benefits during continuous preference optimization.\n\n---\n\n> **Q4: Do the authors observe any notable differences in reward margins between models trained using baseline preference optimization methods and those augmented using SamS during evaluation?**\n\nUnder the experimental setup in Section 5.2, we compared the average reward margin on the test set during training for DPO and DPO+SamS. Specifically, we used Pythia2.8B as the model and the HH dataset, calculating the average reward margin on the test set every 100k steps.\n\n| Training Steps | Reward Margin of DPO | Reward Margin of DPO+SamS | Difference |\n| -------------- | -------------------- | ------------------------- | ---------- |\n| 100k           | 0.27                 | 0.31                      | +0.04       |\n| 200k           | 0.42                 | 0.47                      | +0.05       |\n| 300k           | 0.44                 | 0.51                      | +0.07       |\n| 400k           | 0.52                 | 0.61                      | +0.09       |\n| 500k           | 0.61                 | 0.78                      | +0.17       |\n\nAs shown in the table above, we observed that DPO+SamS consistently achieves significantly higher reward margins than DPO across different training steps, with this difference becoming more pronounced as training progresses. On one hand, this is due to our sample-level reward, which assigns higher rewards to samples with larger reward margins, encouraging the scheduler to prioritize such samples. On the other hand, through iterative training between the scheduler and the policy, the scheduler’s capability improves, leading to a continuous increase in the quality of selected samples. In contrast, DPO selects samples entirely randomly, resulting in a growing disparity in sample reward margins between the two approaches. In summary, our method consistently and significantly enhances the reward margin during training, validating the effectiveness of SamS.\n\n---\n\n> **Minor Typos:**\n\nThank you for pointing out this typo. In fact, what we intend to express is that the size of $X_t$  is $|X_t|=K$, and the size of its subset $\\hat{X_t}$ is $|\\hat{X_t}|=k$. The expression in L102 is indeed prone to misunderstanding, and we will revise it to this in subsequent versions: \"Given a subset $\\hat{X_t} \\subset X_t$, where $|X_t|=K$, $|\\hat{X_t}|=k$\". We are very sorry for the impact on your reading experience.\n\n\n\n\n---\n\n[1] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Ad1081 vances in neural information processing systems, 36:34892–34916, 2023."}, {"review_id": "KE0MHf3sKN", "reviewer": "Reviewer_Cv1P", "summary": "This paper introduces SamS (Sample Scheduling for Direct Preference Optimization)​​, a method to dynamically select training samples during DPO based on the evolving states of the language model.\nSamS formulates sample scheduling as a contextual bandit problem, with the reward defined by combining batch-level DPO loss improvement and sample level margin & uncertainty.\nThe SamS model maps LM's hidden states with two output heads, which regressively fit the reward (exploitation network) and reward difference (exploration network) respectively.\nEmpirically, SamS improves performance on benchmarks like AlpacaEval 2 and enhances robustness to noisy data,", "questions": "1. Can the authors compare with more improved baselines (such as SimPO) for both the main experiment (Table 1) and generalization experiment for different learning methods (Table 2)?\n2. Can the authors provide ablation study regarding different components of the reward and network design?", "limitations": "The authors have discussed the limitations in their manuscript.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n1. Novelty: Introduces a fresh perspective on DPO by focusing on online dynamic sample scheduling.\n2. Practicality​​: The contextual bandit setup's reward can be computed straightforwardly during the DPO process, and the lightweight scheduler model also reduces computational overhead. Empirically, the sampled 50% preference data achieves better performance than the full data, further validating the practicality.\n\n### Weaknesses\n1. Limited Baselines: Most of the compared learning objectives are generally weaker than DPO in performance, such as IPO, KTO. It remains unclear whether the proposed methods can work with other improved learning objectives or online/iterative DPO variants.\n2. Insufficient Ablations: Lack of ablation study on different components of SamS, including the batch/sample-level reward and the exploration/exploitation network design.", "was_revised": true, "final_justification": "The rebuttal has effectively addressed my key concerns, leading me to maintain a positive stance. The authors have successfully incorporated SimPO as a baseline in both main and generalization experiments, significantly strengthening the validity of their results. The additional ablation studies on reward components and network design provide valuable insights into the inner workings of SamS. While the authors have started adapting SamS to GRPO and iterative DPO, the results of these adaptations are not yet presented. However, this does not overshadow the positive contributions of the paper at this stage. Overall, the authors' efforts in addressing the reviewers' comments have improved the quality of the paper, justifying a positive score.", "rebuttal": "Thank you for your insightful feedback. In response, we have incorporated SimPO as a baseline in both the main and generalization experiments, as per your suggestion, and conducted a more detailed ablation study, including the individual components of the scheduler’s reward and the exploration network. We hope this addresses your concerns, and we are happy to answer any further questions you may have.\n\n---\n\n> **W1&Q1: Can the authors compare with more improved baselines (such as SimPO) for both the main experiment (Table 1) and generalization experiment for different learning methods (Table 2)? It remains unclear whether the proposed methods can work with other improved learning objectives or online/iterative DPO variants.**\n\n\nFollowing your suggestion, we included SimPO as a baseline in Section 5.1 and evaluated SamS's generalization ability on SimPO in Section 5.2.\nFor the generalization experiments, we included SamS+SimPO and compared it with SimPO. Specifically, we replaced the DPO loss in the batch-level reward with the SimPO loss, while keeping the rest of the method unchanged. For dataset and model selection, we followed the experimental setup in Section 5.2. Regarding hyperparameters, we adopted the settings recommended in the SimPO paper, namely $β=2.0$ and $γ=1.0$.\n\n| Dataset | Method       | Test-Acc(%) |\n| -------- | ------------ | ----------- |\n| HH       | SimPO        | 57.4%       |\n| HH       | SimPO+SamS   | 61.4%       |\n| HH       | Improvement  | 4.0%        |\n| SHP      | SimPO        | 56.4%       |\n| SHP      | SimPO+SamS   | 61.1%       |\n| SHP      | Improvement  | 4.7%        |\n\nThe experimental results are shown in the table above. After deploying SamS on SimPO, we observed accuracy improvements of 4.0% and 4.7% on the HH and SHP datasets, respectively. These improvements are more significant compared to DPO (2.8%/2.4%) and KTO (3.1%/2.3%), demonstrating that SamS can achieve substantial performance gains even on a stronger baseline like SimPO.\nAdditionally, we included SimPO as a baseline in Section 5.1.\n\n| Method   | Mistral-instruct (7B)          |       Mistral-instruct (7B)           |    Mistral-instruct (7B)              | Llama3-Instruct (8B)           |       Llama3-Instruct (8B)            |               Llama3-Instruct (8B)    |\n|----------|--------------------------------|------------------|------------------|--------------------------------|------------------|------------------|\n|         | AlpacaEval 2                   |  AlpacaEval 2              |        MT-Bench     | AlpacaEval 2                   | AlpacaEval 2             |        MT-Bench           |\n|          | LC                             | WR               | GPT4 Turbo       | LC                             | WR               | GPT4 Turbo       |\n| SimPO    | 32.1                           | 34.8             | 6.6              | 41.9                           | 38.0             | 7.0              |\n| DPO+SamS | 33.6                           | 36.2             | 6.7              | 42.2                           | 40.5             | 7.1              |\n\n\n| Method   | Llama3-Instruct v0.2 (8B)      |     Llama3-Instruct v0.2 (8B)              |        Llama3-Instruct v0.2 (8B)           | Gemma2-Instruct v0.2 (9B)     |      Gemma2-Instruct v0.2 (9B)            |     Gemma2-Instruct v0.2 (9B)             |\n|----------|--------------------------------|------------------|------------------|--------------------------------|------------------|------------------|\n|          | AlpacaEval 2                   |      AlpacaEval 2        |    MT-Bench              | AlpacaEval 2                   | AlpacaEval 2     |        MT-Bench           |\n|          | LC                             | WR               | GPT4 Turbo       | LC                             | WR               | GPT4 Turbo       |\n| SimPO    | 53.7                           | 47.5             | 7.0              | 72.4                           | 65.0             | -                |\n| DPO+SamS | 51.5                           | 48.2             | 7.0              | 70.8                           | 67.4             | -                |\n\nThe experimental results are presented in the table above. It can be observed that our method achieved higher AlpacaEval 2 win rates and MT-Bench scores across four different model settings. DPO+SamS underperformed SimPO in length-controlled win rates on Llama3-Instruct v0.2(8B) and Gemma2-Instruct v0.2(9B), likely due to SimPO’s loss incorporating regularization for sequence length, which makes it particularly strong in length-controlled win rates. However, it is noteworthy that our method achieved superior performance to SimPO in most cases while using only half the number of samples for training.\n\nAdditionally, Regarding whether SamS is effective on other preference optimization objectives, the answer is affirmative. We are currently adapting SamS to GRPO and iterative DPO, and have achieved promising results. We invite you to follow our subsequent work for further details.\n\n---\n\n> **W2&Q2: Insufficient Ablations: Lack of ablation study on different components of SamS, including the batch/sample-level reward and the exploration/exploitation network design.**\n\nWe conducted detailed ablation experiments on the individual components of the multi-level reward signal and the exploration network. We followed the experimental setup in Section 5.2, using the HH dataset and Pythia 2.8B as the model.\n\n| Method                     | Test Acc（%） |\n|----------------------------|--------------|\n| SamS                       | 67.1         |\n| SamS without Preference Margin     | 66.2         |\n| SamS without Model Uncertainty     | 66.7         |\n| SamS without Sample Level Reward   | 65.7         |\n| SamS without Batch Level Reward    | 65.1         |\n| SamS without Exploration Network                   | 65.0         |\n| DPO                        | 64.0         |\n| DPO(50%)                   | 62.8         |\n\n\n\nThe experimental results are shown in the table above:\n\n- **Batch-level Reward**: When the batch-level reward is removed, the test set accuracy drops by 2.0% compared to SamS. Although all samples within a batch receive the same reward signal, the scheduler can still learn the potential benefits of samples across different batches, which is crucial for enabling exploration and exploitation across the entire sample space.\n- **Sample-level Reward**: When the sample-level reward is removed, the test set accuracy decreases by 1.4% compared to SamS. When we further ablate the preference margin and model uncertainty components of the sample-level reward, the test set accuracy drops by 0.9% and 0.4%, respectively. This indicates that computable metrics play a significant role in reward signal allocation. The preference margin provides more critical guidance in the sample-level reward signal, while the uncertainty measurement may be related to the dataset distribution and the model’s distribution shift.\n- **Exploration Network**: When the exploration network F2 is not used for error estimation, the test set accuracy decreases by 2.1% compared to SamS, highlighting the necessity of retaining this module.This demonstrates the necessity of the exploration network as a substitute for exploration-exploitation algorithms like UCB/TS in addressing the sample scheduling problem, which we modeled as a multi-armed bandit problem."}]

Author rebuttal:

Meta-review: The paper introduces the problem of Sample Scheduling for DPO (SamS), focusing on dynamically selecting training samples during DPO based on the evolving state of the language model. SamS formulates the sample scheduling problem as a contextual bandit problem, allowing for efficient and adaptive sample selection. The reward function combines batch-level DPO loss improvement and sample-level margin & uncertainty, enabling the scheduler to learn and prioritize valuable samples. The proposed algorithm demonstrates consistent performance gains across various benchmarks and large language model backbones.

During the discussion phase, the reviewers raised a few concerns regarding the significance of performance improvements, comparisons against other methods, and abalation studies. The authors have effectively addressed the reviewers’ concerns, providing additional experiments and clarifications that strengthen the paper’s claims.  

In the end, all reviewers agreed that the technical contributions in this work are significant, and the proposed method demonstrates clear benefits in terms of performance and robustness. Therefore, we recommend acceptance of this submission.
Decision: Accept (poster)