Is On-Policy Data always the Best Choice for Direct Preference Optimization-Based LM Alignment?
Authors: Zetian Sun
Dongfang Li
Xuhui Chen
Baotian Hu
Min Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: DPO
Preference Candidates
On-policy Sampling
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: The alignment of language models (LMs) with human preferences is critical for building reliable AI systems. The problem is typically framed as optimizing an LM policy to maximize the expected reward that reflects human preferences. Recently, Direct Preference Optimization (DPO) was proposed as an LM alignment method that directly optimizes the policy from static preference data, and further improved by incorporating on-policy sampling (i.e., preference candidates generated during the training loop) for better LM alignment. However, we show on-policy data is not always optimal, with systematic effectiveness difference emerging between static and on-policy preference candidates. For example, on-policy data can result in a  $3\times$ effectiveness compared with static data for Llama-3, and a $0.4\times$ effectiveness for Zephyr. To explain the phenomenon, we propose the alignment stage assumption, which divides the alignment process into two distinct stages: the preference injection stage, which benefits from diverse data, and the preference fine-tuning stage, which favors high-quality data. Through theoretical and empirical analysis, we characterize these stages and propose an effective algorithm to identify the boundaries between them. We perform experiments on $5$ models (Llama, Zephyr, Phi-2, Qwen, Pythia) and 
$2$ alignment methods (DPO, SLiC-HF) to show the generalizability of alignment stage assumption and boundary measurement.

Published as a conference paper at ICLR 2026
IS ON-POLICY DATA ALWAYS THE BEST CHOICE FOR DIRECT PREFERENCE OPTIMIZATION-BASER LM ALIGNMENT?
Zetian Sun, Dongfang Li, Xuhui Chen, Baotian Hu; Min Zhang Harbin Institute of Technology, Shenzhen zet zetiansun. cs@ gonim.mal.ccom com
(1 fldongfang,, hubaotian, zhangmin2021)ehh edu.
ABSTRACT
The alignment of language models (LMs) with human preferences is critical for building reliable Al systems. The problem typically framed optimizing an LM policy to maximize the expected reward that reflects human preferences. Re- cently, Direct Preference Optimization (DPO) was proposed as an LM alignment method that directly optimizes the policy from static preference data, and further improved by incorporating on-policy sampling (i.e., preference candidates gener- ated during the training loop) for better LM alignment. However, we show on- policy data is not always optimal, with systematic effectiveness difference emerg- ing between static and on-policy preference candidates. For example, on-policy data can result in a 3x effectiveness compared with static data for Llama-3, and 0.4x effectiveness for Zephyr. To explain the phenomenon, alignment stage assumption, which divides the alignment process into two dis- tinct stages: the preference injection stage, which benefits from diverse data, and the preference fine-tuning stage, which favors high-quality data. Through theoret- ical and empirical analysis, we characterize these stages and propose an effective algorithm to identify the boundaries between them. We perform experiments on models (Llama, Zephyr, Phi-2, Qwen, Pythia) and alignment methods (DPO, SLiC-HF) to show the generalizability of alignment stage assumption and the ef- fectiveness the boundary measurement algorithm.
propose the
Alignnent Area
Alignnent Area
[--- o |--->
RC
(a) Preference injection stage.
(b) Preference fine-tuning stage.
Figure 1: Illustration of alignment stage assumption and different characteristics of (a) prefer- ence injection stage and (b) preference fine-tuning stage. The alignment area indicates the preferred region of preference candidates corresponding alignment stages. The stage boundary is estimated by the distance between ground truth text distribution (TG) and simulated text distribution (Ta, Te.).
INTRODUCTION
Large language models possess broad world knowledge and strong generalization capabilities in complex tasks under minimal supervision (Brown et al. 2020). However, the powerful models still produce biased (Bender al., 2021), unfaithful (j et al. 2023) and harmful (Bai et al. 2022) re- sponses due to the heterogeneous sources of their pre-tinin corpora. It is important to ensure models to generate desired responses that conform humans' ethical standards and quality pref- erences for building reliable AI systems, which is well known as language model (LM) alignment
"Corresponding Author

---

Published as a conference paper at ICLR 2026
with human preferences (Ouyang al., 2022). Generally, the LM alignment problem is formulated as optimizing a policy model To to maximize the expected reward ras where the reward Top reflects human preference regarding the completion y for given prompt 2.
The most widely adopted approach to address the LM alignment problem is through reinforcement learning (RL) in on-policy manner (Ziegler al.) 2019 Stiennon et al. 2020 Quyang et al.) 2022]. Specifically, the on-policy manner requires Ty iteratively refines its policy by performing on- policy sampling (i.e., sampling completions generated under its current parameters), ensuring that gradient estimates align with the latest behavior policy. The LM policy is then optimized via RL solutions. However, these approaches incur significant computational cost due to repeated sampling from the LM policy, and are observed to be unstable due to the high variance in estimating the policy gradients or value functions, which potentially worsens sample complexity and thus compromises efficient model convergence (Papini et al. 2018 Anschel et al. 2017).
Direct Preference Optimization (DPO, Rafailov et al. (2023)) was proposed to be a competitive alternative to the RL solutions. Specifically, DPO optimizes TTe via reward modeling loss on prefer- ence candidates following the off-policy manner, i.e., the LM policy is optimized on a static dataset without additional sampling during the training loop. It is more resource-efficien,, and shares the theoretically equivalent optimization objective with those RL, solutions. Despite all the advantages, as off- off-policy method, DPO can struggle in out-of-distributing performance due to the absence of on-policy exploration (Tang et al. 2024
scenarios and result in sub-optimal
To tackle these issues, recent works proposed iterative DPO, method integrating on-policy sam- pling into regular DPO training, as observed to outperform vanilla DPO in several benchmarks (Wu al., 2024| Zhang et al., 2025a Rosset al., 2024). These findings highlight the potential of on- policy sampling for enhancing LM alignment via off-policy methods like DPO. However, the practi- cal recipe of using on- polpoiicy data lacks discussion or clear guidelines. Several works choose train the LM policy on on-policy data directly (Yuan et al. 2024], Liu et al. 2024), while other works choose train models on off-policy preference candidates first as a cold start phase (Zhang et al.) 2025a Kim et al. (025). Such discrepancy and arbitrariness indicate an absence of comprehensive understanding about the relationship between LM alignment and preference candidates, which may limit the model performance and sample efficiency. This motivates us to study the following research question: What is the requirement of preference candidates during the LM alignment process? In this work, we answer the research question from two aspects, i.e, the qualitative description of the LM alignment process (RQ1) and the actionable insight of the description of the LM alignment process (RQ2). Through detailed experiments, we reveal a patterned dynamic requirements of pref- erence candidates during the alignment process, and further provide an alignment stage assumption to explain the phenomenon from the perspective of DPO. Based on the assumption, we answer RQs through massive empirical experiments and mncoreticalggronede method.
Firstly, we conduct a two-iteration training experiment on Llama-3, Zephyr and Phi-2. The ex- perimental results reveal the existence of a patterned effectiveness discrepancy between the use of on-policy preference candidates (PCon) and off off-policy preference candidates (PCog), and models exhibit varying performances and dynamic requirements for preference candidates. Motivated by this observation, we propose the alignment stage assumption, which posits that the alignment pro- cess can be divided into two stages, i.e., the preference injection stage and the preference fine-tuning stage, as illustrated in Figure [ Based on the alignment stage assumption, we answer the research questions subsequently. Specifically, we conduct extensive experiments to demonstrate the char- acteristics of each alignment stage (for RQ1). We find that models in preference injection stage favor data of high preference diversity, while those in preference fine-tuning stage favor data of high preference quality. We propose the boundary measurement algorithm, a measurement to determine which stage the policy is currently in, and perform extensive experiments show the effectiveness of our algorithm (for RQ2). Moreover, we provide a theoretical perspective to interpret the stage characteristics and the boundary measurement algorithm. Notably, we show that the requirements of preference diversity stems from more accurate approximation of the ground-truth preference given the Bradley-Terry definition. The goal of selecting preference candidates is better estimate the general text distribution, which is based on human preferences or the ground-trutt reward model used for preference annotation. We also show that our boundary measurement algorithm identifies a better estimation of the general text distribution. Finally, we conduct experiments more mod- els (Qwen 2.5, Pythia) and more methods (SLiC-HF) to show the generalizability of our conclusions. To provide a clear image, we illustrate the assumption and its subsequent conclusions in Figure
2

---

Published as a conference paper at ICLR 2026
Ry Current Policy
p ,
TG Boundary Measurement
P": GT Preference
Cround-truth Distribution
Preference Consistency
Case Preference Injection stage
Allgrment
Diversity
Off- Off-policy
matlers
dala
Aligument Process
Aligrment area -- to-
Quality
On-policy
1
Case 2: Preference Fine Fineetuning slage
matters
data
Estimated Text Distribution (Preference Distribution)
LM Alignment
Figure 2: Illustration of the alignment stage assumption. The alignment process is continuous transition from preference injection stage preference fine-tuning stage. We demonstrate the char- acteristics of stages (Case and Case 2). We build the relationship among preference distribution, reward model and text distribution, which help us understand the alignment process from the per- spective of distribution distance and preference consistency. Practically, we propose the boundary measurement, a measurement to decide which stage the policy currently in by judging which dis- tribution (Toff and TT6) is better estimation of the ground-truti truth distribution (TG)).
We summarize our contributions this paper:
We reveal patterned effectiveness discrepancy between on on-policy data and off- off-policy for different models, and propose the aligmment stage assumption (preference injection stage, preference fine-tuning stage) to model the dynamic requirements for preference candidates.
We analyze the alignment stages through empirical analysis on two characteristics (i.e., diversity and quality). showing that models in preference injection stage favor data with high diversity, while models preference fine-tuning stage favor data with high quality.
We provide theoretical insights into the underlying mechanism about the LM alignment process, and propose the boundary measurement algorithm to decide stage boundaries.
2 RELATED WORK
Iterative DPO. Based on vanilla DPO, iterative DPO aims at improving DPO by incorporating on-policy sampling data. Yuan et al. (2024) constructs the preference dataset automatically where both preference candidates and instruction prompts generated by LM in an on-policy manner. Tajwar et al. (2024) further discusses the requirements fine-tuning with preference data through extensive experiments and detailed theoretical analysis, showing that approaches that on-policy sampling are generally more preferred practice. These works provide theoretical analysis about on-policy sampling. Our work builds this line by describing the overall alignment process from a systematic and methodological perspective and improving the efficiency and effectiveness of on- policy sampling for model training, rather than selecting preference data manually and empirically, which is neither scalable nor optimal for LM alignment.
Data Diversity. The diversity of preference data be separated into sections: preference diversity and candidate diversity, both facts can help improve LM alignment. The former is due to the complexity of values, environments or populations, which result in the mismatch and diversity of preferences among different annotators. Several works model the diverse preference alignment problem as multi-object optimization problem, addressing the problem using methods like Pareto optimality (Guo et al. 2024 Zhou et al. (2024) or reward ensembling (Lou et al., 2024; Zeng al.) 2024 Ramé et al. 2024). Our work focuses on the latter one, the candidate diversity. It is due to the limited coverage of the general text space given the condition of finite sampling, which results in an insufficient and incomplete preference representation. By labeling preferences using the same reward model, our work introduces the crucial role of candidate diversity at the preference injection stage. can help models construct the general reward distribution effectively that is aligned with the reward model, and thus achieve more valuable explorations the preference fine-tuning stage.
3

---

Published as a conference paper at ICLR 2026
3 PRELIMINARIES
In this section, we first formally review the concept and objective of the language model alignment problem. Then we review existing approaches that are applied to address the alignment problem via reinforcement learning and direct preference optimization.
3.1 LM ALIGNMENT WITH HUMAN PREFERENCES
Given
vocabulary V, a language model defines probability distribution x(x) I[= T(2 "",, It- over a sequence of tokens x (I), ...; Tn.. We apply и to text gener- ation task with input space X' = yMt and output space y v modeled by (v) = ক(,, v) (x).
preference dataset Drref consists of pairs of responses as the preference candidates, and their corresponding preferences pre-annotated by humans (Dubey et al.) 2024) strong LMs through prompting-bai based techniques (Dubois et al. 2024a). Then, a reward model Tá x y + R is learned on Deref and trained by minimizing the pair-wise preference loss by its general form:
((ra)= B
( yw) (((r))))
(=(,,,))))
(I)
where Mur V1 are the chosen and rejected preference candidates, and a function that maps the difference between the two rewards into probability; its specific form:
((r+) = EE ~ log (ra( + (to
where the preference discretized, i.e., the chosen response yu is always annotated as better than the rejected response Y1 among different annotators, and the preference formulation is based Bradley-Terry (BT) model definition.
Finally, policy my is learned maximize the following alignment objective (Ziegler et al. 2019 Ji et al. 2024)
((TO) = RE (Fw.DDEByy (e) v)] (To Tref (v)
where D is a task-specific dataset, Tret is the reference model, which is usually the initial checkpoint of ma. typically model supervised-finettuee (SFT-ed) instruction-following datasets. DRL. is the Kullback-Ledd Leibler divergence loss and /3 is a density coefficient.
3.2 RL FINE-TUNING
One standard approach to optimize the alignment objective Eq. (3) is to use RL algorithms, which is consequence of the discrete nature of language generation. Recently, Ziegler al. (2019) proposed
to search for ny that maximizes a KL-regularized reward r y) - B log [
xe(a|a)
)x)) which can be
achieved by policy gradient methods, such as Proximal Policy Optimization (PPO, Schulman et al.
(2017) and Group Relative Policy Optimization (GRPO, Shao al. (2024)).
3.3 DIRECT PREFERENCE OPTIMIZATION
Rafailov al. (2023) proposed DPO that optimizes we directly from the preference data. Eq. (3) can be organized as
min E [KL((T [I - log Z(x)),
(4)
where the function Z(x) satisfies Z(x) = E, Tee(()))) exp( f (z, v)), and the optimal solution T satisfies n" (y (x) = Z Tinit (y xx) exp( (T)
The optimal solution of Eq. (4) is obtained when KL(TØlIT*) is minimized. Let क be the opti-
mal solution of Eq. then Ty equals to T². The relationship between ra and mo can be further expressed as:
To(2, y) = Bog (z(xzz + 3 log ((x).
(5)
[y(e(y(z))

---

Published as a conference paper at ICLR 2026
Then, they proposed to directly optimize the policy ne by replacing Th with Te and substituting the corresponding reward function into a pair-wise preference loss:
(TPO(TT) = (())) [ log (3 log
To (o
To (vx (z) Too (yr (x) )
[z)
B log
(6)
Treef (Y
Our goal is understand the requirements of preference candidates during the alignment process when performing alignment methods like DPO. In the following sections, we try to achieve our goal by answering the following two research sub-questions (RQs) empirically and theoretically:
RQ1: Can perform a qualitative description of the alignment process, can characterize the requirements of preference candidates through the alignment process?
Is possible to ensure that the qualitative description of the alignment process has action- able insight and can help conduct the effective alignment approach?
RQ2:
EMPIRICAL ANALYSIS
4.1 ANALYSIS SETUP
Models. We use different models including Llama-3-8B-Instmuce ( Meta 2024), Zephyr-sft- full (Tunstall et al. 2023) and Phi-2 (Li et al. 2023) for experiments. We select these models based on their parameter scales and training stages. We use PairRM (Jiang et al. 2023b) as the ground- truth preference model in our experiments, acting surrogate to expensive human preference for preference annotation. More details are shown in Appendix C.1
Dataset. We use the prompts and preference candidates from UltraFeedback (Cui et al. 2023) then relabeled the preference by PairRM to get the final off-policy dataset, aiming at ensuring the identical preference between different preference datasets. More details are shown in Appendix
Benchmarks. Following previous works (Meng et al. 2024 et al. 2024), We use AlpacaEval 2.0 (Dubois et al., 2024b) as our evaluation benchmark and report the length-controlled win rate over the reference responses. More details are shown in Appendix
4.2 MAIN RESULTS: THE EFFECTIVENESS DISCREPANCY BETWEEN OPF-POLICY/ONNPOLIC DATA EXISTS
Firstly, we propose a two-iteration training framework for each model, incorporating full combi- nation of off-policy and on-policy candidates. For each model, we conduct four distinct training configurations: 1) PCoot+off: Two consecutive iterations using off-policy candidates; 2) PCoff-oom First iteration with off-policy candidates followed by on-policy candidates; 3) PCon-oof: First iter- ation with on-policy candidates followed by off-policy candidates; and 4) PCon Two iterations exclusively using on-policy candidates. We provide more details in Appendix [C.4
We present result in Table T Our observation and conclusions ness discrepancy between PCoff and PC.n exists among different models. For Llama-3, models trained with PCon consistently outperform those trained with PCoff given the same initial model in every setting (A<1), which suggests PC... generally improve Llama-3 better than PC However, results on Zephyr are observed to be different from those of Llama-3, Models trained with PC.. outperform those with PC.ff when the initial model has been trained with PC in the previous iteration (A>1). other cases, PC... leads to worse performance for Zephyr compared with PC.# (A<1). For Phi-2, the results are opposite to those of Llama-3. Model trained with PC. consistently outperforms that with PC.. all settings (A>1). 2) The alignment process may result in failure when using PC..- We observe slight performance drop for Phi-2 when trained with PC... particularly if the initial model is the SFT model or has been trained with PC. in the previous iteration. 3) The effectiveness of PCof varies within the cireumstances. We observe varying improvements when optimizing Zephyr by PC across dif- ferent training iterations (12.7/3.0/s 5/8.5.0//..ppoin increase). The discrepancy between PC.#f and PC...
follows. 1) The effective-
model under different
5

---

Published as a conference paper at ICLR 2026
Bers
Iire22
Win
Avg. Tem A(8)
Win Rate
Ang. Ean
A(s)
Win
Arg. Tan
A/x)
Llama-3-8B- Iustrtttttttt
Zeplox-7B
P1a-2-2.7B
34.50
24.17
1921
8.12
524
$.72
915
PC
PClt
34.04;
27.77) u.
$1.17
1935
2014
0.53
20, 20.77 17 (0)
13.700 1
9.500
19013
1228
2.27
4.21: 1.:)
983
400
PCir
PClt
PCaa
PC
27.58,
10.57)
12:40)
41.89
2004
1723
<0.01
33.28;
23. 13,77,1
30.85
21.67
2525
1767
0.24
(.44
4.02) 149)
3-10
1077
8A5
PC
PGG.
PCaa
PCu.
26.36
41.52;
asst
(0.49)
90.57
2010
2473
22, 18-00)
19.10 (0-
18.05
1656
17:46
1.56
5.65, (134)
3.73 1:00
946
1.13
Table 1: Results of full-combination two-iteration experiments for all three models. "PCon" " and "PCof" " refer to on-policy and off-po preference candidates respectively, "iter" is the abbrevia- tion of "iteration". As focusing on the length-controlled win rate (LC Win Rate) of the benchmark, the red number shows the relative increase compared to the initial model (i.e., iter-2 compared to i iter-1 compared to SFT) while the green number shows the relative decrease. A shows the ratio relationship of relative increase between models trained with Pf and PC "+o0" means there is performance drop when training on PCoff or PCon-
shows that during the alignment process, the requirements of preference candidates are dynamic. This patterned dynamic nature motivates our central proposal: the alignment stage assumption.
We introduce the alignment stage assumption to model the dynamic requirements of preference candidates. Specially, the alignment process can be divided into two stages, the preference injection stage and the preference fine-tuning stage. During the preference injection stage, PCoff will be more effective; when the model comes into the preference fine-tuning stage, PC will be less effective than PC According to the results in Table [ and the aligmment stage assumption, we note that Llama-3 has been in the preference fine-tuning stage in all settings; after training PC Zephyr is in the preference fine-tuning stage; Phi-2 is in the preference injection stage in all settings.
4.3 DIVERSITY AND QUALITY AS THE CHARACTERISTICS OF ALIGNMENT STAGES (RQ1)
To answer RQ1, following previous works (Ding et al. 2024 Grillotti et al. 2024), we focus on the two key characteristics of preference data: intra-diversity and answer quality, and perform experi- ments on Zephyr. We use Zephyr since it shifts from the preference injection stage to the preference fine-tuning stage after training with PCoff. To de-confound the effects of data characteristics from their on on-policy/off- policy nature, we introduce PCllama. a dataset constructed off-policy with regard to Zephyr by sampling from -lama-3-8B- I-stuuctt,,, then amnotating preferences using PairRM. All prompts of PClamma are the same as PC.n and PCoft We provide more details in Appendix C.5
PClaama is designed to isolate the impact of data characteristics. Through experiments, we show that the preference candidates in PC. have higher intra-diversity than those in PC and quality of preference candidates in PC is lower than that PC We provide experimental details about the comparison between PCoff and PCllama in Appendix C.5 Besides results of models trained with PCoff and PClamaa we also include the PC... results as references.
We present our results in Table 2 Our ob- servations and conclusions are as follows. 1) High diversity is more effective for models in the preference injection stage. Compared with the SFT baseline, model trained with PC. achieves a 12. 7-point performance increase. In contrast, model trained with PCltama achieves a 5.4-point performance increase, which is similar to the model trained with PCon that achieves a 5.6-point performance increase. However, when Zephyr has been in the preference fine-tuning stage, PCaff achieves a relatively smaller perfor- mance increase, which is 3.0 points, compared with PCitama and PCon. which are 8.6 points and 12.5 points, respectively. Similar results are also observed from experiments in $ 44.2 where PC.rr attributes to slight improvement for L.ama-3. 2) High quality will be more effective for models in the preference fine-tuning stage. For the model in the preference fine-tuning stage, being trained with PC achieves a 8.6-point increase. However, the relative performance increase is only 5.4 points when trained with PC for model
Table 2: Results of Zephyr-7B for RQ1.
Iter-1
Iter-2
LC Win Rate Win Rate
8.12
4.25
PC
20.77 (7 (+166)))
19.99
PC
135.3
10.15
PCon
13.70 (+5.58)
9.90
PC
PC
23.77 (+3.00)
21.67
PC
PCllama
29.32, (+8.55)
37.03
PC
PCon
33.28 (+12.51)
36.85

---

Published as a conference paper at ICLR 2026
in the preference injection stage. As PCltama being dataset with off- off-policy preference candidates with regard to ZBphyr-7B, the dynamic effectiveness is attributed to the dynamic requirements for models in different stages, where we conclude that quality matters at the second stage.
The narrative explanation of different stage characteristics through dynamic alignment goals. Model in the preference injection stage performs poorly and lacks knowledge about ground-truth preference and its corresponding high-reward region. The exploration will be low-effective since the high-reward region can hardly be explored. Data with high diversity aims at injecting preference knowledge into policy models. For the models in the preference fine-tuning stage, it is low-effective to perform large-scale preference injection, and the alignment goal shifts explore high-reward region, sampling responses that are of high quality.
5 BOUNDARY MEASUREMENT ALGORITHM THAT DETERMINES THE BOUNDARY BETWEEN ALIGNMENT STAGES (RQ2)
In this section, we analyze the requirements of preference data from theoretical perspective. We show the equivalence between the DPO objective and the alignment optimization objective ($5.1)
and conclude that we are finding better text distribution estimation general text distribution defined by ground-truth preference model when choosing preference candidates ($5.I) To find a better text distribution, we introduce the preference consistency, which is the sufficient condition of identical distributions between some text distribution π and general text distribution ne ($5.2) Finally, we propose the boundary measurement algorithm, a practical estimation of the preference consistency measurement ($5.)). We illustrate the relationship between text distribution estimation and boundary measurement algorithm in F Appendix All proofs are shown in Appendix
Notation. Generally, let x be policy that represents a text distribution. Following the notation in $3.1 let P : X' x y x y -> [0, 1] be the preference distribution that satisfies Bradley-Tery definition with respect to reward model The output PP(11 > y2|2:) represents the preference of y1 outperforming 1/2 Specifically, let TH be the general policy and the general text distribution, Toff be an abstract policy that generates the candidates of PCoff, Ty be the policy that generates the preference candidates of PCo., π be an optimal solution of T. Ip is the ground ground-truth preference distribution derived from the ground-truth reward model r" I9 is the parameterized preference distribution derived from ras which is the analytical solution of Eq. given Ty and Traf-
5.1
OPTIMIZATION CONSISTENCY ANALYSIS
Eq. establishes a one-way mapping between the reward model and policy model that for every reward model Tg. there exists policy mh that satisfies Eq. (5) and Th is the optimal solution of Eq. 43. First of all, we show that the one-way mapping is reversible, i.c., Eq. (5) satisfies for every To when optimizing through Eq. (6)-
Theorem 5.1. (Bijection between reward function and policy) Under mild assumption, for any policy my and the static reference model Tref, there exists a unique reward model re satisfying me being the optimal solution of Eq.
Theorem 5. indicates that the optimization objective of Eq. (6 and the alignment objective Eq. (3) theoretically equivalent. We then discuss the condition that achieves the optimal solution of Eq. (3) via Eq. (6).
Theorem 5.2. (The Necessary condition of the optimal solution of Eq. 3 The optimal solution of Eq. can only be achieved if the preference dataset Deref has infinite preference data.
indicates that 1) The optimal solution of the general alignment objective is practi- cally intractable, as it is impossible to construct preference dataset with infinite preference candi- dates. Given limited preference candidates, the optimization objective is the preference consistency between Ip* and IP, within the limited dataset. 2) The alignment process will be more effective if the limited preference dataset is a well-defined proxy of the infinite-sample preference dataset. Assuming that the preference candidates, i.c., text-based responses of the anfinit-sample preference dataset, are sampled from the general text distribution, then we are estimating general text distribu- tion when selecting preference candidates.
Theorem 5.2
7

---

Published as a conference paper at ICLR 2026
5.2 THE GENERAL TEXT DISTRIBUTION ESTIMATION
In this section, we aim at finding measurement that can estimate the distance between the general text distribution अद and the parameterized text distribution me. Regular distance measurement like KL divergence does not work since both text distributions are intractable. Instead, we aim to measure the consistency of the preference distributions between !' and Pg. which we will show to be sufficient condition of TG and Te being identical. First of all, we formally introduce the definition of gT and Po in Definition 5.3
Definition 5.3. The general text distribution ng is defined by the ground-truth P that satisfies
P (11 > y2 (2(2)) = a (log no (in (æ)) log ()(2)=))
and the parameterized preference given my is defined
Po(an > y2 = a (log Ne (in (x)) log (o(12
(8)
We note that Definition 5.3 is not related with the optimal condition defined in Eq. (3) and Eq. (5) That is because we will not introduce any assumptions premised on optimizing Eq. and the general text distribution should be irrelevant hyper-parameter and reference model Wref-
Theorem 5.4. (The uniqueness of TT) There exists unique лh under Definition 5.3 given well- defined P
Theorem 5.4 and Definition 5.3 indicate that F and मe form pair of bijections, which allows
us to estimate TH by estimating P.. We can thus measure the distance between two preference distributions that are derived from F and Va respectively as a proxy of the estimation between text distributions. First of all, we provide the definition of preference consistency in Definition 5.5
Definition 5.5. Given preference distribution F1 and F2 based on BT definition, the consistency between P1 and P2 is defined by the following formula:
Eg,11.2 [T > /2|²)] @ I [P2(11 > y2 y2[=)]]
where I: j0, 1| + {0, 1} is the indicator function that maps values in the interval [o, 0.5] into 0 and values in (0.5, 1] into 1. @ is the XNOR operator.
The preference consistency defined in Definition 5.5 achieves its maximum when [ > [/|=)] II [P2(1)) y2|z)] satisfies for any {x, 3/ 3/2), which is sufficient condition of two identical preference distributions. In other words, preference consistency is to determine probabilities of identical samples exhibit identical rank orders for both text distributions.
5.3 PRACTICAL ESTIMATION OF PREFERENCE CONSISTENCY
Given on- on-policy distribution To and oof-policy distribution Toll. we perform the preference con- sistency measurement between these distributions and the general text distribution nG. Let {yi)., {y2} )n be the responses sampled from na and Roff given prompt x with size m and n, re- spectively, For each prompt x, We estimate the preference consistency by responses sampled from both and Roff to reduce sampling variance:
[ (vi > vállc) 01 Pa + válw))
1
(10)
VÍ
which measures the consistency between and Po, and
I vdl=s) [Por > vdlm))]
(11)
which measures the consistency between IF* and Port Practically, we assume that To and wo are highly divergent text distributions and responses are sampled from largely distinct regions of the and [[ > v2(x=)] = O, an assumption empirically supported in Appendix F.I This allows the preference consistency between IP* and
vast text space, which allows that [[Poovv
f > v [z)=
Fo, Port to be simplified into
II
IP (vi > 2ol v and Σn Σn
[P' > yí x
8

---

Published as a conference paper at ICLR 2026
Reed
LC Wis Rute
LC
Win (initial)
A(x)
Win (initial)
Lluma 3 Inst-tstret
Zeplor-7B
P16-2-2.77
24.00
24.47
8.32
4.20
5.81
8.72
PCur PC..
27778
22.85
$4.47
0.62
0.53
20.27
18.70
13.00
9.50
0.40
2.27
5.97
4.21
2.50
3.52
0.23
4:20
PC.
PC.e
PC
PGe
20383
30.77,1284
41.99
24.35
0.06
<0.01
35128 283280403 a:a
23.77:43000
38.85
23.67
8.4440
4 on
3.46
4.43
4:0
PC..
FCe
(233)
22.22 10000
5.73 sas
PC.s
PC..
5.50 Da)
Table 3: Results of full-combination two-iteration experiments. The "BS (initial)" denotes the rel- ative boundary score of each initial policy, calculated as Vott (Voit + Von) from the results of the boundary measurement algorithm shown in Algorithm [ icy in the preference injection stage and thus dataset with better intra-diversity will be more effi cient (A>1). Otherwise, it is in preference fine-tuning stage and thus the quality matters (A<1).
A score less than 0.5 indicates the pol-
respectively. Under mild assumptions, these equations indicate that it is possible to select a better proxy of T from Ta and Toff by comparing preference consistency of Ty and Toff regarding P
Algorithm Boundary Measurement Algorithm
1: Input Preference datasets PC... PCon, Preference model P.
2: Von, Voir +- 0,0
3: for (x, U1, V2) ~ PC... do
4: Sample the paired responses (y) 3/) from PCot where the input of paired responses x' equal to æ.
5: for v. where E (31, ya ha {ví, do
6: Update Von + Von + 1 if T prefers better than y' given Otherwise, update Va#.
7: end for
end
9: if Von then
10: return Model is in the preference injection stage, PC.r.
11: else
12: return Model in the preference fine-tuning stage, PC...
13:
end if
We then provide the boundary measurement algorithm in Algorithm T. which is the preference consistency measurement when letting = = 2. The algorithm shows that alignment stages are decided by preference dataset and preference model jointly. In other words, one initial policy can be in preference injection stage and preference fine-tuning stage the same time given different off-policy preference candidates and preference models. However, once the preference model and off- off-po sef-policy preference dataset are given, we can decide the alignment stage that model is currently in, and thus optimizing preference data for policy models.
5.4 EXPERIMENTS OF THE BOUNDARY MEASUREMENT ALGORITHM
Following the experiment settings in $ we perform experiments on three base models to verify the effectiveness of the boundary measurement algorithm. We present our result in Table 3 For Llama- 3, the results the stage assumption well. The boundary scores are greater than 0.5 for all initial models, indicating that Llama-3 is in preference fine-tuning stage. The results for Phi-2 also align with the stage assumption, as the boundary scores are less than 0.5 for all initial models, showing that the model is in preference injection stage. For Zephyr, the results fit the assumption well given the SFT model or the model trained with PC as the initial models. We notice that the model trained with PCon has positive score (0.58), but the follow-up training with PC.ff (an 8.5-point increase) is still more effective than PC.n (a 5.5-point increase). We attribute it to the lower quality of PC.. relative to PC.g. We measure the quality of PC.. following the comparison method used in Appendix C.55 The result shows that that the length-controlled win rate of PC... compared with PCoff is 0.46, indicating that the quality of PCon is lower than that of PC,
GENERALIZABILITY ANALYSIS
In this section, we further extend the experiments on two models (Qwen2.5-1.5B YYang al. 2024 and Pythia-6.9P (Biderman et al. 2023)) and another LM alignment method (SLiC-HF (Zho et al. 2023) to verify the generalizability of our conclusions.

---

Published as a conference paper at ICLR 2026
Ite--1
Iter-2
LC Win Rate
Win Rate
BS (initial)
A [K)
LC Win Rate
Win Rate
BS (initial)
(xx
Qwen2.5-1.58
Pythia-6.91
5.41
3.00
1.81
1.06
PC.s
(+24(+1.88)
8.78
1.28
2.45
0.35
0.22
PCo.
4.85
2.69
1.02[
1.48
PCor
PCos
PG.e
PCcn
7.65(40.41)
(.27(+3.86)
11.12
10.06
0.47
9.41
2.01(+0.73)
2.51(+
4.72
3.25
0.26
1.68
PCas
PCaE
7.08(+2.23)
8.58
2.79
3.46
0.38
2.48
0.24
1.49
PCon
PCcn
5.75;40.90)
3.45
2.21
3.12
Table 4: Results two-iteration experiments in Qwen2.5-1.5B and Pythia-6.9B.
Generalizability Analysis on Additional LMs. We further extend experiments on Qwen2.5- 1.5B (Yang et al. (224) and Pythia-6.95 (Biderman et al. 2023). We follow the experiment settings in $4 and train the models on UltraChat for one epoch first. We report the results in Table [4] The results show that the effectiveness discrepancy between PC.. and PC exists. Specifically, the boundary score shows that the initial checkpoints of the two models, i.e., the SFT checkpoint and the checkpoints trained on PC and PCoff in the first iteration all in the preference injection stage. As shown in the results, the performance of models trained on PC, outperforms those trained on PCon given the same initial checkpoint among different models, which fit the our conclusions well.
Generalizability Analysis on SLiC-HE. Though the empirical analysis of the two-stage assump- tion and the theoretical analysis of the boundary measurement are based on DPO, we show that the assumption and our conclusions can be further extended to other LM alignment methods. In this section, We perform experiments on SLiC-HF (Zhao et al. 2023).
We report the result in Table The results show similar trend those aligning with DPO, where we observe the effectiveness discrepancy between PC and PC. for different models. By performing the alignment stage assumption for these modcls and performing the boundary measure- ment, we observe a similar result as those aligning with DPO, which shows that the effectiveness discrepancy exists, and we can apply the two-stage assumption and judge the boundary between stages via the boundary measurement we proposed in Algorithm I
IterJ] Mer-2 LC Win
(intiaa) A(X]
Rate
Giiitall
A(*)
(initiah Arx)
Llame Instruct
Zephyn-71I
Phi-2-2.77 3.72
21.50
21.47
8.12
4.30
Sat
PCue
17.73
PC..
41.06:4441)
15.26
-ocen on
1:0
PCL.
PC..
PC
12.6 on
25.00
7.07 wa
PC..
PC..
PC
PCu
38 46:
1263 euan)
0.2
19,84
0.98
3.63
5.85
1.51
Table 5: Results of full-combination two-iteration experiments performed with SLiC-HF loss. The boundary score can be good measurement to decide the boundary between each alignment stage.
Though the result matches the assumption and algorithm in most cases, we also observe a model collapse phenomenon for Llama-3 trained with PCof and PCon subsequently, where very serious performance degradation is observed. It may result in the difference between DPO and SLIC-HF, as a similar performance degradation not observed when aligning with DPO as shown in Table
CONCLUSION AND LIMITATION
In this work, we reveal the effectiveness discrepancy between on-policy data and off-policy data for different models, and propose the aligmment stage assumption when performing LM alignment through DPO. We characterize each alignment stage through analyzing the discrepancy by diversity and quality. We provide the boundary measurement algorithm, a theoretical -groundd-- method to decide the alignment stages. Though being an effective simplified abstraction of alignment process, the aligmment stage assumption inspires exploration of smoother and more adaptive data blending strategies rather than rigid switch, which is not included and we leave for future research.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our work and to facilitate a clearer understanding of our contribu- tions, we provide extensive supporting materials. In the main text, we describe the models, bench- mark and training parameters we used in our experiment in $4 In Appendix C we provide further detailed information, including model details, evaluation details, data details and training details. Our work is based on open-sourced models, open-sourced dataset and open-sourced benchmark, which ensures our results are reproducible.
ACKNOWLEDGMENT
We thank the insightful suggestions of anonymous reviewers. This work is supported by grants: National Natural Science Foundation of China (Grant No. 62406088 and 62422603), Guangdong Basic and Applied Basic Research Foundation (Grant No. 2025A1515011376, Guangdong Basic and Applied Basic Research Foundation (Grant No. 2024B0101050003
REFERENCES
Meta. Llama model card. 2024. URL https://github.com wwwp://ttwt....ooottoetotoe 11ama/
lama3/bor 3tt/3///llbmaa/////m n/MOBEL CARD md
Oron Anschel, Nir Baram, and Nahum Shimkin. Averaged-dqn: Variance reduction and stabilization for deep reinforcement learning. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 176-185. PMLR, 2017. URL htttt:t/ppp/oeeeeeeeeee
ngs ml www.l..pres//v000 wnwphheenncchee111e html
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jack- son Kernion, Tom Conerly, Sheer El Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Her- nandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, Benjamin Mann, and Jared Kaplan. Training helpful and harmless assistant with reinforcement learning from human feedback. CoRR, abs/2204 05882, 2022. doi: 1044550/ARXIV2204.05822 URL https:://oi.... org/00......0.... 48550/ arwiv. 2204 05862
Emily M. Bender, Timnit Gebru, Angelina McMillan-Majon and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Madeleine Clare Elish, William Isaac, and Richard S. Zemel (eds.), FAccT '21: 2021 ACM Conference on Fairness, Accountability, and Transpurency, Virtual Event Toronto, Canada, March 3-10, 2021, pp. 610- 623. ACM, 2021. doi: 3442188. 3445922
011555/442118.3445550 URL https:// ottt:::://ci.org///.. 10.41..1.7
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning. ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. proceedings mlr. hateess/v22//bicce
PMLR, 2023. URL https://
html
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan,
11

---

Published as a conference paper at ICLR 2026
and Hsuan-Ticc Lin (eds.), Advances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL http:: /ttp:::/prooeeddnn...eeeee 1457c0d6bf cb4 4674166 8bfb8ac1 42 42f64a 9 ot httpp///w......aap///////////aaa
Hardy Chen, Haoqin Tu, Fali Wang, Hui Liu, Xianfeng Tang. Xinya Du, Yuyin Zhou, and Cihang Xie. SFT rl? an early investigation into training rl r1-like reasoning large vision-language models. models CoRR, abs/2504. 1468, 2025. doi: 10.4855/ARXIV.250411
10. 07..4455/axxi// 2504 2504. 11468
Wei-Lin Chiang, Zhuohan Li. Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
https:
03 00-30--00-000000000000000000000- 100-00-v--00 vicuna/
Tianzhe Chu, Yuexiang Zhai. Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V. Le, Sergey Levine, and Yi Ma. SFT memorizes, RL generalizes: A comparative study of foun- dation model post-training. CoRR, abs/2501 17161, 2025. doi: 1044550/ARXIV:355..771 URL httpthttt///ooot wrgp:/o.oorg.1... 48550/ www.v./rxvvv.... 2501 2501.17161
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. CoRR, abs/2310.01177, 2023. doi: 1044550AARXW..2110003777 URL huttps://doi.
48550/ 2310. 01377
Li Ding, Jenny Zhang, Jeff Clune, Lee Spector, and Joel Lehman. Quality diversity through hu- feedback: Towards open-ended diversity-driven optimization. In Forty-first International Conference on Machine Learning, 2024. URL
net www.:/uneeffuummm//// forum?io
99 ZuAAb08
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou Enhancing chat language models by scaling high-quality instructional conversa- tions. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.). Proceedings of the 2023 Con- ference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, De- cember 6-10, 2023, pp. 3029-3051. Association for Computational Linguistics, 2023. doi: 111.6///1/222..MM.MMMMMM em..p main. 183
UtRl:ttps::/dde .eriilorgg//00 10t5//000016633/1/222233
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshom, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia- Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle I Lee, Georgia Lewis Ander- son, Graeme Nail, Grégoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Ko- revaar, Hu Xu. Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lec, Jan Geffert, Jana Vranes, Jason Park, Jay Ma- hadeokar, Jeet Shah, Jelmer der Linde, Jennifer Billock, Jenny Hong. Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Al- wala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. The llama 3 herd of models. CoRR, abs/2440.22178. 2024. doi: 1048550AAXXI.2407222232 URL https: doi. www.oog...... 48550 arw.v..iiv....... 2407 21783
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. Length-controlled
alpacaeval: A simple way to debias automatic evaluators. CoRR, abs/2404.044 2024a. doi: 10.
URL //doi wrgg/0rg....... 48550 a v 2404.244444. 200...444.444..54...0 04475
12

---

Published as a conference paper at ICLR 2026
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled al- pacaeval: A simple way to debias automatic evaluators. arXiv preprint axXiv:2404.04445, 2024b.
Xinyang Geng, Arnay Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song. Koala: A dialogue model for academic research. Blog post, April 2023. URL https:
7/bair. www.eley..e..... wwt..edduubloooo 202 /04/ 03/koala/
Luca Grillotti, Maxence Faldor, Borja G. León, and Antoine Cully. Quality-diversity actor-critic:
Learning high-pertin performif and diverse behaviors via value and successor features critics. CoRR, doi: 10448550/ARXII.2403.0099
abs/2403.09930, 2024.
URL htthhtts:: //dttps:/doi... orq/10.
48550/arXiv 2403. 09930
Yiju Guo, Ganqu Cui, Lifan Yuan, Ning Ding, Zexu Sun, Bowen Sun, Huimin Chen, Ruobing Xie, Jie Zhou, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Controllable preference optimization: To- ward controllable multi-objeecive alignment. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.). Proceedings of the 2024 Conference on Empirical Methods in Nutural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pp. 1437-1454. Associ- ation for Computational Linguistics, 2024. URL httpss///cclaantoooooooo/o////// emm1pp main.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using clectra-style pre-training with prudient-disentange embedding sharing. In The Eleventh International Confer- ence on Learning Representations, ICIR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL /ttptt:://ppseeeeeee
net/ forum?
Shengding Hu, Yifan Luo, Huadong Wang, Xingyi Cheng, Zhiyuan Liu, and Maosong Sun. Won't get fooled again: Answering questions with false premises. In Anna Rogers, Jordan L. Boyd- Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 5626-5643. Association for Computational Linguistics, 2023. doi: 1/21655//1/2223. acl long.309
10w.0..86533vvv ACL-LONG.309 ottp://..o....ttprpp///ooooooo// /tttt:/://bol..rrr1111116///////
Haozhe Ji, Cheng Lu, Yilin Niu, Pei Ke, Hongning Wang. Jun Zhu, Jie Tang, and Minlie Huang. Towards efficient exact optimization of language model alignment. ICML. OpenReview.net, 2024.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Comput. Surv. 55(12):248: 1-83.283 2023. doi: 10.1145/3571730. URL 1145/3571730
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William Sayed. Mistral 7b. CoRR, abs/2310.06825,, 2023a. doi: 10.48550/RRIV.2310.00
URL httts:://doo .ww.orrg/110 4...00.445500/000 2310.
06825
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. Lim-blender: Ensembling large language models with pairwise ranking and generative fusion. Anna Rogers, Jordan L. Boyd--ooaobbeee and Naoaki Okazaki (eds.). Proceedings of the ól Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 14165- 14178. Association for Computational Linguistics, 2023b. doi: 18 792. URL httplhttttss::oo..rgggggg 1tt.://///1.11113///22022233
792
Dongyoung Kim, Kimin Lee, Jinwoo Shin, and Jachyung Kim. Spread preference annotation: Direct preference judgment for efficient LLM alignment. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net 2025.
URL
net ftte/forum?idce /otp
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Due Nguyen, Oliver Stanley, Richard Nagyfi, Shahul ES, Sameer
13

---

Published as a conference paper at ICLR 2026
Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. Openassistant conversations democratizing large language model align- ment. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, De- cember 10 16, 2023, 2023, URL htt::: nips cttpspprccppppppppppp serer_fieesspprrr 2023/ hww.hahhhaa0 94 ww0944400833338 haaas//4900fff2222 6722 www.:cccddeeeeeee22eeeee Abstract Datasets.
and Benchmarks .tmll
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. Tülu 3: Pushing frontiers in open language model post-training. CoRR, abs/2411..51222 2024 doi: 10. 08550/ARXIV.2411.555
URL http:: //doi 10. 48550/
Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. Generative judge for evaluating alignment. In The Twelfth International Conference on Learning Rep- resentations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https: /ttt::://ppeneeiew..no forum? 3d=edddktttsssssss
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need II: phi- 1.5 technical report. CoRR, abs/2309.05463 2023. doi: 10. 48550/ARXXI.23000054444 URL htt.t:ttttst..... //doi wwwgroo..o....ooo.oooooo 485508850/// 205/550/axxvv..m http://aww....2....300
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic hu- man falsehoods. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Pro- ceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pp. 3214-3252. Asso- ciation for Computational Linguistics, 2022. doi: 10. 229
URL
Jie Liu, Zhanhui Zhou, Jiaheng Liu, Xingyuan Bu, Chao Yang, Hansen Zhong, and Wanli Ouyang. Iterative length-regularized direct preference optimization: A case study on improving 7b lan- guage models GPT-4 level. CoRR, abs/2406.11817, 2024. doi: 10.48550AARX1..26611188 URL http::ttt///cct /wi. www.://g......./........ wttp:///arii.vvoo 2406. 11817
Zihan Liu, Zhuolin Yang, Yang Chen, Chankyu Lee, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Acereason-nemmotronn 1.1: Advancing math and code reasoning through SFT and RI. synergy. CORR, abs/2506. 1 13284, 2025. doi: 10.48550/ARXIV25006.1 13284. URL hhtpp::/
doi 0rgg/0..4850000 arxiv. 250..1285064.444
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoe V. Le. Barret Zoph, Jason Wei, and Adam Roberts. The flan collection: Designing data and methods for effective instruction tuning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Confer- ence Machine Learning, ICMI, 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 22631-22648. PMLR, 2023. URL https://procee
mlr prw.s w20p//22222222222 http///22///oo222aaaemm http://oonrreeaaaae
Xingzhou Lou, Junge Zhang, Jian Xie, Lifeng Liu, Dong Yan, and Kaigi Huang. SPO: multi-dimensional preference sequential alignment with implicit reward modeling. CoRR, 0is/2405.11739) 2024. doi: 00-4850/ARXI...455.2277 URL https://doi. org/10. 48550/arXiv.2405..
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference- reward. In NeurlPS, 2024.
OpenAl. GPT-4 technical report. CoRR, wbw.2303..08774,220000 2023. doi: 10.45550AAPXX222933007744
URL wttptttt:://// wtt..o///....../araiii.i..oo
14

---

Published as a conference paper at ICLR 2026
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddic Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feed- back. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35; Annual Conference on Neural Informa- tion Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28- December files wwwpsetwwwweeeeeee.tapt..appteee /atpe////pper//2///aa.t html
9. 2022, 2022. URL http: wwwppapppaapaaaaa / mips.
Matteo Papini, Damiano Binaghi, Giuseppe Canonaco, Matteo Pirotta, and Marcello Restelli. Stochastic variance-reduced policy gradient. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stock- holmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. html
PMLR, 2018. URL http::/proceeddnng mmlr.
press/v80/papie
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NewrIPS 2023, New Orleans, LA, USA, December 10- 16, 2023, 2023, URL TtRLhttp:/// /papers///pppppppppeeeeeee nips. wwtpaperppppppppteepcpeeeeeee.ee fwwes/ /ttps//ppeer/222//aaaa Abstrr-bsstttttc Cow.rrccctooooffecec...oo html
Alexandre Ramé, Nino Vieillard, Léonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. WARM: on the benefits of weight averaged reward models. In Forty-first International Conference on Machine Learning. ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https::/ http:s///ppeeeeiee...o net forum?id- RRDnNUJJ6
Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. Direct nash optimization: Teaching language models to self-improve with gen- doi: 1048550/ARXX.2400.07777 URL https::/dolt 20tp:///....5.//.2222...........
eral preferences. CoRR, abs/2404.03715, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1 7004706677, 2017. URL bttp:://axivv orw.v..rrg/aa. abs/
1707.06347
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. CoRR, abs/2402.03300, 2024. doi: 10:4550/AAXX..2222.030.0 URL https::/doii www.orgg...... 48550 wttp5///aaxii..2.......2 233000 03300
Nisan Stienmon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. Learning to summarize from human feedback. CoRR, abs/2009.01325,, 2020. URL htthtttss///axtt .rg. / abs /ttpg//bb//...../////// 2ww.:00000.00 000.0.9...11..
Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Ste- fano Ermon, Chelsea Finn, and Aviral Kumar. Preference fine-tuning of Ilms should leverage suboptimal, on-policy data. CoRR, abs/2404. 14367, 2024. doi: 10:4550ARRIV..204.4440 URL /tl.:ttp://ooi.ogg1..... 48550/485556660 htwp//ariivv.... 2wt..i.i.........3...
Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Yuan Cao, Eugene Tarassov, Rémi Munos, Bernardo Ávila Pires, Michal Valko, Yong Cheng, and Will Dabney. Understanding the performance gap between online and offline alignment algorithms. CoRR, abs/2405.08448, arrivii.........
2024. doi: 10.48550/ARXIV224055 088488. URL https: /ooi org/10..16000.ooooooo. 48550/885000000 2405. 08448
15

---

Published as a conference paper at ICLR 2026
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of Im alignment, 2023.
Yizhong Wang, Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannanch Hajishirzi. Self-instruct: Aligning language models with self generated instructions. In Anna Rogers, Jordan I.. Boyd--rrabes and Naoaki Okazaki (eds.), Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023. pp. 13484-13505 Association for Computational Linguistics. 2023. doi: 20.18653/11/0023..00 2023. long--ong.... 754
A/0-220 ACL-LONG. 754. URL httttp:://ooi.org/10..o
Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. CoRR, abs/2405.000 2024. doi: 10. 48550/ARXIV 00675.
wwt.oo/./........... 48550/ arxiv. www......0..000.2..
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qing- wei Lin, and Daxin Jiang. WizardIm: Empowering large pre-trained language models to fol- low complex instructions. In The Twelfth International Conference on Learning Representa- lions, ICLR 2024, Vienna, Austria, May 7-11, 2024, OpenReview.net, 2024. URL https: //openzeview. net/ forum? (onddddCCfhh3NBN
Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. Learning to reason under off-policy guidance. CoRR, abs/2504. 14945, 2025. doi: 10.48550/ 140UV.5504.14445. URL htttps://cc..og///// 485500.5550///4/ 2504..554... 14945
An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu. Mei Li, Mingfeng Xue. Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. CORR, 155/2412.15115, URL https: /wc.. www.oo.gg1100.... 48150/ /ttp:///....001018888000000000/0
2024. doi: 10.45550/ARXX..4412.555
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang. Jian Yang, Jianhong Tu, Jianwei Zhang, Jian Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin. Xingzhang Ren, Xinyu Wang, Xinyu Zhang. Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang. Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. CoRR, abw/25555009355 2025. doi: 10:45550/ARXIV.5500000
URL
htttps://dl/ wrtgooogooooooo.o..oooooooo....e 48540 wwt.0/aaxivv...m www.xivv..550.5 2505. 09388
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng. Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan Dai. Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei- Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. DAPO: an open- source LLM reinforcement learning system scale. CoRR, abs/2503. 14476. 2025. doi: 10. 48550/ARRXV.2503..120 14476. URL htttps:://ooo. org/10. 48550/ www.a.aiiiv...... 2503. 14476
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. CoRR, 10t/22201.100020 2024. doi: 10.48550/ARXIV. 2401.10020. URL https:/:////
http:/.....55550 18550/arXiss 2401. 10020
Dun Zeng, Yong Dai, Pengyu Cheng. Longyue Wang. Tianhao Hu, Wanshun Chen, Nan Du, and Zenglin Xu. On diversified preferences of large language model alignment. In Yaser Al-Onaizan,
16

---

Published as a conference paper at ICLR 2026
Mohit Bansal, and Yun-Nung Chen (eds.). Findings the Association for Computational Lin- guistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024, pp. 9194 92100. Asso- ciation for Computational Linguisties, 2024.
Finding: Sandingseeme 5.38 538
Shenao Zhang, Donghan Yu, Hiteshi Sharma, Han Zhong, Zhihan Liu, Ziyi Yang. Shuohang Wang, Hany Hassan Awadalla, and Zhaoran Wang. Self-exploring language models: Active preference elicitation for online alignment. Trans. Mach. Learn. Res., 2025, 2025a. URL
htt.ps://openreilli
net/ wwwffformmioooo Ffo.rrrmmimFFggA3mmWY
Wenhao Zhang, Yuexiang Xie, Yuchang Sun, Yanxi Chen, Guoyin Wang, Yaliang Li, Bolin Ding. and Jingren Zhou. On-policy RL meets off off-policy experts: Harmonizing supervised fine-tuning and reinforcement learning via dynamic weighting. CoRR, abs/2508. 14008, 2025b, doi: 10.
48550/ARXIV22508. 1408. httttss::://dt //dot http://ooooorg/////// 48550 arxivv 2508 11408
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter Liu. Slic-hf: Sequence likelihood calibration with human feedback. CoRR, abs/2305. 10425, 2023, doi: 10.
48550/ARXIV.2305..1 10425. URL http:::/tpooooo otg./oi..orrgrgg /10 48550/ www........114444.
Zhanhui Zhou, Jie Liu, Jing Shao, Xiangyu Yue, Chao Yang, Wanli Ouyang, and Yu Qiao. Beyond fie- alignment: Multi-objective direct preference optimization. In Lun-Wet Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguisties, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-
16. 2024, pp. 10586-10613. Association for Computational Linguistics, 2024. doi: 10.
URL ht:tp:://6ll.org/10.88858 /tt..////....155////22323
Findings-acl. 630
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paull F. Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. CoRR, abs/1909.08593, 2019. URL
www..orggaaooe.. abtg///b//ttee 1909. 08593
17

---

Published as a conference paper at ICLR 2026
A THE USE OF LARGE LANGUAGE MODELS
In this work, we utilized LLMs solely for the purpose of polishing writing. The LLMs were not used for content generation, and all research, analysis, and conclusions presented are the result of our own work and independent thought.
B FURTHER DISCUSSION
B.1 COMPUTATIONAL COST OF ALGORITHM
The boundary measurement algorithm requires one-time comparison subset of the data, which requires performing on-policy sampling by current poliey to acquire PC In our experiments, we use 2, 000 prompts in the "test_prefs" split of UltraFeedback (binarized) dataset for this measure- ment. Specifically, we compare the on-policy samples generated by current policy and the off-policy samples derived fom the "esttppeefs" split of the UltraFeedback (binarized) dataset, using PairRM as the preference model. Compared to full DPO training on the 61, 135-sample dataset, the com- putational overhead of our boundary measurement is negligible, estimated to be 3.2% of single training epoch. This demonstrates that our method is not only effective but also highly efficient and practical for real-world application.
B.2 DEPENDENCY OF PREFERENCE MODEL
A key aspect of our boundary measurement is its reliance given preference model P define the ground truth for the stage decision. This means the resulting stage boundary is relative to the preference model I.. IP weak or biased, the boundary decision might be suboptimal for alignment towards true human preferences, but it will still be optimal for aligning towards the world view of P. This highlights the importance of the choice of the preference model, a factor common to all preference- based alignment methods.
AMPLOORATTO-EXXXLOIIATIOO B.3 CONNECTION WITH EXPLORATION EXPLOITATION
Our two-stage assumption can be viewed as a simplified instantiation of the classic exploration- exploitation trade-off in reinforcement learning within the context of LM alignment. While tradi- tional reinforcement learning focuses on exploration in state-action space, our work suggests that alignment methods like DPO, exploration happens in space of preference candidates. Choosing preference candidates with high diversity as form of exploration, where the model seeks to learn broadly about the reward landscape defined by preference model; while choosing high-quality preference candidates can be regarded of exploitation, where the model refines its policy within high-reward regions defined by preference adaptive switch between the
LM alignment via preference-basee
be regarded
form
model. Our boundary measurement algorithm, therefore, exploration phrase and the exploitation phrase.
B.4 DISCUSSION ABOUT DISTRIBUTION SHIFT THEORY
One possible confusion about the empirical analysis about stage characteristics we introduced in $4.3 lies in the contradiction between stage characteristics and distribution shift theory. Different from quantifying preference candidates by diversity and quality, PC.n is an "in-domain" dataset, as its preference candidates are sampled from the current policy, while PCof an "out-of-domain" dataset, its preference candidates are sampled from models different from the current policy. As a consequence, the effectiveness of PC... may lie its sharing the identical sampling distribution during the alignment process with regard to current policy. We alleviate the influence of distribution shift from two aspects.
First of all, the distribution shift theory posits that on-policy data is always superior to off-policy data. However, our results in $4.2 showing that optimizing models based on preference candidates sampled from their identical distribution is not always effective, which indicates that distribution shift is not the sole, or even the primary factor towards LM alignment. For example, for Phi-2, training with PC.n leads performance drop, while training with PC.#t, whose samples from
18

---

Published as a conference paper at ICLR 2026
more distant distribution, leads to performance increase. Secondly, de-confound the effects of data characteristics (i.e., diversity and quality) from their on pollicc//off-ooo policy natures. Specifically, use PCitama in $4.3] whose preference candidates are sampled from another model (i.e., Llama- 3-8B-Instruct) that is distant to current policy (i.e., Zephyr-7B). Through empirical analysis about PCoff and PCllama introduced in $C.5. we quantify the characteristics of PC.ff and PCltama- This allows us to isolate the impact of data characteristics.
B.5 DISCUSSION ABOUT IMPORTANCE OF PREFERENCE DATA SELECTION
As the field of LLMs matures beyond the primary pursuit of scale, the central challenges have shifted towards efficiency, reliability, and cost-effective customization. The decision of how con- struct the effective dataset lies at the heart of this new paradigm. On-policy data generation, while providing highly relevant samples, introduces significant computational and financial overhead, act- ing as a major bottleneck for the widespread adoption and specialized fine-tuning of models. Our work addresses this challenge by moving the data selection process from an empirical art to prin- cipled, stage-aware science. In an era increasingly focused on Data-Centric AI, instead of simply assuming on-policy data is superior, off-policy data can be more effective than on-policy data in some cases. By introducing the same LM inference overhead to construct the on-policy preference candidates and then optimizing LLMs, the model will achieve better performance when it has been in the preference fine-tuning stage. Our work provides diagnostic framework to understand the model alignment stage and to strategically choose data that maximizes efficiency. Our research of- fers a critical methodology for building better-aligned models more efficiently and reliably, a core necessity for the next generation of Al systems.
B.6 ADDITIONAL RELATED WORK ABOUT OPTIMIZATION VIA ON-POLICY AND OFF-POLICY CURRICULUM
On-policy reinforcement learning encourages LMs perform active exploration during the opti- mization process, which enhances their generalization ability by learning from feedback on their own sampled outputs (Chen et al., 2025 Chu et al. 2025). However, on-policy sampling is expen- sive and time-consuming, and can result in policy degradation caused by entropy collapse or over- exploitation of sub-optimal responses (Yu et al., 2025). Relatively, optimizing LMs via off-policy data is cheap and stable, while it struggles with limited exploration and learning from novel, self- generated responses. To this end, several works focus on the optimizing LMs via both on on-policy and off-policy data in empirical and straight-forward way. LUFFY incorporates off-policy re- sponses in Group Relative Policy Optimization method (GRPO, Shao et al.) (2024)) by adding them directly to the group of on-policy responses (Yan et al.) 2025). Qwen3 utilizes weak-to-strong curriculum strategy during the optimization process, training models in off-policy responses and on-policy responses subsequently in supervised fine-tuning manner improve the reasoning ca- pability of LMs (Yang et al.) 2025). Other works aim at incorporating off-policy data and on-policy data in a sequential SFT-then-Ri RL paradigm, either utilizing multi-task learning to balance SFT loss and RL objective at the same time (Zhang et al. 20256) or performing SFT first, then RL (Lambert al. 2024, Liu et al., 2025). In LM alignment scenario, our work reveals patterned effectiveness discrepancy between off- oof-policy data and on- on-policy data across different models, and proposes the alignment stage assumption to model the dynamic data requirements during the alignment process. Our work is aligned with findings in Reinforcement Learning from Verifiable Reward (RLVR) and offline RL sencarios, and can provide valuable and actionable discoveries for these fields.
C TRAINING AND EVALUATION DETAILS
C.I MODEL DETAILS
Llama-3-88 3-------BB--uuul is a large language model with SB parameter size, and has been aligned with human preferences for helpfulness and safety through supervised fine-tuning (SFT) and reinforce- ment learning from human feedback (RLHF). Zephyr-sft-
is a large language model with 7B parameter size, and is an aligned version of Mistral-7B (Jiang al. 2023a) that has previously pervised fine-tuned on UltraChat (Ding et al. 2023) dataset. Phi-2 is a pretrained language model with 2.7B parameter size, and has not been fine-tuned or aligned downstream tasks. Following
19

---

Published as a conference paper at ICLR 2026
the setup process and training settings of Zephyr-sft-full, we conduct supervised fine-tuning Phi- 2 on UltraChat for one epoch to get the fine-tuned checkpoint for alignment experiments. These models vary the model scale and training stage, which will result in different behavior in the subsequent experiments and be helpful for our analysis. We use PairRM (Jiang et al.] 2023b) as the ground-truth preference model in our experiments, an efficient pair-wise preference model of size 0.4B. PairRM is based on DeBERTA-V3 (He et al. (023) and has been fine-tuned on high-quality preference datasets. Results on benchmarks like Auto-J Pairwise dataset (Li et al. (024) show that PairRM outperforms most of the model- based reward models and performs comparably with larger reward models like UltraRM-133 13B (Cui al., (023). The reference model Tyee we used in different experiment is the initial checkpoint of the corresponding policy model.
C.2 DATASET DETAILS
UltraFeedback (Cui et al.
2023) is a large-scale, fine-grained diverse preference dataset for LM
alignment.
consists of 63, 967 prompts from diverse sources (including Ultra- ShareGPT (Chiang et al. 2023), Evol-Instruct (Xu et al. 2024)) Truth-
Chat (Ding et al. 2023),
fulQA (Lin al. ).22), FalseQA (Hu et al. 2023 and FLAN (Longpre et al.] 2023)). For each prompt, the authors query multiple LLMs to generate 4 different responses, then the responses are scored and ranked by GPT-4 (OpenAl 2023) based on criterion including foollociinn-folloiingg truthfulness, honesty and helpfulness. To construct the UltraFeedback (binarized) dataset, the re- sponse with the highest overall score selected as the "chosen" completion, and one of the remain- ing 3 responses at random as the "rejected" one, thus constructing the preference pairs.
We sample two answers by the current policy to acquire on-policy preference candidates. Specifi- cally, we use all of the prompts derived from UltraFeddback, sample two responses as the preference candidates, then annotate the preference between the preference candidates by PairRM. We called "blender.compare cetrmprrreccovvrrattoons method to annotate the preference between preference candidates, which is the official method provided by the authors of PairRM. To ensure the consistency of prefer- ence annotators between off-policy preference dataset (whose preferences annotated by GPT-4) and on- on-policy preference dataset (whose preferences are annotated by PairRM), We relabeled the preference of preference candidates in UltraFeedback (binarized) dataset by PairRM in the same way as labeling the preference in the on-policy preference dataset.
C.3 EVALUATION DETAILS
AlpacaEval 2.0 (Dubois et al., 2024a) is a leading benchmark that assesses LLMs' instruction- following ability and alignment with human preference. To construct the AlpacaFval test set, the authors combine a variety of instruction-folloving datasets like self-instruct (Wang et al., (023)) open-assistant (Köpf et al. 2023), vicuna (Chiang et al. 2023). koala (Geng et al. 2023) and hh- rlhf (Bai et al. 2022). and finally construct a dataset with 805 samples. It calculates the probability that LLM-based evaluator (gpt-4-1106 100----i0--rr prefers the model output over the response gen- erated by GPT-4, which provides an affordable and replicable alternative to human preference an- notation. The win rate over the GPT-4 baseline is computed as the expected preference probability. The length-controlled win rate is a modified version that reduces the length bias, which alleviates reward hacking and prevents flawed judgment. We report the length-controlled win rate as it corre- lates best with Chatbot Arena (Dubois et al. 2024b), the real-world alignment benchmark based on human evaluation.
C.4 EXPERIMENT DETAILS
For each training iterations, we use the initial checkpoint of current policy the reference model. For on-policy experiments, we sample two answers from the current policy, using prompts same as UltraFeedback, then annotate the preference by PairRM. The hyper-parameters when training mod- els are shown in Table The parameers when generating on-policy preference candidates shown in Table 6
In practice, we seldom see researchers perform the third approach (i.c., PC. which may be because the goal of on-policy sampling is to alleviate the out-of-distribution problem that training
20

---

Published as a conference paper at ICLR 2026
Value
Parameter
SFT
DPO
Epochs
1
Learning Rate
2.0 x 10-5
3.0 10-7
Batch size (per device)
Gradient Accumulation Steps
8
8 0.01
8
warmup ratio scheduler
0.1
0.1
cosine
cosine
GPUs
x A100
4 X A100
Table 6: Training typer-parameters (SFT and DPO).
Parameter
Value
top k top_p 0,9
50
temperature
0.7
Table Inference
(sampling on-policy preference candidates).
on off-policy data solely suffers, but the third approach can not handle it empirically for its end up training on off-policy data, We include this setting for the completeness of the experimental setup.
C.5 DETAILS ABOUT PCllama
Data Construction
To construct PCllama> we use the raw Llama-3 Bantaa---B-Inntttee model to gener- pair of on-policy reference candidates, following the settings introduced in Appendix C.22 and Specifically, we use the prompts same as PC.ft which are derived from Ultra- Feedback, and annotate the preference of on-policy preference candidates by PairRM. PCIlama and PCoff have identical prompts but different preference candidates. We abstract the core difference between PCltama and PCoff into two key characteristics, the intra-diversity and the answer quality, introduced in $4.
Appendix C.4
We then analysis the characteristics.
40
00
-20
aloggn
PUl PiCina PCis
Plan Fi
Pile PC
PCs
-01
PCir
300
umininn Aigs
Buikinng skgg
Snkiiing steg
Zephyr-7B results.
(b) Qwen2.5-1.5B results.
(c) Qwen3-4B results.
Figure 3: The iitra-diversity between PC and PCllama that is defined by the difference A) of log probabilities between the chosen and the rejected answer cross different models, inclding Zephyr- 7B, @wwe2.5511.5P and Qwen3-4B. The curves of PCon are also included as reference.
Diversity This section discusses the intra-diversity between preference pairs. We define the intra- diversity as the difference between generation probability preference pairs by given model, operationalized by the log-probability difference between paired responses as follows:
N (log To (vi (x) log (1)12));
Diviutra
N Σ
(12)
21

---

Published as a conference paper at ICLR 2026
where vi and v/ are the chosen and the rejected answer for the iu sample respectively. To compare the intra-diversity between preference pairs that derived from PCoff and PCllama- we record the log probabilities of preference pairs individually when training Zephyr-7B, Qwen2.5-1.5B. and Qwen3-4B, and present the result in Figure As shown the figure, during the training procedure, the difference in log probabilities of PC.ff has larger fluctuation range but the difference in log probabilities of PCllmma remains stable and close to zero. The results show that PCoff is more diverse than PCiamma The log probabilities of preferences pairs derived from PCon are also included in Figure 3 as reference. As shown in the result, PC and PC share similar trend during the training process. It indicates that though PCllams is an off-policy preference dataset for models except for Llama-3, it still holds the low intra-diversity characteristics as an on-policy preference dataset for different models. a result, experiments on comparing data characteristics including intra-diversity and answer quality while avoiding their on ca-polcccooff-oliiii nature.
reasonable dataset for conducting comparative
Quality We define answer quality as the degree of alignment with human preference. We compare the quality by measuring the preference labeled by the ground-truth preference model between swers sampled from Pf and PCllama- Specifically, we followed the official recipe of AlpacaEval benchmark and annotate the preference using GPT-4-turbo. The preference candidates are one ran- domly sampled answer from the preference candidates of PCllama and the chosen answer of PCoff. then report the result of length-controlled win rate on 805 cases that were randomly sampled from the training set. Our results show that the length- contth-controlled (LC) win rate that answers of PC being preferred is 58.84. The result shows that the quality of PCllama is higher than that of PCoff-
D ILLUSTRATING THE RELATIONSHIP BETWEEN TEXT DISTRIBUTION
ESTIMATION ($ (55.2 AND ALGORITHM ($5.))
We illustrate the relationship between the general text distribution estimation and our purposed boundary measurement algorithm in Figure
Numerical Calculation Distribution is Intractable
Goal: Estimating General Text Distribution Bg
1. Build Connection between Text Distribution æ and Preference Distribution IP (Definition 5.3)
2. Show и and form pair of Bijection (Theorem 5.4) ---- Then estimating IP instead of ı
3. Measure Distance between P, and P2 by Preference Consistency (Definition 5.5, Equation 9)
4. Compare Preference Preference Consistency between 1 + Pg, P +> Pof (Equation 10, 11)
5. Practical A pproximation (Appendix E.1)
Solution: Boundary Measurement Algorithm (Algorithm 1)
Figure 4: Illustration of the relationship between the general text distribution estimation and our
boundary measurement algorithm discussed and $5.3 The boundary measurement algorithm is derived from preference consistency measurement. The preference consistency measurement is purposed for estimating the consistency between two preference distributions, which are defined as proxies towards the intractable text distribution.
22

---

Published as a conference paper at ICLR 2026
E PROOFS AND DEVIATIONS
E. PROOF OF THEORRM
Proof. Eq. (5) shows that given any reward model Té. there is unique policy aa that ne is the optimal solution under Eq. (3). Then, we prove that given any policy ne, the corresponding reward model is unique, too.
Given Ty as the optimal solution and Tzel is fixed, we can transform Eq. (5 into:
To(y|æ)) B log (
fz, y) v) B log Tre((y(z))
(13)
where f(x, y) is always equals to zero. For some given z0, yo. we rewrite f as function of (z vo):
fzo.yo (20. yu))
= =ro vo) - B To (volzo)) log Z (ara).
(14)
Tref (30 (mo)
Let (To, yo) be an independent variable with range R. we can calculate the partial derivative of with respect to ra (200 yo):
0 So (ro( yo))
Ör (xo. yo)
Or (20, y0)
OZ(x))
Or (zo, yo)
((zo) Ör (Za, yo)
eepp( ( (TO,
Z(zo) Thet (Bul
(15)
@ra(zo, yo)
ne (yor(yo/ (n)) exp( ( un))
Ör (zo vo) Ör ( yo)
(1
(
Fea (vol exp( J (2o. yo))
(
The partial derivative of f with respect to (2 8o) is always greater than or equal to zero. Due to its monotonicity, there at most one value n (r Ho) that can satisfy f(xo, Bo) = 0. ßee is not 0 for any y * yo), then the range of a one-hot distribution (i.e., Tref (yo (r)) = and The R because the domain of r's R, there will be an (T yo) that satisfies [( yo) 0. In other words, for any given Te, there exists rs that satisfies Eq. (5), and completes the proof of Theorem 5.1
E.2 PROOF OF THEOREM 5.2
Proof. Let P(11,82, M2, E ( [0,
the generalized form of preference that 3/1 preferred than y/2
given prompt First of all, we prove that the optimal solution of Eq. (6) satisfies for each
(y1 U2. V2. x). Eq. (6) can be rewritten into the following (I, y1, 822 ~ D, we have (P U2, 2) = (y)
format:
min E ( Dai (P (v1, U2, x) (v1 [2,
(16)
Given that the KL divergence between two Bradley-Terry (BT) models has an exact calculation, implies that the optimal solution for each preference pair in D satisfies Po(y), U2, 12, F (y), x). However, we will demonstrate that IP, = Jp* holds only under the assumption of infinite data. Suppose that Pe is the optimal solution of Eq. (6) obtained from dataset D. For any sample (I, y1, 3/2 ~ D, the optimal solution ensures that Fe > 3222) = 1 (y1 > y2(z). Conversely, for any (x. y1, U2) DD' where D' nD o, there is guarantee that this equality persists, as Ip* is constrained for such out-of-distributtin samples. Nevertheless, under the infinite data assumption, D achieves full coverage of the sample space, making D' an empty set. Consequently, F, P holds for any (I, y1 y2), which completes the proof of Theorem 5.2
23

---

Published as a conference paper at ICLR 2026
E.3 PROOF OF THEOREM 5.4
Proof. We can rewrite the equation in Definition 5.3 with the following form:
Pd (yn > y2(œ) = allog
(17)
xG(22la))
Let A' be the state space and A be the action space, define f(z, 31) X' cocycle that for each (ar, y1, 82), the following equation holds:
A - R be the
(v1
f(, V1 v2) (1/2z)
(18)
Then f is fixed function given me. then prove that wy which satisfies Eq. (18 does not exist unless va mg. Without loss of generality, assume there exists that satisfies
(31
(xr v1, (V222))
(19)
which is equivalence to
To(U1 (x) f(x, v1, y2 To
(20)
Let y/2 be a static point that has a specific value, sum yn on both sides of the equation, we have
Σ T0(31
Σ f(z f1 v2)
(21)
21
311
Since To is text distribution, we have Ey To(()x)) 1. Substitute the equivalence into the above equation then simplify the above formula, we have
To (x) Eyn [(x, V1.
(22)
The right hand side can be accurately calculated since the function is determined. The left hand
side, which is Te (32 (r), can be uniquely determined. And thus we prove ( Applying the result all y2, we have Ty = TG. and completes the proof of Theorem 5.4
na (12 (r).
F FURTHER EMPIRICAL ANALYSIS
F.I REASONABLENESS OF THE DISTINCT ASSUMPTION
In this section, compare the sampling probability between on-policy preference candidates and off- off-policy preference candidates. Since Woff is intractable, verify o 1²(=)]] 1 and extend the result I[F ot (ví > v [>]] O. Specifically, we sample 2, 000 prompts from Ultra- Feedback, as well as their corresponding off-policy preference candidates and their corresponding on-policy preference candidates. For each prompt, we compare the sampling probability between one off-policy preference candidate and one on-policy preference candidate by performing lan- guage modeling task using the corresponding policy. As for each prompt, we have two off-policy preference candidates and two on-policy preference candidates, time, then performing a macro average and report the final win rate. The win rate calculated as on-policy preference candidate having higher probability than off-policy preference candidate for all the initial policy we used in our previous experiments. We provide the comparison results in Table 8 The results show that, compared off-policy samples, initial policies assign higher proba- bilities to the on on-policy candidates in all cases. Notably, the win rate is 84.3% ~ 96.5% for different models, indicating that our assumption is reasonable in most cases.
perform four comparisons each
G FURTHER VISUALIZATION RESULTS
G.1 SYSTEM PROMPT OF GPT-4 EVALUATION IN ALPACAEVAL
We follow the standard recipe of the authors of AlpacaEval, where the system prompt illustrated in Table
24

---

Published as a conference paper at ICLR 2026
Iter-1
Iter-2
Win Rate
Llama-3-8B-Inttre
91 .6
PCon
93.97
PCon
91.11
Zephyr-7B
88.80
PC
89.56
PCon
96.50
Phi-2-2.7B
86.96
PC.
84.32
PCon
85.89
Table Results of the comparison between the sampling probability between PC and PC for different initial models. The win rate getting close to 1 shows that the initial policies assign higher probabilities to on-policy candidates.
G.2 CASE FOR ALPACAEVAL
We provide case from the A AlpacaFval generated by Zephyr Table TO Though this case neither cherry-picked nor lemon-pio picked, it is not randomly selected as we choose this case by its relatively short prompt length and generation length for better visualization effect.
25

---

Published as a conference paper at ICLR 2026
<limsstart >>system
You are a highly efficient assistant, who evaluates and rank large Language models (LLMs) based the quality of their responses to given prompts. This process will create leaderboard reflecting the most accurate and human-preferred answers.
<lim end
<limsstart start
require leaderboard for various large language models. I'll provide you with prompts given to these models and their corresponding responses. Your task is to assess these responses, ranking the models in order of preference from human perspective. Once ranked, please output the results a structured JSON format for the makopartial..ccderbbb. function.
# Prompt
"instruction";
""([nstruction)"""
Model
Outputs
Here are the unordered outputs from the models. Each output is associated with specific model, identified by unique model identifier.
"model": "m" "output": ".* (output 13" 3"""
} "model": "M",
"output": "** """(output22)""
## Task
Evaluate and rank the models based on quality and relevance of their outputs. The ranking should be such that the model with the highest quality output is ranked first. 

Table Prompt for GPT-4 evaluation the Alpaca Eval configuration file. Texts in blue can to be substituted by the real data.
26

---

Published as a conference paper at ICLR 2026
Instruction
Complete the sentence by providing appropriate word. She was wearing dress.
Reference Output
The word "red" would an appropriate word in the blank the sentence "She was wearing dress."
Zephyr (SFT)
Red
Zephyr (PCoft)
floral (adjective) dress (noun)
Confidence: 95%
Zephyr (PCon)
Cocktail dress. She wearing cocktail dress.
Zephyr (PCoff-ofl)
Flowery (if the dress had floral designs)
Red (if dress was color red)
3. Sequined (if the dress had sequins)
4. Lace (if the dress was made of lace)
5. Vintage (if the dress
from older time period)
Off Off-shoulder (if dress had an off sfoshhulder design)
7. Maxi (if the dress was very long)
8. Little black (if the dress was a simple, black, versatile dress)
9. Bohemian (if the dress had loose, flowing style)
10. Cocktail (if the dress was appropriate for formal event)
Zephyr (PCoff
Flowery: She wearing **floral** dress with intricate patterns of blooming flowers.
2. Sequined: She was wearing **sqquined** dress adorned with sparkling rhinestones and tiny beads.
(omit for 6 lines)
Bold-printed: She wearing **hold-printed** dress featuring striking, eye-catching pattern, such stripes, polka dots, or abstract designs.
10. Off-shoulder: She was wearing an **off-shoulder** dress a dress style with the neckline falling off one or both shoulders, often featuring a halter-style strap or spaghetti straps.
Hope these suggestions help! Let me know you need more context specificity.
Zephyr (oi)
She was wearing fowyairysheerddaphanoss (adjectives indicating a light and transparent fabric) maxi dress.
(Alternatively, could also use: She was wearing chiffonsilkllaceorganza dress, depending the specific fabric used.)
Zephyr (PCon-oo) Cocktail dress
Alternatively:
Little black dress (if it was black, form-fitting dress suitable for a formal or semi- semi-formal occasion) (omit lines)
A-line dress (if skirt flared out from the waist in triangular shape)
Table 10:
Responses generated by Zephyr-7b under different training iterations and trained with different preference data. We omit the outputs of Zephyr (PCoff-von) and Zephyr (PCon-voft).
27


Reviews:
[{"review_id": "zr5kxv2qNQ", "reviewer": "Reviewer_cNBy", "summary": "The submission proposes an qualitative breakdown of the preference optimization into the preference injection and preference fine-tuning stage, based on varied effectiveness of on- and off-policy preference data usage on different models. Authors validate the proposed breakdown on multiple models and datasets, and propose a quantitative boundary area measurement to determine the preference stage a given model and preference dataset pair is in.", "questions": "- What are some practical applications of the propositions made in the paper, when it comes to deciding between on- and off-policy data for preference optimization?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Taking inititative on the varied results we see in the preference optimization literature is good to see. The breakdown of the preference optimization at least makes sense qualitatively, if not theoretically\n- The boundary area measurement provides a single quantitative metric that really distills the core of the paper\n- The research questions are validated on multiple model families", "weaknesses": "- While the boundary area measurement is a good starting point, I still think the proposition given in the paper lacks practical application. The boundary area measurement is easy to calculate in retrospect, after we have the models and all versions of the preference data, but doesn't seem so easy when decisions need to be made on-the-fly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jFGNz9BGO8", "reviewer": "Reviewer_NSmT", "summary": "The authors finds that On-policy data for alignment is not always optimal and attempts to explain it with dividing the alignment stage into a \"preference injection\" stage which requires data coverage (diversity) and a preference fine-tuning stages (which requires high quality data). The authors verify their findings on different models and alignment algorithms (DPO, SLiC-HF).", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper investigates the learning dynamics of preference learning, which is understudied. The findings are novel and could be useful to practitioners who are working on human preference learning. \n\n2. The paper is well written with comprehensive experiments demonstrating the generalizability of their findings. The finding that off-policy DPO followed by on-policy DPO on two iterations provides a simple recipe for people to try out. \n\n3. The discussion between on- v.s. off-policy data could be extended to many other settings that involves Language Model post-training.", "weaknesses": "1. It seems that people are not really excited about DPO / RLHF anymore. For example, the latest open-source frontier models (Qwen3, GLM 4.5, Kimi-K2...) only adopts a RL process using rubric rewards and verifiable rewards on math / coding tasks. This is a good and interesting paper, it is just that I don't know how much impact would a DPO paper make in 2025. It would be better if the authors can show  a curriculum of on- and off-policy data also generalizes to RLVR experiments.\n\n2. The findings are somewhat validated on, in my understanding, weak models. (e.g. Zephyr-7B and Phi-2-2.7B). I don't know if the base model is stronger (e.g. Qwen3 series) how much the findings would still hold. It seems to me that the on-policy data is not good simply because that some of the reference model were not good enough to produce reasonable outputs, so we need to reply on off-policy good demonstrations to perform alignment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3Tyn3NMzjS", "reviewer": "Reviewer_j12w", "summary": "This paper investigates when and why on-policy preference data (generated during training) improves or harms language model alignment compared to static preference data. While previous work such as DPO and SLiC-HF assumes that on-policy sampling always benefits alignment, the authors empirically find that this is not universally true, where performance varies systematically across models (e.g., Llama-3, Zephyr) and alignment stages. To explain these findings, they introduce the alignment stage assumption, distinguishing between a preference injection stage that benefits from data diversity and a preference fine-tuning stage that benefits from data quality. They provide both theoretical analysis and empirical evidence supporting this two-stage perspective, along with an algorithm to identify the boundary between stages. Experiments across multiple model families and alignment methods demonstrate the generality and practical usefulness of this framework.", "questions": "Please refer to weaknesses part.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.  The writing is clear and easy to follow.\n\n2. Research on the problem of using on-policy data for DPO is interesting.\n\n3. The authors have great literature review for related works.", "weaknesses": "1. I have some concerns about models to measure diversity. As mentioned in the paper, the authors use Zephyr to measure diversity. To further confirm this conclusion, it's better for authors to see whether this diversity patterns are similar for different models.\n\n2. This method can be used to recognize the training stage of LLMs. Does this method have great potential to make DPO training more efficient (like convergence speed)?\n\n3. I find that the authors conduct some experiments on DPO and SLiC-HF. However, some of iterative DPO algorithms which use on-policy data have different loss objectives as DPO or SLiC-HF. Does this method still work on that iterative DPO algorithm like [1]?\n\n[1] Self-play preference optimization for language model alignment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TFwpX4d5FN", "reviewer": "Reviewer_fQXn", "summary": "This paper investigates whether on-policy data is always the optimal choice for DPO-based language model alignment. The authors show that the effectiveness of on-policy data varies significantly across different model architectures and scales. To explain this, they propose an alignment stage assumption that divides the alignment process into two distinct phases: the preference injection stage, where diverse data is more beneficial, and the preference fine-tuning stage, where high-quality data is preferred. The authors further propose an algorithm to identify the boundary between these two stages.", "questions": "- In Table 2, is the diversity of PC_on higher than that of PC_llama?\n\n- Could you briefly summarize the purpose and contribution of Section 5.1?\n\n- Could the boundary measurement algorithm be used to adaptively schedule on-policy and off-policy sampling in large-scale industrial alignment pipelines? I found that the computational complexity of the proposed algorithm appears relatively high.\n\n- The paper mainly evaluates on AlpacaEval 2.0 (length-controlled) — have you tested whether similar trends hold on MT-Bench, Arena-Hard, or other evaluation suites?\n\n- In practice, high-quality data often has low diversity. Have you explored data blending strategies to balance these two properties?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper provides the important observation that on-policy data can sometimes degrade alignment performance.\n\n- The authors empirically demonstrate that the effectiveness of on-policy versus off-policy data varies across model scales and architectures, offering nuanced insights into LM alignment dynamics.\n\n- The notion of preference injection highlights the mid-training importance of data diversity.", "weaknesses": "- Some theoretical sections (especially §5.2–§5.3) are mathematically dense and could benefit from more intuitive explanations, visualizations, or pseudocode examples.\n\n- The boundary measurement algorithm is under-emphasized. It would be better if it were moved from the appendix to the main text.\n\n- The paper could include more comparative experiments across different model families in the **main body** to validate the proposed assumption and algorithm further.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)