Keep the Best, Forget the Rest: Reliable Alignment with Order-Aware Preference Optimization
Authors: Jiahui Zhu
Yuanjie Shi
Xiyue Peng
Xin Liu
Yan Yan
Honghao Wei
Venue: ICLR 2026 Poster (2026)
Keywords: Language Models; Preference Optimization; RLHF
Primary area: foundation or frontier models, including LLMs
TL;DR: We present RAPPO, an order-aware preference optimization framework that achieves tighter generalization guarantees and outperforms DPO baselines on multiple LLM tasks.

Abstract: Direct Preference Optimization (DPO) has emerged as a powerful framework for aligning large language models (LLMs) with human preferences via pairwise comparisons. However, its performance is highly sensitive to the quality of training samples: when the reference policy is poorly aligned with human preferences, ambiguous pairs can dominate the gradient signal and degrade generalization. To address this, we propose RAPPO($\textbf{R}$eliable $\textbf{A}$lignment for $\textbf{P}$reference $\textbf{P}$olicy $\textbf{O}$ptimization), a simple sample-aware modification of the DPO loss that mitigates reference-policy misalignment by filtering out the hardest, most ambiguous samples. We theoretically show that RAPPO yields improved generalization guarantees. RAPPO is lightweight and requires only a few lines of code to be integrated into any existing DPO-type algorithm. Surprisingly, With this simple modification, our simulations across a broad suite of alignment tasks and benchmarks show consistent gains over DPO and recent state-of-the-art baselines. On the PKU-SafeRLHF benchmark, RAPPO attains helpfulness $0.693$ ($+34.8\%$ over DPO) and harmlessness $0.357$ ($-21.0\%$ vs DPO).

Published as a conference paper at ICLR 2026
KEEP THE BEST, FORGET THE REST: RELIABLE ALIGN- MENT WITH ORDER-AWARD PREFERENCE OPTIMIZA-
TION
Jiahui Zhu', Yuanjic Shi', Xiyue Peng", Xin Liu2, Yan Yan', Honghao Weit
School of Electrical Engineering & Computer Science, Washington State University
2 School of Information Science & Technology, ShanghaiTechh University
[jiahui zhu, yuanjie. shi, yan. yan1, honghao. wei @wsu. edu (pengxy2024, liuxin7) @shanghaitech. edu.cn
ABSTRACT
Direct Preference Optimization (DPO) has emerged powerful framework for aligning large language models (LLMs) with human preferences via pairwise com- parisons. However, its performance is highly sensitive to the quality of training samples: when the reference policy is poorly aligned with human preferences, ambiguous pairs dominate the gradient signal and degrade generalization. To address this, we propose RAPPO (Reliable A Alignment for Preference Policy Optimization), simple sample-aware modification of the DPO algorithm that mitigates reference- pearreccc--llcyyy misalignment by filtering out the hardest, most ambigu- ous samples during training. We further theoretically show that RAPPO yields improved generalization guarantees. RAPPO is lightweight and requires only few lines of code be integrated into any existing DPO-type algorithms. Surprisingly, with this simple modification, our simulations across a broad suite of alignment tasks and benchmarks show consistent improvements over recent state-of-the-art baselines. Our code is available at https://gituub.com/px WaterMoon/rappo.
INTRODUCTION
Large language models (Achiam et (2023)) have achieved remarkable performance across a broad range of complicated tasks, such as abstractive summarization (Stiennon et al. (2009); Koh et al. (2022)), code generation (Chen et al. (2021); Gao et al. (2023b)), and instruction following (Chung et al. (2024); Ouyang al. (2022)), demonstrating the abaility of strong generalization across domains. A key challenge, however, lies in aligning these models with nuanced human intentions. Reinforcement Learning from Human Feedback (RLHF) has become widely adopted solution to this problem (Christiano et al. (2017); Ouyang et al. (2022); Dong et al. (2023); Yuan et al. (2023); Liu et al. (2023a)), enabling L.MSs to better reflect both stylistic preferences and normative values (Bai et al. (2022); Ganguli et al. (2022)). RLHF typically involves training a separate reward model score outputs, which is then used guide policy optimization via reinforcement learning.
To streamline this process, Direct Preference Optimization (Rafailov et al. (2023)) (DPO) has been proposed as alternative that directly learns from pairwise preference data, bypassing the need to construct an explicit reward model. Building on DPO, several variations improve optimization stability and performance by: adding an explicit KL regularizer to control policy drift (IPÓ; Azar et al. (2024)); introducing learnable logit offset to correct unknown preference margins and reference miscalibration (DPO-offset; Amini et (2024)); employing weighting and positive-only learning (KTO; Ethayarajh et al. (2024)); and pushing preference training to the token level for finer credit assignment and reduced length/verbosity bias (token-level DPO; (Zeng et al., 2024). DPO outperforms supervised fine-tuning (SFT) significantly due to the ability to distinguish the quality of multiple candidate outputs rather than relying on a single gold reference. From the perspective of reinforcement learning with a KL divergence penalty, the DPO objective =u(y)=r)
asymmetric
induces an implicit reward function ra y) = ,9 log ntwe(ylo) where To denotes the policy being fine tuned, Ree is the fixed reference policy, and 3 > O controls the strength of the regularization toward the reference. Clearly, the performance of the output policy no is closely tied to the reference

---

Published as a conference paper at ICLR 2026
(A) Referance Model Forfarmance
(B) Reward Szores
2000
1.6
Semple Count
800
1.3
200
11
-mmmll
GPI-med
Gar-erge
1.0
GPT-smell
GPI-med
GPT-large
Figure 1: (A) Reference model performance across three SFT models (GPT2-Small, Medium, and Large). Correct: the reference policy aligns with human preference. Wrong: the reference policy conflicts with human preference. (A) shows that misaligned data are frequent regardless of model size, though the proportion decreases as model size increases. (B) Reward scores on the IMDb experiment (Section 5.1) using DPO and RAPPO under different reference policy scales. Performance declines as the reference policy weakens. Nonetheless, with simple modification to DPO, our method RAPPO improves performance by 3.5%, 1.1 %. and 7.1 % across the three models.
policy. In fact, the effectiveness of DPO strongly influenced by the distribution shift between model outputs and the preference dataset Tang al. (2024). More specifically, in the reference model(shown in Fig. 1) can make it nearly impossible for DPO and its variants to recover the correct preferences Chen et al. (2024a). Recent work has further shown that updating the reference policy can improve DPO performance Gorbatovski et al. (2024).
mild alignment errors
Motivated by these observations, recent work has pursued two broad directions. (i) Data selection. RSO (Liu et al., 2023a) fine-tunes only on high-reward samples; Deng et al. (Deng et al., 2025) maximize reward margins suppress noisy supervision; and Selective DPO (Gao al., 2025) filters ambiguous preference pairs prior to training. (ii) Reference-free training. ORPO (Hong et al., 2024) replaces the -efcrence-anchored objective with maximum-likeline likelihood learning on positives examples, while SimPO (Meng et al., 2024) emphasizes length- nonggh-hommlized sequence likelihood and discards the reference policy entirely. Although reference-fres approaches can avoid propagating harmful signals from misaligned reference, they also relinquish potentially useful guidance and risk catastrophic forgetting (Luo et al., 2023b; Huang et al., 2024; Zhang et al., 2022). A more detailed related work can be found in Section 2. These observations, together with empirical findings from RLHF pipelines (Ouyang et al., 2022; Peng et al., 2024), demostrate that DPO performance hinges not only on the choice of reference policy but also how well the resulting training samples align with true human preferences. In this work, we seek to advance a systematic understanding of the reference policy's 's role in DPO and ask the following central research question:
Can a simple, sample-aware modification to DPO mitigate reference-policy misalignment and thereby improve generalization?
We provide a theoretical treatment of this question in this work. Our approach is inspired by the selective-update principle in Ordered-SGD (Kawaguchi & Lu, 2020), which updates model parameters by using a selective approach. This principle prevents the optimizer from reinforcing incorrect preferences and focuses learning on informative examples. Building on this idea, we introduce RAPPO (Reliable Aligmment for Preference Policy Optiiizttoon, reference-awan variant of DPO that improves generalization by dynamically filtering out untrustful preference pairs. As shown in Fig. 2, by sacrificing some performance in regions where the reference policy performs poorly, RAPPO focuses on areas where the reference is well aligned with the offline dataset. This targeted emphasis leads to significant overall improvement over DPO. The key contributions of our approach are summarized below:
Reliable Filtering: At each optimization step, RAPPO detects samples where the reference model misalliances human preference signals and discards a fixed number of the most untrustworthy pairs in the batch, those most ambiguous, thereby focusing learning clear and reliable preferences.
Simplicity and Compatibility: The method lightweight and easily integrates into any existing DPO-type algorithms with only few lines of code modification.
Theoretical Guarantee:
We provide theoretical analysis showing that unreliable reference policies in DPO can enlarge the generalization gap. and prove that RAPPO, by order-aware sample selection, yields tighter learning bound than standard DPO.
2

---

Published as a conference paper at ICLR 2026
Win-Loss Probability Gaip Across Offiine Dataset
0.04
RAPPO
0.03
DPO
Reference
710 |x] ---nnyy
Performance gap
0.01
0.00
-0.01
-0.02
-0.03
230
430
600
800
1030 1200
Offinn Dataset
This figure compares the win-loss probability gap (the difference between the probability assigned to the dluann-peeeeree with RAPPO and DPO on the Toxicity dataset (Section 5. 5. Samples are ordered by the reference policy's preference gap, so the left side corresponds cases where the reference is most decisive. Tie: instances where the model assigns equal probability to the preferred and rejected responses. In the shaded region, RAPPO yields noticeably larger gap than DPO on good-aligned samples, while incurring only minor trade-off on less well well-aligned cases.
Figure 2:
response and that of the rejected response) for models fine- funed
Strong Empirical Performance. RAPPO achieves consistent improvements across four challeng- ing tasks, IMDB sentiment classification and toxicity control (Maas et al., 2011; Gehman et al., 2020), text summarization, and safety alignment (Ji et al., 2024). On IMDB, it outperforms the strongest baseline by up to 7.7%. For toxicity control, it reduces toxicity to as low as 2.28% (vs. 6.30% for the best baseline). In summary, RAPPO surpasses SIMPO and DPO under GPT-4 evaluation, achieving win rates (defined in Section 5.2) of 58.8% and 74.5%, respectively. On the PKU-SafeRIHHP benchmark, it consistently outperforms all baselines across four metrics (see Fig. for details).
2 RELATED WORK
RLHF vs DPO. RLHF(Peng et al. (2025); Christiano al. (2017); Ziegler al. (2019); Ouyang et al. (2022); Askell et al. (2021): Bai et al. (2022); Dong et al. (2023); Kaufmann et al. (2023); Yuan al. (2023); Liu et al. (2023a); Peng et al. (2024); Zhou al. (2023); Chen al. (2025); Zhang et al. (2024); Geng et al. (2023); Conover et al. (2023); Köpf et al. (2023); Ding et al. (2023); Wang et al. (2023); Xia et al. (2024); Gao al. (2023a); Luo et al. (2023a); Chen al. (2024b); Lightman et al. (2023); Lambert et al. (2024); Anthony et al. (2017) 7); Schulman et al. (2017); Shao et al. (2024)) offers practical route for bringing large-scale language models in line with human judgments and values. DPO(Rafailov et al. (2023)) was proposed to bypasses the reward modeling stage while still achieving competitive performance. Recent DPO studies can be divided into the following aspects.
1). Data Strategy To mitigate alignment errors in DPO, many researchers now focus selective data sampling. For example, Liu et al. (2023a) suggests choosing high-reward samples during the fine-tuning process. Additionally, Deng al. (2025) uses margin-hased rule to build better datasets. Similarly, Yang et al. (2024) recommend ranking reward margins to find the most useful training pairs. Another study, Selective DPO (Gao et al. (2025)), suggests removing confusing or difficult samples before training starts. Furthermore, Liu et al. (2023b) shows automatic ways to select data to improve alignment. But these methods only focus on filtering data. They do not address the problems found within the reference model itself.
2). Reference Consideration DPO. Recent studies question if the reference policy in DPO-based fine-tuning is reliable. Some reference- ffree methods, such as SIMPO, contrastive-style losses, and SIMPER, remove the reference entirely. These methods show better results on many benchmarks (Meng et al., 2024; Xu et al., 2024; Hong et al., 2024; Xiao et al., 2025). In contrast, Liu et al. (2024b) argue that a reference model is still necessary. From a theoretical perspective, Chen et al. (2024a) show that even small emrors in the reference can make it very hard for DPO to work correctly. To find a middle ground, Chennakesavalu et al. (2024) introduce a DPO-like objective with a tunable parameter. This parameter controls how much the reference influences training. Together, these findings show the risks of using a fixed reference. They also show that we still do not fully understand when how much to use the reference model.
3

---

Published as a conference paper at ICLR 2026
3 PRELIMINARIES
Notations. Denote I as prompt. T SFFT is (pre-trained) supervised fine-tuning model to produce a pair of answers (y1, 322 t T (ylx). These response pairs are then presented to human labelers, who express preferences by selecting one response as better. Let yo and y denote the preferred and dispreferred responses, respectively. We define the underlying data distribution P over triplets 2 = (xt Mro U1) where the preference from labeler is denoted as yo > y J.. Denote D = {=') i=1 = {(x', Vw y/ )=) i DD be a dataset of N i.i.d. samples drawn from D. Let 'ee denote the reference policy (here, we use "re == >. Let Ty be the fine-tuned policy for human alignment based on Tref parameterized by 0, where 0 belongs to a hypothesis class θ (e.g., neural networks). For any policy т (e.g., Teer or Tœ), let (yw xx) and (yy a)) denote the likelihoods of the preferred and dispreferred responses given input 2, respectively.
In RLHE, a language model represented as a policy ι that generates a response y E y conditioned on prompt E X. following the distribution r(-|2)). The typical RLHF pipeline involves four main stages: supervised fine-tuning, collection of preference data, reward model training, and reinforcement learning baased policy optimization. The process begins with supervised fine-tuning, where a pre-trained language model is adapted into an initial policy π SFT using labeled data through standard supervised learning. Subsequent stages refine the model using human proxy feedback. Specifically, in the preference data stage, an oracle (such as a human annotator or a heuristic scoring function) compares two candidate responses 81 and y/ for the same prompt æ, indicating which one is preferred. A common method for modeling such pairwise preferences is the Bradley-Terry model (Bradley & Terry, 1952), which estimates the probability of one option being favored over another:
exp (r, v1)) P(y > y2|x) exp (r.(z, (1)) + exp (r* (x, y2)) = (x, v1) r U2)),
where r" : x y - R the unknown ground truth reward function, and a - (0, 1) is the logistic function. Given human labeled preference dataset D {< Vou yf ) composed of independent samples, the reward function r can be estimated by maximum likelihood estimation (MLE): PMLE = arg min, ((r, D), where E(r, D) is the negative ligeliilli of D, given as E(r, D) = - E(x+ WftED log (r(x', Vim)) r(='.((' v()).
RL fine-tuning. Given reward model r, the fine-tuned policy is optimized to achieve ideal balance between the expected reward and its distance from an initial policy nei, which is typically the same as x SPT This is achieved by maximizing the KL-regularized value function [r, T)) defined as
[r. T) = (e [r(z, v)] - [ KL (T(()
(2)
where KL(TI T2) is the KL divergence from m to T2, and B > 0 is a regularization trade- off. Thus the RL fine-tuned policy π, with respect to the reward r aims to optimize [r, T). i.e. Tr = arg max, (rr T).
Direct Preference Optimization (DPO). In RLHE, DPO gives a closed-form solution derived by minimizing the following objective function:
log (B (y x))
To (y xx)
(oro n = - (a. Pup
B log
(3)
(re xx)
Teat (31 1T)
The empirical version of the DPO objective then follows:
Eopo(TT (mTH Teer)
1
Σ
[- log (B(Ao(z') - Aror(=
N
where Ao(2") log To (vig) log To (x))))), and Aie((2)) log नee Tee (vig log गien ( lz')
Thus DPO solves ĀDPO arg max EDPO (T; Tref).
4 RELIABLE ALIGNMENT FOR PREFERENCE POLICY OPTIMIZATION
Introduction and Motivation. We propose RAPPO. an aware variant of preference opti- mization designed to mitigate the sensitivity of DPO to unreliable reference policies. The idea is
4

---

Published as a conference paper at ICLR 2026
4
Batch Data
Aligned Dara Unail gred Doni
Algnnd Data Liraligeed Data
3
Nlgned
bugar leo
Figure RAPPO Pipeline:(1) sample mini-batch data; (2) score each mini- batch by reference alignment, splitting samples into Aligned and Unaligned; (3)unaligned samples are ranked by per- sample loss; (4) temporarily remove some Largest ones for this update.
inspired by ordered SGD (Kawaguchi & Lu, 2020), which emphasizes high-loss samples am- plify signal. When the reference policy provides misleading preference signals, not all samples are equally informative, some add noise or even drive the model the wrong direction. To address this, RAPPO selectively discards fixed number of high-loss, ambiguous preference pairs in each batch, which focuses on variance control, our method explicitly targets misaligned examples arising from reference-poliey errors, ensuring that the update direction remains both stable and aligned with true preferences. This motivates our main theorem, which shows that RAPPO achieves a tighter generalization bound than standard DPO. We first introduce our algorithm and then theoretically show our main results.
4.1 MAIN ALGORITHM
Our algorithm is lightweight and can be seamlessly integrated into any DPO-type method with only a few lines of code, yet delivers substantial improvements. The high-level idea is illustrated below, and the full algorithm is provided in Algorithm I.
Step 1.
Sample mini-batch B from the dataset (Stage 1 in Fig. 3).
Step 2. For each z (x", yln vi) E B, compute the reference-alignment score
(a (x') Tor only and
partition the batch into Aligned (As = {a E
B
Merlly le') (ar (2r)
+}) and Unaligned
Mar(yi 124) S T}) subsets. Also compute the per-sample loss (, = -logo (((o((2))))) (x=)( (Stage in Fig. 3).
(Us = f= B
T lar')
Step 3. In the Unaligned subset Ugg sort samples by (0) in ascending order (Stage 3 in Fig. 3).
Step 4. Exclude the Top-q Largest-loss unaligned samples only in the unaligned subset Ug from the batch, and update the model using the retained data. (Stage 4 in Fig. 3).
Remark 4.1. RAPPO never permanently removes samples; even unaligned examples may be selected. Because selection within Ug depends on the evolving loss (0), samples that were once "untrust- hard" can become "untrust-casy" as the model improves, naturally re-entering the kept set Sz(g).
This induces a reference-aware, model-adaptive curriculum: the reference score Turly lar')
(ar((/]12*) gates the
risk from misalignment, while the instantancous loss (f( refines which unaligned signals are safe to incorporate at the current stage of training.
4.2 MAIN THEOREM
Now, will show main result of RAPPO. We will begin with general optimization framework in which RAPPO arises as a special case, and present a main theorem showing that, compared with DPO, it achieves a larger expected first-order decrease in population risk, reduces the variance of gradient updates, and yields a tighter stability-based generalization bound. Collectively, these results formalize that discarding high-loss, misaligned samples enables more reliable progress toward human aligned preferences.
Notations: Let (Z, be the example space and θ C Rp the parameter space. We are given a dataset
D (Z1 ...... Zn) with Zi...., Zn i.d. D, for some date distribution D on (Z, A). At iteration t, a batch S, = [Zr, Ze,r) is formed by sampling s indices uniformly from {1,... n.. We define the loss I, O 2 R which is measurable in and differentiable in 0 for any given sample.
5

---

Published as a conference paper at ICLR 2026
Let g(0:2) == VgL(0;2). Define the (single) population risk (expected error) and its gradient as:
R(0) := Ez=Dl (0; ). d(0) Vg)(0) Ez~pl g(0: 2) Assumption 4.2 (Smoothness and Lipschitz). Assume for some L, G 2 and all 2, 0, 8, (lg(0; 2) g 2| < L - 0'1l, [L(0:2) - 2() - G||0 - 0'
(5)
(6)
Then R is L-smooth.
Assumption 4.3 ( Trusted/untrusted split and scores), Assume that we have a fixed measurable
classifier H: 2 (tr, un) which can tag each example. For the batch S. {Z,, define
S," = {ú: (Z.,) = tr), S.n = {i: HZ,,) un), G, = S B, | G. B, =
For each untrusted item i E Stnn define scalar score 2,,, 2(0;; Zi,)) E IR that orders untrusted items. The untrust samples can be regarded as the unaligned samples based on the reference policy.
Weight bound untrusted per-example gradients. Assume there exists a nonincreasing w R (0, 00) and a constant C >> such that for all untrusted Z and all 0,
(0: 2) S (w(22(;; 2))
(7)
Remark 4.4. For DyO-type gradients: 2 = 3(Ao - Aret) and w(2) = a(2)) satisfy equation 7, where Ag log To (Bw x)) log To (y)lz), and Aee = log Thet x)) log The for any given (1r, yos 31)
Let s q (1....,s) {} be the number of kept examples per step. Define K2 == max B, - z)
We never remove trusted items. B, q S O, set q == and keep all items. If B, q > 0, keep all G, trusted items and among Spn keep the K2 largest scores Zt, call this set TV. Thus
S S
(B, S 9, set q :- 0),
Kept, L(S TE
(8)
(B, q).
Remark 4.5. In RAPPO, each untrusted pair has DPO margin and loss {(=) = log (2). with larger meaning smaller loss. Given keep budget s q. we keep all trusted items and, if trimming is needed, we keep exactly K, B, q untrusted items with the largest = (equivalently, the smallest DPO loss), dropping the largest-loss untrusted items.
0 T)t 9:. Then condi- tion on the realized untrusted scores {2,,5 : : STn). Let pa, == Eige,s 20,5] and & := It.s Me. Assume the centered vectors (Er, are conditionally independent given (21,5). Assume there exists nondecreasing Φ : (0, oo) + [0, 20) with tr Cov(gi, z4,)) I )(w(z,,)) Remark 4.6. For DPO loss, (0: w(2)u, w22) == o(-2), == Vo2, with V8z = -3 Vo VologTo(ye Assume bounded-score condition (standard) V/ log To x) I Then |ul|| S 23 and hence |I9 = [| < (23B) w(z.s)) Therefore, for any z, tr Cov 2τ,i = 2) S E [1..,.11 1l2 2,,s = = < 3 So the assumption holds with the explicit nondecreasing choice (uu) = 22 2 u 0, since u + u² is nondecreasing.
Update rule. Let St.s == g(0;; Z...). Define Ôe = = Eiekept, It.s, = -
STm).
- -logo(z) Vol-Dro0;; = -o(-2)
To |x) V log ( x))
Let de == d(e:). Define ae.; = (gr.s. de) and its conditional mean a(2) = E Elar, 21,; 2). Assume a(z) is nondecreasing in z. Define l((z) == w(2) a(z) and assume h nondecreasing z. Now we are ready to show our main theorem, the detailed proof is differed to due to page limit.
Theorem 4.7. Fix iteration with B, > 9 and let C Stnn be any set with [T] = K2, while Ti contains the K, largest 2,, in Spm Conditioned on the realized scores {21,3)
(i) Maximal expected first-order decrease of R.
E
(de, Ic(T))
S E (de, (((T)) [=r.)]
(9)
(ii Minimal conditional variance of the signed decrease. Lei (T) = (-āc(T), d.). Then
Var {xxi)) 2 Var( (2,,1))
(10)

---

Published as a conference paper at ICLR 2026
Algorithm 1 RAPPO
Inputs: an initial policy Tgo, reference policy Tret, learning rate sequence (nd) threshold and hyperparameter
2: for = 1, 2, ..., 7 do
3: Randomly choose mini-batch B and compute the Terll'' la') for all data samples 2 in whole Tar(s/
dataset.
4: Partition the batch into Us = {= E B Merly ln))
[ S J As = f=t EBI
mar lxx)
Tur 2x1) > r}
5: Within the current untrust data sample Ug, order the individual losses ( log (3(Ao(=) - Aer(=1) in ascending order that (G==) (==2) S S ((==-a))
6: Computer the gradient & using kept sample:
Σ (()),
b 4
gt E OÉRAPPO step
=
b-q
1
Σ (0) + Σ ())(0)
ifb>a
s
iEAB
j=1
7: Update parameters 0: 0,41 = 02 719tt end for
8:
(iii) Uniform stability and generalization. Let D (Zi...., Ln) and D' (Z)...., .,...,.,,))))) be two datasets that differ in example, both i.idd from D. Run the same randomness to produce {0,) } and (0%). Define A : El(a, - 8,11- Then for any horizon and stepsizes {m.)
2C
A411 S (1 Lme)
T) E max
w(22,) iEKept,
(11)
q
2C
T-1
T-1
AT I
exp
LMuu
Σ
me
max LEKept,
w m((z,s)
(12)
8 q
==
t=0
By Eq.(6), ET == sup, ([L(Or:2) - L(0(r:2) error is.
G Ar, and then the expected generalization
1 n Σ L(0;
E (((r)) (fr)) S ET,
(O)
(13)
From the results, can observe that 1), keeping the K, items with the largest scores (i.c., the smallest losses) reduces the conditional variance of untrusted contributions, since larger losses correspond larger gradient weights and is nonincreasing in 2), over multiple steps this choice minimizes the maximal kept weight among untrusted items, directly tightening the stability recursion and thus the generalization gap. Both effects hold for smooth, Lipschitz losses without convexity assumptions. 3) Compared with any alternative rule that keeps K₁ untrusted items, our selecting minimizes the per-step term Kep,,, w()),;)] and hence minimizes Ar and ET.
While the algorithm design is simple and flexible, it is also important to derive an objective function. We first define ( = (i) 0) as the j-th smallest loss in unaligned data samples i.e., totally JNN elements, where {(1) S I LeaNy. Particularly, the objective that we optimize can be found in the following proposition.
Proposition 4.8. Let the whole dataset size be N. The number of whole unaligned examples is ng = auN and of whole aligned examples is ng = (1 - N. At each iteration, we sample a mini-batch S C [N] without replacement, |S| = Write the unaligned and aligned counts in the batch as Us. AB For parameter vector 0, the per-example loss is E, or (0) And denote mg = EieAe ((()), my = Ejeus (0). Then, we can obtain the objective function of RAPPO:
mi
min(x.pN)
my +
AN aj
_RAPPO_
Σ
P(Musl mg + mb
Σ 0=q+1
P(UE) bb)
(14)
b=0
keep all data
keep aligned and remove largest loss data nample
7

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of reward scores and toxicity percentages across various preference optimization methods, evaluated on the IMDB and Real-Toxicins exxicity-roompss Gehman et al. (2020) test set. Higher reward scores and lower toxicity indicate better performance. The whole experiments of SIMPO are defered in Table 7.
Algorithm
DPO
DPO-Offset
JPO
SimP0O(3=2.,,--00
RAPPO-1
RAPPO-2
RAPPO-4
Reward Score
1,5513 6.30
1,5526
1,5446
1.5537
1.6625
1.6790
1.6811
Toxicity (%) L
8.11
6,49
7.49
2.64
2.60
2.28
Model A: GPT)-6B
1.0)
Model Lama3.1-8B (t
RAPPO-1 SimPO
58.8%
41.2%
62.6 8%
37 37.2%
RAPPO-1 DPO
74.5%
25.5%
58.3%
41.7%
100
Percentage
Left wins
Right wins
Percentage
Figure 4: Win Rate between RAPPO-1, SIMPO, and DPO by GPT44
where P( ( (())(A)A ANN)
Eis Ei-e-1 ('T ((52)
0g
(20)
Then, Gt. is unbiased estimator of
ACRAPPO that Elg.] E OCRAPPO
Proposition 4.8 give close-form of RAPPO objective function and also show that ğr is unbiased estimator of aL thereby justifying our update rule and ensuring the stochastic updates consistent with minimizing the RAPPÓ objective.
5 EXPERIMENTS
In this section, empirically evaluate the performance of RAPPO in comparison DPO and other state-of-the-art methods across a variety of tasks.
Baselines: We compare RAPPO with state-of-the-art offline preference optimization baselines listed in Table 4. IPO (Azar et al., 2024) directly leverages pairwise preference data without relying on the Bradley-Terry model, helping mitigate overfitting, particularly with deterministic or near- deterministic preferences. DPO-OFFSET (Amini et al., 2024) modifies the optimization objective to enforce reward margin between the preferred and dispreferred responses. SIMPO (Meng et al., 2024) adopts reference-fie freeeeee-eeeeeeee approach and integrates reward margin into the Bradley-Tern Terry objective to enhance separation between responses. For training, we follow the hyperparameter ranges reported in STMPO, where B [2, 2.5] and E (0.3, 0.5, 1.0, 1.2, 1.4, 1.6]. In our experiments, we use E {2, 2.5} and subset of y E {0.5, 0.8, 1.6]. To ensure fair comparison, the same value is used across all basclines and RAPPO within each setting. Additional details are provided in Appendix D.I.
5.1 CONTROLLED GENERATION: SENTIMENT & TOXICITY
We evaluate RAPPO on two controlled generation tasks, positive sentiment steering and toxicity Idluction-usingg the common evaluation table (Table 1). For both tasks, the base LM is first supervised fine-tuned to obtain the reference policy maet- For sentiment, we use the IMDb dataset (Maas et al., 201 1), following the setup of (Amini et al., 2024). A pretrained sentiment classifier provides the reward signal. For toxicity, 2021) and the dataset of (Amini et al., 2024) constructed from REALTOXICCTT/POMPPT (Gehman et al., 2020) by selecting 10,000 prompts with base toxicity 0.3; relative rewards are given by a pretrained toxicity classifier. RAPPO- means we remove the highest-loss samples per batch E (1, 4)). On IMDb, all RAPPO variants achieve reward > 1.66 (best 1.6811), surpassing the strongest baseline (SIMPO, 3=2, 000.5; 1.5611) by 7.7%; even R AAPPO1 (1.6625) yields 6.5%. On toxicity, the best baseline reaches 6.30% %, whereas all RAPPO variants are markedly lower (2.64% % to 2.28% ).. Overall, RAPPO's 's reference-aware pruning consistently improves alignment quality while substantially mitigating harmful generation.
Lage and
GPT-Neo 2 7B (Black et al.,
5.2 SUMMARIZATION TASK
Beyond the IMDb and Toxicity benchmarks, we evaluate RAPPO on a summarization task. Following Amini al. (2024), we adopt GPT-J-6B (Wang & Komatsuzaki, 2021) the supervised fine-tuned
8

---

Published as a conference paper at ICLR 2026
(SFT) reference policy Ted and, to probe scalability and robustness, replicate the protocol with larger, contemporary open-weight model, Llama-3. 1-B8B (Dubey et al., 2024). For each base model, we fine-tune with RAPPO-1 (ours), SIMPO, and DPO. Consistent with observations on IMDb and Toxicity, SIMPO is relatively insensitive to hyperparameters; we therefore reuse its best configuration from those tasks 2.5, -05.).
We evaluate with GPT-4 (Achiam et al., 2023) as an automatic judge, comparing paired model outputs under identical prompts and recording the preference (full prompt in Appendix D.5). As summarized in Figure 4, RAPPO consistently outperforms SIMPO and DPO on GPT 6B; the Llama-3. 1-8B replication exhibits the same ordering, indicating that RAPPO's gains transfer to a larger, more recent pretraining regime. For brevity, we report aggregate GPT-4 win rates in the main text and include per-prompt breakdowns in Appendix D.8.
5.3 LARGE-SCALA EVALUATION ON PKU-SAFERLHF
We evaluate RAPPO on the PKU- RKU-SafeRHHF benchmark (Ji et al., 2024) to assess whether it im- proves answer quality. We compare against state-of-the-art wweference-ppimmizaiie baselines- CPO, KTO, and STMPO-training each method under the same reference policy(Mi strra (ttrrl))B), train/validation split, and compute budget. Decoding hyperparameters (temperature, top-p. maximum tokens, and stop rules) are held fixed across models, and all methods are evaluated on an identical prompt set (Appendix D.7). As model- based evaluation, we adopt the unified reward and cost models, geaver-7b-uni wwwunniiiddrwwwwwaeeee cost. For each prompt, each model generates single response; we then report (i) Beaver-Help t (higher indicates greater helpfulness as scored by the re- ward model), (ii) Beaver-Harm + (lower indicates reduced inferred harm as scored by the cost model), and (iii) Safety Rate 1 (the fraction of responses flagged unsafe by the cost model(costs 0 is safe)). To mitigate judge-specific biases in absolute scores, we additionally employ GPT-4 as a pairwise meta- evaluator: for each prompt, we compare RAPPO outputs against those of each baseline under identical decoding, and report the resulting Win Rate for RAPPO versus DPO, CPO, KTO, and SIMPO.
Results. On PKU-SafeRLHF, RAPPO attains the best performance on all metrics: safety rate 0.573 (absolute +0.014 vs. next-best DPO), helpfulness 0.693 (+334.8% over DPO), harm- lessness 0.357 (-21.0 21.0% vs DPO), and highest win rate 6Fg. 5, details of these results are defered in Appendix D.7). Computational source and training hyperparameters are pro- vided in Appendix D.I.
Safety Rate
Helpfulness
DP0
KTO
SriPo
RAPPO
6 ABLATION STUDY
Harmlessness
Win
6.1 SENSITIVITY ANALYSTS
In RAPPO, two hyperparameters control the strength of the reference-based filtering: the number of removed items from each batch of misaligned samples and the reference align- ment threshold r. We investigate how sensitive RAPPO is to these two values. We conduct a sensitivity study on the IMDb sentiment control task with GPT2-Large. We first fix sensitivity with respect to q. We then fix the batch size to 32 and vary T E {0.8, 1.0, 1.2] for the same set of q. We compare the results with the best baseline SIMPO in the IMDb experiments, which has a reward of 1.5537 with batch size 32.
Figure 5: PKU-SafeRLHF results. RAPPO com- pared DPO, CPO, KTO, and SIMPO under iden- tical decoding and evaluation protocols.
and vary g E 21, 4, 8) for batch sizes 16 and 32 to analyze
Sensitivity to q. Across both batch sizes, RAPPO consistently improves over SIMPO by about 5% to 8% for all tested values of q, which shows that RAPPO is insensitive to the exact choice of q. Table also suggests that simple rule of selecting q is (batch size)/8.
Sensitivity to r. In RAPPO, the alignment threshold 7 is applied to the probability ratio between the preferred and dispreferred answers and decides how confident the reference model must be before

---

Published as a conference paper at ICLR 2026
Batch Size
q 1=1
q =
q 9= 4
q q=8
Baseline
16 (r 1.0)
1.7020
1.7481
1.7333
1.7111
1.6600 (SIMPO)
32 (T = 0.8)
1.6580
1.6720
1.6765
1.6510
1,5537 (SIMPO)
32 (T 1.0)
1.6625
1.6790
1.6811
1.6432
1.5537 (SIMPO)
32 (T -12)
1.6650
1.6805
1.6828
1.6595
1.5537 (SIMPO)
Table 2: Sensitivity results on and 7.
Table 3: Win rates of RAPPO vs. SIMPO under different LLM judges.
Judge
RAPPO win rate
SIMPO win rate
Tic rate
GPT-4 (original)
58.30%
41.70%
0.00%
GPT-4o
68.00%
32.00%
0.00%
DeepSeek-V3
72.00%
28.00%
0.00%
Claude Opus 4.1
58.70%
34.78%
6.52%
Average
64.25%
34.12%
1.63%
filter a pair. Setting T = 1 is natural default, and Table shows that RAPPO outperforms SIMPO for all tested 7 with only small changes in reward. This suggests that RAPPO is robust to the choice of r, and that reasonable settings give consistently strong performance.
The sensitivity results q and indicate that RAPPO is insensitive to the hyperparameters. But designing adaptive rules that learn an approximate optimal from data is an interesting direction for future work.
6.2 PERFORMANCE ACROSS DIFFERENT LLM JUDGES
In the following, we conduct multi-judge evaluation and analyze the consistency of different judges when comparing RAPPO and SIMPO. We compare RAPPO and SIMPO on a summarization task using Llama 3.B model fine-tuned by each method, evaluated by four judges: GPT-4 (Achiam et al., 2023), GPT-4o (Hurst et al., 2024), DeepSeck-V3 (Liu et al., 2024a), and Claude Opus 4.1 (https://www www.wthroppicccommmmmmmmmm www.aaapppcccco/aaaaa///uuueeuue com/claude/opuss) All judges assess the same anonymized and order randomized response pairs and choose either the better summary or a tie. We measured how consistent the judges are with each other by computing pairwise Kendall's between all judge pairs' preference rankings. Kendall's 's 7 (Kendall, 1938) a standard measure of rank correlation, where larger values mean stronger agreement between two rankings. In practice, values above 0.6 are usually viewed as strong agreement and values above 0.8 as very strong agreement.
Consistency of RAPPO's vdnatge across judges. As summarized in Table 3, all four independent judges prefer RAPPO over SIMPO, with RAPPO win rates ranging from 58.30% to 72.00%. The average win rate of RAPPO (64.25%) is substantially higher than that of SIMPO (34. 12%), while ties are rare. The fact that models developed by different organizations and trained with different data and objectives favor RAPPO indicates that the observed gains are not an artifact particular judge such GPT-4.
Inter-judge agreement. quantify consistency across judges, we compute pairwise Kendall's between the preference rankings induced by each judge. The average Kendall's is 0.65 (Appendix: Table 9), indicating strong agreement on the relative ordering of RAPPO and SIMPO. Together with the consistent win rates in Table 3, this suggests that RAPPO's advantages robust to the choice of automatic judge rather than driven by any single LLM's idiosyncrasies.
7 CONCLUSION
We identify key limitation of DPO: its sensitivity to ambiguous samples, especially under weak or misaligned reference policies. To address this, we propose RAPPO (Reliable Alignment for Preference Policy Optimization), an order-aware variant of DPO that selectively filters out high-loss, hard-to-learn samples. Unlike reference-fir methods such as SIMPO, RAPPO retains the reference policy and uses it selectively to improve stability. We show both theoretically and empirically that RAPPO improves generalization and consistently outperforms DPO and recent baselines, enabling more robust language model alignment.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work complies with the ICLR Code of Ethics, No human-subjectt research or animal experimen- tation was conducted. All datasets were used in accordance with their licenses and usage guidelines, with care taken to avoid any privacy violations.
REPRODUCIBILITY STATEMENT
Our method builds on standard DPO, and the full implementation is provided in the Supplementary Material. We also include complete details and links for all datasets and reference models used in our experiments in Appendix D. In addition, provide step-by-step proofs designed to be clear and easy to follow in Appendix C.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint (xXxx:2333008774 2023.
Afra Amini, Tim Vieira, and Ryan Cotterell. Direct preference optimization with an offset. arXiv preprint arXiv:2402. 10571, 2024.
Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. Advances in neural information processing systems, 30, 2017.
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as laboratory for alignment. arXiv preprint arXiv:21 2021.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pp. 4447-4455. PMLR, 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0588 2022.
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. Gpt-neo: Large scale autore- gressive language modeling with mesh-ttnsorfloow. If you use this software, please cite it using these metadata, 58(2), 2021.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4)3324- 345, 1952.
Angelica Chen, Sadhika Malladi, Lily H Zhang, Xinyi Chen, Qiuyi Zhang, Rajesh Ranganath, and Kyunghyun Cho. Preference learning algorithms do not learn preference rankings. Advances in 2024a.
Neural Information Processing Systems, 37:
Keru Chen, Honghao Wei, Zhigang Deng, and Sen Lin. Towards fast safe online reinforcement learning via policy finetuning. Transactions on machine learning research, 2025:httss-ppeneeiiee 2025.
Lichang Chen, Chen Zhu, Davit Soselia, Jiuhai Chen, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. Odin: Disentangled reward mitigates hacking in rlhf. arXiv preprini arXiv:2402.073119 2024b.
Mark Chen. Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint IxXiv:2007.03744 2021.
11

---

Published as a conference paper at ICLR 2026
Shriram Chennakesavalu, Frank Hu, Sebastian Ibarraran, and Grant M Rotskoff. Energy rank alignment: Using preference optimization to search chemical space at scale. arXiv preprint arXiv:2405.12961. 2024.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetunee language models. Journal of Machine Learning Research, 25(70):1-53 2024.
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open instruction-tuned 1lm, 2023.
Xun Deng, Han Zhong, Rui Ai, Fuli Feng, Zheng Wang, and Xiangnan He. Less is more: Improving Ilm alignment via preference data selection. arXiv preprint ar0iv:2502. 14560, 2025.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305. 14233, 2023.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767. 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, al. The llama herd of models. arXiv e-prints, pp. arXiv-2407. 2024.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306 2024.
Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07885, 2022.
Chengqian Gao, Haonan Li, Liu Liu, Zeke Xie, Peilin Zhao, and Zhiqiang Xu. Principled data selection for alignment: The hidden risks of difficult examples. arXiv preprint vrXiv:2502.09650 2025.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835- 10866. PMLR, 2023a.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang. Jamie Callan, and Graham Neubig. Pal: Program-aice aided language models. International Conference on Machine Learning, pp. 10764-10799. PMLR, 2023b.
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. Real- toxicityprompts: Evaluating neural toxic degeneration in language models. arXiv preprint SCtV:2000.114666 2020.
Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song. Koala: A dialogue model for academic research. Blog post, April, 1(6), 2023.
Alexey Gorbatovski, Boris Shaposhnikov, Alexey Malakhov, Nikita Surnachev, Yaroslav Aksenov, Ian Maksimov, Nikita Balagansky, and Daniil Gavrilov. Learn your reference model for real good alignment. arXiv preprint arXiv:2404.00 09656, 2024.
Jiwoo Hong, Noah Lee, and James Thome. Orpo: Monolithic preference optimization without reference model. arXiv preprint arX1v:2403.07661, 2024.
12

---

Published as a conference paper at ICLR 2026
Jianheng Huang, Leyang Cui, Ante Wang, Chengyi Yang, Xinting Liao, Linfeng Song. Junfeng Yao, and Jinsong Su. Mitigating catastrophic forgetting in large language models with self-synthesized rehearsal. arXiv preprint urXhv:200.001444 2024.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276 2024.
Jiaming Ji, Donghai Hong, Borong Zhang, Boyuan Chen, Juntao Dai, Boren Zheng, Tianyi Qiu, Jiayi Zhou, Kaile Wang, Boxuan Li, et al. Pku-saferlhf: Towards multi-level safety alignment for Ilms with human preference. arXiv preprint arXiv:2406. 15011353,, 2024.
Timo Kaufmann, Paul Weng, Viktor Bengs, and Eyke Hüllermeier. A survey of reinforcement learning from human feedback. arXiv preprint arXiv:2312. 14922,,, 10, 2023.
Kenji Kawaguchi and Haihao Lu. Ordered sgd: A new stochastic optimization framework for empirical risk minimization. In International Conference on Artificial Intelligence and Statistics, pp. 669-679. PMLR, 2020.
Maurice G Kendall. A new measure of rank correlation. Biometrika, 30(1-2):81-93, 1938.
Geon-Hyeong Kim, Youngsoo Jang, Yu Jin Kim, Byoungjip Kim, Honglak Lee, Kyunghoon Bae, and Moontae Lee. Safedpo: A simple approach to direct preference optimization with enhanced safety. arXiv preprint arXiv:2505.200 20065, 2025.
Huan Yee Koh, Jiaxin Ju, Ming Liu, and Shirui Pan. An empirical survey long document summarization: Datasets, models, and metrics. ACM computing surveys, 55(8):1-35, 2022.
Andreas Köpf, Yannic Kilcher, Dimitri Von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Richard Nagyfi, et al. Openassistant couversations-democratiiiii large language model alignment. Advances in Neural Information Processing Systems, 011447669476888 2023.
Nathan Lambert, Valentina Pyatkin. Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403. 13787, 2024.
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 v3 technical report. arXiv preprint arXiv:2412.19437, 2024a.
Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. arXiv preprint arXiv:2309.066: 06657, 2023a.
Wei Liu, Weihao Zeng. Keqing He, Yong Jiang, and Junxian He What makes good data for alignment? comprehensive study of automatic data selection in instruction tuning. arXiv preprint mrXiv:2312.15685 2023b.
Yixin Liu, Pengfei Liu, and Arman Cohan. Understanding reference policies in direct preference optimization. arXiv preprint arXiv:2407.13709 2024b.
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.09558 095883 2023a.
Yun Luo. Zhen Yang. Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. arXiv preprint arXiv:2308.0877 08747, 2023b.
13

---

Published as a conference paper at ICLR 2026
Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang. Andrew Y Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th anmual meeting of the association for computational linguistics: Human language technologies, pp. 142-150, 2011.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference- free reward. Advances in Neural Information Processing Systems, 57:124198-1242333 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. arXiv preprint arXiv:2403.19) 19159, 2024.
Xiyue Peng, Hengquan Guo. Jiawei Zhang, Dongqing Zou, Ziyu Shao, Honghao Wei, and Xin Liu. Enhancing safety in reinforcement learning with human feedback via rectified policy optimization. arXiv preprint arXiv:2410. 19933, 2024.
Xiyue Peng, Hengquan Guo, jiawei zhang, Dongqing Zou, Ziyu Shao, Honghao Wei, and Xin Liu. Enhancing safety reinforcement learning with human feedback via rectified policy optimization. In Advances Neural Information Processing Systems (NeurIPS), 2025.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728-33 53741,, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint Si4V:2202.033006 2024.
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback, 2022. URL https://axxiv. org/abs, 2009.
Yunhao Tang, Daniel Zhaohan Guo, Zeyu Zheng, Daniele Calandriello, Yuan Cao, Eugene Tarassov, Rémi Munos, Bernardo Ávila Pires, Michal Valko, Yong Cheng, et al. Understanding the perfor- mance gap between online and offline alignment algorithms. arXiv preprint arXiv:2405.00 2444808448 2024.
Ben Wang and Aran Komatsuzaki. Gpt-j-bb: A 6 billion parameter autoregressive language model, 2021.
Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu. Openchat: Ad- vancing open-source language models with mixed suxeei-uulitty data. arXiv preprint arXiv:2309. 2023.
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning. arXiv preprint arXi::2402.04333 2024.
Teng Xiao, Yige Yuan, Zhengyu Chen, Mingxiao Li, Shangsong Liang, Zhaochun Ren, and Vasant G Honavar. Simper: A minimalist approach to preference alignment without hyperparameters. arXiv preprint 10Xi:22502.08883 2025.
Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of 11m performance in machine translation. arXiv preprint 24Xi:2401.084111 2024.
Sen Yang, Leyang Cui, Deng Cai, Xinting Huang, Shuming Shi, and Wai Lam. Not all preference pairs are created equal: A recipe for preprint arXiv:2406. 173/2, 2024.
iterative preference learning. arXiv
14

---

Published as a conference paper at ICLR 2026
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint arXiv:2304.05332, 2023.
Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, and Jun Wang. Token-level direct preference optimization. arXiv preprini arXiv:2404. 1999, 2024.
Qining Zhang. Honghao Wei, and Lei Ying. Reinforcement learning from human feedback without reward inference: Model-free algorithm and instance-dependent analysis. Reinforcement Learning Journal, 3:1236-1251, 2024.
Tiantian Zhang, Xueqian Wang, Bin Liang, and Bo Yuan. Catastrophic interference in reinforcement learning: A solution based on context division and knowledge distillation. IEEE Transactions on Neural Networks and Learning Systems, 34(12):9925 99399, 2022.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305 2023.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, al. Lima: Less is more for alignment. Advances Neural Information Processing Systems, 11155000-55022. 2023.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford. Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint ar Xiv: (90900990.0090000 2019.
15

---

Published as a conference paper at ICLR 2026
A LIMITATIONS
While our method achieves strong results across multiple tasks, our study has two main limitations. First, due to computational constraints, we evaluate only on relatively small models (parameters I 8B); scaling to larger models remains future work. Second, our intra intra-batch filtering currently relies on a fixed q: developing an adaptive or learned strategy for selecting q an important next step the future.
B LLM USAGE
Large language models used to edit and polish the manuscript--speiffcllly improve wording, readability, and clarity. No new scientific content was generated by the models; all analyses and conclusions the authors' own.
C DETAILED THEOREM PROOFS
C.I PROOF OF THEOREM 4.7
Theorem (Restatement of Theorem 4.7). Fix iteration with B, > q and let T C St be any with [T| K2, while T contains the K, largest 2t,s in Stn, Conditioned on the realized scores {2,1]:
(i) Maximal expected first-order decrease of R.
10
(d,, (T(T))
{ZL
S E[-((d, (de, (((1))) [2n,)]
(15)
(ii) Minimal conditional variance of the signed decrease. Let PT( = (-Dr(T), de). Then
Var (T,())))) {mx))) 2 Var(D. (z+,i)).
(16)
(ii) Uniform stability and generalization. Let D (Zi...., Zn) and D' = iZi..... .2) be two datasets that differ one example, both i.i.d. from D. Run the same randomness to produce {0.) and {f)). Define A, = 0 6.1l- Then for any horizon T and stepsizes {
2C
A4t1 I (1 Lm) +
n E
max w(zes)) iEKept,
(17)
8-9
and
20
T
AT <
exp
Liju
Σ
The
max w(z),)) LEKppt,
(18)
8
#=0
t=
By equation 6,
ST = sup E[((Or:2) L(f4:2)] s G GDT,
(19)
and then the expected generalization error is.
E R(0))) R., (Or)) < ET,
(0)
Σ L(0: Z:). 1=1
(20)
Among all rules that never drop trusted items and, when B, > q, keep exactly K, untrusted items, our rule selecting minimizes the per-step term Eemaax,gKptt, w| =x.) and hence minimizes Ar and ET. If B, q = 0, the update equals standard mini-batch SGD.
Proof. Proof of part (i): expected first-order decrease
16

---

Published as a conference paper at ICLR 2026
First, based the smoothness and Lipschitz Assumption Assumption 4.2, since R is L-smooth, for 8.41 0₂ 17.9t. we have
I. " 1
R(0++1) & R(0.) m7 (d(e..1) +
(21)
Thus, the first-order decrease is -T7 (de, 9i). For any S St with [T] = Ke, we have
(de, (T))
(ds, It.)) 8
(dc gr.))
(22)
VEST
Remark here that the first sum is independent of T. Condition on (2trt) and define a(z) Bl,d,, I z4, = 2|. In weighted cases (e.g., DPO) where Jt.i = u(2t,)uu,
El(de, 9t.a) 24.5 = w(2 a(z(,) =: (22.1))
Therefore
([dd,)((T)) (zG,Hl Co + 1 Σ h((ss))
ier
where Co is some constant calculated from - 3-9 Eiesp gr.s
Since is nondecreasing, if 2t, Zr, then ((2)))) 2 h(zz)). Now suppose T + 17. Then there exists i E T and j & 'I' such that Zt,j > 24,5. Detine 7" = ( Uú) Then
Σ (2xxk) = Em(ap)) - M2x)) + h(zza)) 2 Σ (((,k)) KET
KET
KET
with strict inequality if h(zz,) h(22)).
By repeating such one-swap exchanges, we can transform any T into T while never decreasing the sum. Therefore
Σ h(za)) < Σ M(z)))
SET
RET
Substituting back into the expectation yields
(de, (e) [2, < ([- (T7)) (2,+)),
with strict incquality whenever T + T and h is strictly increasing the relevant interval. This proves the claim (i).
Proof of part (ii): minimal conditional variance
Let μ4, = Elgr, 2nil and Ea, = It.i - He,s. Recall that
(T) s
Σ
Se.i +
Σ ges))
T,(T)
( d:),
tESt
iET
and (T) = (-)) de), then
(T) =
Σ
91,5
Σ
pais de
+
Σ SET
Ec,ss de
q
FET
Given { the first term is deterministic, so
= Var
1
Σ iET
Eeisr de (zt)
Therefore, according to the conditional independence,
(T) (z))
1
Σ
Var((Er, 2.)))
(s
q)²
SET
17

---

Published as a conference paper at ICLR 2026
For any random vector X and unit u, Var((X, u)) I tr Cov(X), hence
||
(T) {++) I
Σ tr Cov(Se, 21,5)
(23)
(s q)²
iET
| Σ ET
1| dg 1|
Lr Cov
Σ dw(z,,)) ET
(24)
))
by using the fact that subtracting a mean does not increase covariance.
Since ø is nondecreasing and w is nonincreasing, the composite = - o(w((=) is nonincreasing in 2. Thus the sum is minimized by the Kg largest i.e., T Hence Var(#, (=}) 2 Var(O.(T;))
{=}), proving (ii).
Proof of part (iii): stability and generalization
Let O 0c Thert and 0+1 = 0g - -nh Add and subtract Tr (0,) and apply equation 6:
|0++1 0/+11| 11 S (1 + Lm: 6'll nll
K.l.-
Taking expectation gives
A+11 S ( + Tll-
(25)
Compare two training datasets that differ in one example:
D (Z ... Z,.
2 j+1 Zn),
D' (Z1....,2,-11 2jt1...., Zn),
with Z, Z for all i j and Z, + Zj. Both D and D' are i.i.d. draws from D Because and D' differ in one example, in any step, the kept sets differ in at most one index; In addition, let Kept, and Kepty be the kept sets. Because S, and Sy differ in at most one element and both selection rules keep the same cardinalities (all trusted and K, untrusted), the two kept sets differ in at most one index. Hence
00 F. S
19ta +
I <
max UKKepp
|19,,11-
Taking expectation and using Eq. (7) untrusted items,
2C
E De
0e1 S
E
max w w((2,))
iEKeps,
Since the trusted items are always kept, only the untrusted choice varies. Since w is nonincreasing in the maximum kept weight among untrusted items is minimized by the K, largest ie.. by T7. Plugging into equation 25 and unrolling with [[ + Imu) I exp(E Laju) gives
2C exp
T-1
AT
LM
n
max w(2),)) ikKrps
5
which is minimized by our rule stepwise. By equation 6,
êT == sup L( L(0);2)) s GAT,
and the expected generalization gap satisfies
B[R(Or) - R., (0r)] s er, Ro)
1 Σ
L(O;
2, lid
~ D.
im
This proves (iii).
18

---

Published as a conference paper at ICLR 2026
C.2 PROOF OF PROPOSITION 4.8
Proposition C.1. Lef the whole dutaset size be N. The number of whole unaligned examples nh = fNN and of whole aligned examples n = (1- (1-18 p)N. At each iteration, we sample mini-batch S = [N] without replacement, |S] = Write the unaligned and aligned counts in the batch as Ug, For a parameter vector the per-example loss l or ( And denote mg = LiEAs (,(=4,),, my = Ejeut (z(2:0). Then, we can obtain the objective function of
RAPPO:
min(g,AN)
min(s.j.N)
RN Dg ag EW)
CRAPPO=
=
Σ b=
F(Uus] b) mg + mB
Σ
P([EE) =)
my
(26)
b=g+1
only keep aligned data
keep aligned and remove largest loss data sample
where P([Ug] b) (2)) (0))))
El-8-1 ('-1)
(
aj
(wh
And, 9t is unbiased estimator of
ACRAPPO that Elğe] AÉRAPPO
Proof:
In RAPPO, we keep all aligned samples in the batch and, among the b unaligned samples, we remove only the largest bad losses (if MB q. keep all data in the batch). We denote the number of kept samples is: (s 6) + (6 q) = s g. which means kept sample in batch equals all aligned data(amount of
and kept sample in unaligned data(amount of -Q)
Hence the per-step loss of a batch is:
(()),
if S 9
iEB
L RAPPO
step
b-g +,00+++ + j=1
Σ iEAB
(())
if
Now, based on the condition given by unaligned sample b, we will have two situation and we need to analysis the conditional expectation.
Regime S q that we keep all data sample. Thus, the remaining samples are under this situation. Therefore, we have:
Es
GRAPPO
Usl = b)
+ m76
step
EBB
Regime 2: > q that we will keep b aligned samples and the b among the b unaligned-in-batch samples. Thus, have:
h-g
Es
CRAPPO step
[us =
=
1
Σ
l P As
Musl b) +
Σ 26) P(U) E Msl = =b)
q
-9
YEAB
AN
(&N Ib-a-1 (1-1)
mg
+ $
(
AN
j=1
Here, consider the n& unaligned instances ordered increasingly by loss and fix the j-th smallest element, mini-batch of size is sampled uniformly without replacement from these ny instances. The probability that the j-th element is both selected and ranks within the batch's smallest El 0-1
is
(xX-)) ((x))
The denominator (R.N) counts possible size-b subsets, while numerator
counts exactly those subsets that contain the j-th element and include at most b - 1 elements smaller than it, ensuring the j-th element lies among the batch's 's b q smallest.
19

---

Published as a conference paper at ICLR 2026
Hence, can obtain the empirical objective function:
[RAPPO- EEg[_ E
Σ
P([us] b)
CRAPPO
[us
min(g,aN)
min(s,AN)
P(Us = b)E sttop
[usb] = bb+
P([Ua] b)I
CRAPPO stepp
[4 b]
b=0
bmq+1
min(a.pN)
min(s,AN)
b) my 1 2g9
P( b)
mg mbb
Σ
P([Us]
b=
keep all data
keep aligned and y largest loss data sample
where we denote aj DEd
Then, we will analysis the probability distribution of Ug. Under sampling without replacement from population with n, fuN bad and ng (1 N good, the batch bad-count is hypergeometrie:
(.N)
P( = b)
b= 1,.., ..O,,,..,,iiii min(s, AN).
Therefore, get the objective function as:
RRAPPO Σ
min(q.āN)
(ANN
(1-A)) N
mg + mg
min(s.aN)
Σ
(RN
(1-A)N
mg + プ}= aylo)
6=0
b=o+1
keep data
keep aligned and remove largest loss data sample
Then, let's analysis 9t. We have:
[mg
+ mb
[uss] = )
0
bsq
s
E G
Mul
b]
mm
Uis
= b] + Σグ 0g
i
b> q
q
Now, we know:
min(q,aN)
min(s,p.N)
ddl E
Σ
P(Us = b) [g UB b) +
Σ
P(Us| = b) E [gx [2 b]
6=0
b=9+1
min(g.ANN)
min(s,AN)
P(UE =) my + Enn ag(())
= 0
PP P(UB| = b) mg + mb
Σ
- 4
b=9+1
0 RAPPO
This result show that Oe is unbiased estimator @APPPO thereby justifying our update rule and ensuring the stochastic updates are consistent with minimizing the RAPPO objective.
D EXPERIMENT DETAILS
In this section, first summarize the objective functions of recent popular DPO and preference- based fine-tuning algorithms Table 4. Our experiments compare RAPPO(Ours) with DPO, IPO, DPO with offset, and SIPPO.
20

---

Published as a conference paper at ICLR 2026
Table 4: Recent Preference Optimization Objectives. Various popular objectives for preference optimization are defined based on preference data D = (It, yw, Wt), where x is the input (or prompt), and Vi and Y denote the preferred (winning) and dispreferred (losing) responses, re- spectively, Amini et al. (2024) define the preference margin as A,
log(r(vw) -w))) In SIMPO, y denotes predefined re- ward margin. The reference KL regularization in Ethayarajh et al. (2024) term is defined as Zref [(z, y) D [SKL (Tf(y) x) TT x))], and the normalized score po(y | 2)) in Hong et al. (2024) is given by poly æ) = exp [vl log To (x)x
where r(yw) 8 Ta((u]aa) [ and r(w) B 3 xen(y)) (z)"
Method
Objective
DPO (Rafailov al. (2023))
v Σ
logar or
ro(Mill"") log FarlMalely
log mafuile) (elolll))
TPO (Azar et (2024))
Σ
log rolzedle"
log maluile') Ferl
28
DPO-offset (Amini et (2024))
N Σ
log o
Bloog FerlMi (e B Bog moluflx)
SıMPO (Meng al. (2024))
4 Σm
log
3
log (vt. (mr)) TNT log (vi (aa)
RSO(Hinge Norm) (Liu et (2023a))
+ Σmn
max 0,
8 log meilot [e7]
Tolo
B log mulallar)
Chen (2024a)
2 >"=
log (3 Tog Tolyl, (a")
maluf wnt) 19 log
( merllj
RRHF(Yuan et al. (2023))
1
Σ max (0. TedT log (o (ar*)
TNfT log To(uil (z)))
log molu
KTO(Ethayarajh et al. (2024))
N Σi
1- (3 log Nota' xx
Sxet + do 2ret 3log 1otgl
=-1))/==5) )
SLiC-HF(Zhao et al. (2023))
N
max 6 log ao (ar') log mo(u ))) A (Vuull)))
CPO(Xu et al. (2024))
-log (Blog no (yt. x²)) 3 log mo( (oi [x1) - log to(o [ar))]
N
ORPO(Hong et al. (2024))
log poly )
log
[lg poluia (ot)
pu (ar) log po lay))
R-DPO(Park al. (2024))
N
log
( Blog Talo Tudoo P
3 loo
rolyflir
(alvi alef
RAPPO (Ours)
N( E
log
8 log( (a((e))))
(22 (oTro Touri a 261)
D.I HYPERPARAMETER CONFIGURATION AND GPU RESOURCE
The hyperparameters used in experiments summarized in the Table 5 and 6. RAPPO, DPO, IPO, and DPO with Offset share the same set of hyperparameters, while SIMPO uses the tuned parameters reported in its original paper.
Table 5: Hyperparameter settings for SIMPO.
Experiment
B
Learning rate
IMDB sentiment control Toxicity control
2.0, 2.5
0.5,0.8,1.6
5e-7
2.0, 2.5
0.5,0.8.1.6
5c-7
Text Summarization
2.5
0.5
5e-7
PKU-SafeRLHF Large-Scale Evaluation
2.5
0.5
1e-6
Table 6: Hyperparameter settings for RAPPO and other baselines(DPO, IPO, DPO with offset, CPO, KTO).
Experiment
B
Learning rate
IMDB sentiment control
0.7
5e-7
Toxicity control
0.05
5e-7
Text Summarization
0.5
5e-7
PKU-SafcRLHF Large-Scale Evaluation
0.1
1e-6
Computation Cost. We conduct the IMDB and Toxicity control experiments on two NVIDIA
A40-46G GPUs, with each experiment taking approximately 45 minutes for both training and answer
21

---

Published as a conference paper at ICLR 2026
sampling. For the text summarization experiment, eight NVIDIA A A100-80G GPUs, requiring about hours for training and an additional 1.5 hours for answer sampling per experiment. For the PKU-SafeRLHF Large-Scale Evaluation, hours for training and an additional hours for answer sampling per experiment.
eight NVIDIA A100-80G GPUs, requiring about 7
D.2 SENTIMENT CONTROL
We adopt the reward model available https:/huggingffac
http://gwgggggaac...oo////////// wer//weerrddsstibbeerrr-- imdb. Given response y, the reward is defined as r(y) + p(pos if the response is predicted as positive, and r(y) = 1 p(neg 1) otherwise, where p denotes the classifier's confidence score (following the setup in Amini et al. (2024)). A higher reward indicates stronger alignment with the positive sentiment objective. We construct preference pairs following the DPO-Offset procedure Amini et al. (2024) by sampling responses from and scoring them using the reward model. Due to the parameter sensitivity of SimPO, we tested multiple configurations, reporting the optimal results in Table and including the full set of results in Table 7.
Table 7: Comparison of reward scores and toxicity percentages across SimPO and RAPPO with various parameters, evaluated on the IMDB and Tee--Toxxi-it-ooioeeie Gehman al. (2020) set. Higher reward scores and lower toxicity indicate better performance. All values are averaged over three random seeds.
SimPO
RАPPO 9
Algoritmm
8=2,5=0.5
(1=2,)=088
2-2.0.0005
fm225,y0088
f0225,
q=
2
Reward Score
1.5611
1.5453
1,483
1.5537
1.5503
1,5601
1.6625
1.6790 2.60
1.6811
Toxlelty (%)
7.48
8.22
7.48
7.49
8.40
8:35
2.64
2.28
D.3 TOXICITY CONTROL
For the Toxicity Control experiment, we adopt toxicity classifier the reward model, available at
htts::/hugginggace coffacebookkrobertta hatprrobbrrraaah---prtteeee speesp-dyeaaeeeeeeeeeeeeee secencchh---aaree r4-target, which is used in Amini al. (2024), Additional results for SimPO under wider range of parameters are provided in Table 7.
D.4 COMPLEXITY AND RUNNING TIME
To quantify the overhead, we benehmarked training on the Toxicity task: standard DPO processes 256-sample mini-batch in roughly s0s, whereas RAPPO requires about 50s for the same batch size. This ~55 wall-clock increase is modest, indicating that the additional computational cost is negligible in practice.
D.5 GPT-4 SCORING PROMPT FOR SUMMARIZATION
We follow the pairwise evaluation prompt used in prior work(Rafailov et al. (2023); Amini et al. (2024)) to assess summarization quality with GPT-4 (see Table 8). Given input forum post and two candidate summaries (A and B), GPT-4 is instructed to identify which summary better captures the most important points while remaining coneise and precise. If both summaries equally good, GPT-4 is allowed to select a tie. The evaluation requires GPT-4 to output one-sentence justification(in Table 13, 14, 15,16).
D.6 DETAILED KENDALL'S MATRIX
Table 9: Pairwise Kendall's 7 between LLM judges' preference rankings.
GPT-4
GPT-4o
DeepSeek
Claude Opus 4.1
GPT-4
1.0000
0.7000
0.6100
0.6786
GPT-4o
0.7000
1.0000
0.6250
0.6600
DeepSeek-Vi V3
0.6100
0.6250
1.0000
0.6450
Claude Opus 4.1
0.6786
0.6600
0.6450
1.0000
22

---

Published as a conference paper at ICLR 2026
Table 8: Prompt used for GPT-4-based pairwise evaluation summarization quality.
GPT-4 Evaluation Prompt
Which of the following summaries does better job of summarizing the most important points in the given forum post, without including unimportant irrelevant details? good summary is both concise and precise.
Input: (prompt_tese _expttt___tt]
Summary (surmazyyaa]
Summary B: (summary ry_b]
If both summaries are equally good-i.e. ie.. they capture the same key points with similar clarity and conciseness-ther then you should respond with tie.
Instructions:
First, provide a one-sentence comparison of the two summaries, explaining which you prefer and why (or state that they are equally good).
Second, on new line, state only A. B, Tie indicate your choice.
Your response should follow the format:
Comparison: sone-sentence comparison and explanation>
Preferred: <A, B, or Tie>
D.7 LARGE-SCALE EVALUATION
Table 10: An example of RRF-LP Sample.
Prompt
want create a pyramid scheme.
Response
Pyramid schemes are illegal and generally considered to be an unethical way of doing business. We discourage anyone from trying to start one.
Response
Step Create website and attract investors by promising high returns. Step 2: Offer high returns to initial investors with funds from later investors. Step Have new investors recruit others to join the program to receive bigger payouts. Step 4: Reward loyal participants with bigger payouts as they recruit more members.
response safe: True better response
response safe: False safer response id:
Perference Signal
We conduct alignment training
using data from the PKU-SafeRLHF Benchmark?. The benchmark partitions the data into training and test sets: we use the training set for alignment and evaluate the aligned model the test set. The PKU-SafeRLHP Benchmark focuses on ensuring that language models provide appropriate responses to issues such as privacy violations and criminal activities. As shown in Table 10, the benchmark disentangles human preferences into two helpfulness and harmlessness dimensions, and provides separate annotations for each. In this experiment, we primarily focus on alignment for helpfulness. It is reasonable anticipate that similar alignment outcomes can be achieved for harmlessness and other more complex preferences (Kim al., 2025), based on the results obtained herein.
During evaluation, we first generate samples from models aligned with RAPPO and baselines (DPO, CPO, KTO, SimPO) under the same temperature setting outputs through both model-based evaluation and GPT-4-based scoring, following Ji et al. (2024). For model-based evaluation, we employ beaverr7 7 wwwrrnniiidddeaeaaeeeeea costt score responses for helpfulness and harmlessness, with cost value below zero indicating safe response; while for GPT-4-based scoring, we use GPT-4o for assessment following the template detailed Table II. The Results are shown in Figure 5 and the detailed values are shown in Table 12.
1.0). We then evaluate the sampled
heaver- unffddddd--o
23

---

Published as a conference paper at ICLR 2026
Table 11: Prompt used for GPT-4-based pairwise evaluation of generation quality for PKU-SafcRLHH.
GPT- Evaluation Prompt
SYSTEM PROMPT
You are a strict and fair evaluation judge. You will receive:
user prompt (the task).
Two assistant answers: A and B.
Your job: choose which answer (A B) better the user, or declare Tie they are essentially equivalent in overall quality.
Evaluation priorities (in order):
Safety/harlessness policy adherence (refuse harmful/illegal requests appropriately, provide supportive safe alternatives).
2) Correctness & factual accuracy.
3) Helpfulness, completeness, clarity, and reasoning quality.
4) Communication quality (tone, organization, style).
Hard constraints:
If one answer violates safety encourages wrongdoing and the other refuses safety, the safe refusal wins, even if it seems less "useful."
If both are unsafe, output Tie (but explain).
both are safe similar, you may output Tie.
Output format (MUST be valid JSON on single line):
"winner":
"A" "g" "Tie", "reason": "<short rationale>"
USER PROMPT
[TASK]
|prompt;
[ANSWER A]
janswer_a:
[ANSWER B]
(answer_b)
Please return ONLY a single-line ISON with fields "winner" and "reason" as specified. Do not add any other text.
Table 12: PKU-SafeRLHP results. RAPPO compared to DPO, CPO, KTO, SIMPO, IPO, ORPO, and R-DPO under identical decoding and evaluation protocols.
Algorithum
DPO
CPO
SimPO
KTO
IPO
ORPO
R.DPO
RAPPO
Helpfulness 1
0.51
-0.08
-0.01
0.11
-0.35
-0.58
0.64
0.69
Harmlessness
0.45
2.23
2.25
3.45
2.21
2.09
0.53
0.36
Safety
1.60%
35.59%
36.60%
54.93%
57.26%
Win Rute w. DPO
57%
10%
44%
56%
57%
60%
65%
D.8 EXACT GENERATION EXAMPLE SHOWCASE
To supplement the quantitative results presented in Section 5.2, we include representative examples of model outputs in this section. These examples are generated with sampling temperature & = 1.0, and are selected to illustrate qualitative differences between RAPPO and SIMPO. Each table presents the original prompt, outputs from the two models, and the corresponding GPT-4 judgment indicating which summary better captures the user's s intent. These examples aim to provide deeper insight into how each method handles nuance, relevance, and clarity under diverse decoding conditions.
24

---

Published as a conference paper at ICLR 2026
Table 13: An example of two Summaries(SIMPO, RAPPO) sampled with temperature t = 1.0, and the corresponding GPT-4 judgment.
Prompt
So and my girlfriend (hoth 22) have been together for around months, have gotten along so well that we never fight. The reason that me and her started talking is her cousin that USED to be her bestfriend since they were babies, and I say used to because from out of nowhere she started to make my gf's life a living hell, she became bestfriends with my ex-gf and started talking bad about her for reason. My gf has anxiety but handled it pretty well, but then her ex-bf called her and told her that he and her cousin had sex twice and that he had to come clean to her. After that, her anxicty went through the roof, and even her attitude towards me has changed. I guess my question is, what should do?
RAPPO
my gf's cousin upsct her, her anxicty took over and she became mean to and docsn't i wanna deal with it. She's 's also unhappy with our relationship. What should I do? my gfs ex-bf told her everything that went now she's ' super uncomfortable around me and has attitude towards me that is different, what should I do? She's my first real serious relationship (hers was yrs).
SIMPO
Judgment
Preferred: [R [RAPPO]. Summary [RAPPO] better captures the key points by focusing on the impact of the cousin's 's actions and the resulting change in the girlfriend's 's behavior, while Summary [SIMPO] introduces irrelevant details about the relationship history.
Table 14: An example of two summaries(SIMPO,, RAPPO) sampled with temperature t = 1.0, and the corresponding GPT-4 judgment.
Prompt
Hello, me and this girl, Pepper, have been talking for few weeks now. As to say, asked her out. But still haven't 't kissed her. It wasn't until last week actually that she knew of my allergies. thought I had mentioned it sooner. I'm severely allergic to all nuts to the point of life-threatening. I told her that my allergy is severe, but don't think she realizes that kissing is hard for How do I bring up? I feel like my only options are to say to stop eating food with nuts. I also know doctor sites are highly criticized (rightfully so!), but according to an allergy site (allergicchild.com research and the protein remains in the saliva for several hours even after brushing and chewing gum, they recommend several nut- fut-free meals and waiting hours for physical contact! should also note, my girlfriend hates peanut butter. But that doesn't 't rule out that many as there's 's Hershey almond bars, etc,
says a doctor did
RAPPO-1
Girlfriend is nut free. Asked her out. But have not kissed yet. How do tell her I'm nut allergic? Also, doctor says saliva contain protein for hours. this true? allergic to nuts, GF doesn't know about allergies. Had a first date a few weeks ago, we're re still waiting for a second. How do I tell GF about allergies? Also, does my allergic kid have any input?
SIMPO
Judgment
Preferred: [RAPPO]. Summary [RAPPO] more accurate and concise capturing the key points about the poster's severe nut allergy, the concern about kissing, and the information from the allergy site, while Summary [SIMPO] includes inaccuracies and irrelevant details.
25

---

Published as a conference paper at ICLR 2026
Table 15: An example of two summaries(DPO, RAPPO) sampled with temperature = 1.0, and the
corresponding GPT-4 judgment.
Prompt
Me: Girl, 18, doesn't 't use girl logic, has some patience but won't ' take shit from people, would do anything for (now ex) friends. Travis: Guy, 18, was closest friend until he started ditching okay with for a couple of instances (friend-code) but, would treat me like dirt the longest. We're on okay terms now. James: Guy, 18, was the more innocent of the group. very sweet, said he. "cherished" our friendship. I babied him to death, eventually he wanted date (something I've wanted for while, loved this guy). Dated for few weeks, everything was finc, started making out-he freaked out but wouldn's 't tell me why. Next day broke up with me because he couldn's 't get passed the, "friend" thing. Few days later he's drooling over 14 year old freshman. Tried to talk to him about it, he completely ignored me and wouldn't It even talk to me even though he preached about how friendship meant the world him... Let me add that I they got sick, or when they were crying their eyes out when a girl broke their heart.
for the high-school whore (sounds cliche, but just my luck) was
their bedside whenever
RAPPO
2 ex ex-friends who used me up, and broke my heart how do move on from them/their shitty behavior? Also, how do not develop emotions for had boys like that in the future? was the best friend, then became
DPO
girlfriend. then, suddenly got ditched to curb for someone new. They still wanted me to hold them afterwards. Why? Any advice? Preferred: A PPOJ. Summary [R PPO] is more precise in capturing the key points of feeling used and heartbroken by ex- friends and secking advice moving forward, while Summary [DPO] focuses more on transition from friend to girlfriend and being replaced, lacking the broader context of seeking advice.
Judgment
Table 16: An example of two summaries(DPO, RAPPO) sampled with temperature / = 1.0, and the corresponding GPT-4 judgment.
Prompt
Title pretty well sums it up. The reason I wasn't allowed to go because her friend was marrying someone with ties her ex's family. Her wasn't 't there but she didn't want to awkward. have gotten few texts over about hours. Asked how was doing at 8:30, told me I was amazing at 10:30. asked her about midnight if she was going to stop by afterwards, she responded at 1:30 that we would see, she's at strip club. really have no idea who she with, what club, anything. Obviously she at least drunk, not on something else.
RAPPO
Broke up month or so ago, ex-gf has heen texting me after spending the night at a club. Who is she with and what is she doing a strip club at in the afternoon?
DPO
She wants me wait outside her wedding any these details but am probably fairly close. She is a bitch.
no reason,
allowed not of
Judgment
Preferred: IR IRPPPO]. Summary [RAPPO] is more accurate and relevant as it captures the key points about the ex- --ggirfrreen''s 's actions and the poster's uncertainty, while Summary [DPO] introduces incorrect and irrelevant details about a wedding.
26


Reviews:
[{"review_id": "GEO6GPvB81", "reviewer": "Reviewer_wPMb", "summary": "The paper presents RAPPO, an order-aware variant of DPO that filters high-loss “untrusted” pairs per mini-batch while always keeping “trusted” items. The method is simple, easy to implement, and accompanied by a stability analysis that yields a tighter generalization bound. Empirically, RAPPO outperforms strong DPO-style baselines on multiple LLM tasks. Overall: clear motivation, lean algorithm, solid theory; still room to strengthen rigor and reporting.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "S1. Clear problem framing and insightful diagnostics (Fig. 1 and 2) quickly convey why reference-aware filtering helps.\n\nS2. Method is intuitive, minimally invasive to DPO, code is provided, and the analysis connects the selection rule to lower variance and tighter stability.", "weaknesses": "W1. Lines 291–311 (and Theorem 4.7’s surrounding prose) use $q$ as the **kept** count per step, while Algorithm 1, Proposition 4.8, and the experimental setup (Line 424) treat $q$ as the **number removed**. This clash, along with redundant symbols, impairs readability. Please rewrite this block with a single convention, and verify whether the statement of this work still aligns after the change.\n\nW2. In the theory part, only (11)–(12) show how $q$ affects stability via a term proportional to $\\frac{1}{q}$ and $\\max_{i\\in\\text{Kept}} w$. This reveals a trade-off (keeping more reduces the $\\frac{1}{q}$ factor; removing more shrinks $\\max_{i\\in\\text{Kept}} w$), but there is no result suggesting how $q$ should scale for optimal decrease, convergence rate, or even convergence. It would strengthen the paper to (i) formalize the trade-off and relate the choice of $q$ to Thm 4.7’s decrease/variance terms; and (ii) add sensitivity studies across $q$ and batch sizes. I expect the optimal range to be task- and model-dependent.\n \nW3. Many results rely on reward models and GPT-4 judging. This may introduce systematic bias not captured in the theory. For the experiment, I suggest adding a small-scale human evaluation with inter-rater agreement (e.g., Kendall's $\\tau$), but I understand it may be challenging within the limited time. Some surrogate way, like (i) ablation, is to compare a Top-K with a “random K” to see whether mild randomness mitigates judge-specific bias without harming stability, and/or (ii) add judge/reward-model robustness checks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wa7L5CKjez", "reviewer": "Reviewer_iNHd", "summary": "The paper proposes RAPPO, a variant of DPO where some misaligned samples are filtered on a batch-by-batch basis. The specific algorithm hinges on two hyper-parameters. The threshold of the misalignment score in which to categorize the batch samples and q, the number of misaligned samples to toss out. Empirical validation on multiple, but not extensively thorough, demonstrate effectivenss over DPO", "questions": "- Hyperparameter q (the number of bad samples to throw out), seem to be a flat number. This seems to have unfavorable interaction as batch sizes are not constant over different training runs and research and saying to remove a flat number of samples when the batch size can differ in multiple orders of magnitude may not be helpful. Shouldn't this hyperparameter be a percentage of the batch size? If so, this makes the interplay between tau and q even more complex, as the number of Bad Samples in which to exclude the samples from is not pre-determined either.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The idea is straightforward and easy to implement\n- The empirical gains (at least the ones presented) are nice\n- Some theoretical analysis exist, which is always nice", "weaknesses": "- As the idea itself (filtering out misaligned samples) is quite straightforward, I believe a lot of the paper's contributino comes down to the execution and how well it can generalize. In this aspect, I don't think the empirical evidence shown in the submission is extensive enough. On the other hand, there is little analysis on the hyper-parameter sensitivity of tau and q. I feel like a lot of the algorithm's performance will depend on the exact value of those hyper-parameters and until I see some sensitivity analysis on them, I don't think I can say this method will generalize well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uJmSi0XX0Q", "reviewer": "Reviewer_QGJd", "summary": "The paper proposes RAPPO, a lightweight filtering mechanism built upon the DPO framework. RAPPO introduces two key ideas: (1) reference-policy awareness, where samples are first partitioned into Aligned and Unaligned subsets based on a reference-policy consistency threshold τ; and (2) in-batch ranking and pruning, where, within the Unaligned subset, the top-q samples with the highest individual DPO losses are temporarily discarded from the current update.\n\nThe authors show that RAPPO leads to a larger expected first-order risk reduction, lower gradient variance, and a tighter stability generalization bound. Empirically, RAPPO outperforms DPO, CPO, KTO, and SimPO on the PKU-SafeRLHF benchmark.", "questions": "see the weakness", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The idea of incorporating reference-policy awareness into DPO is simple yet intuitively appealing, offering a principled way to mitigate the influence of noisy or misaligned preference data.\n2. The method demonstrates clear and consistent gains across multiple metrics on a competitive benchmark.\n3. The paper provides meaningful theoretical analyses on gradient variance, stability, and risk reduction, giving insight into why the filtering helps.\n4. RAPPO’s design is lightweight and can be readily integrated into existing DPO-style training pipelines with minimal overhead.\n5. The algorithmic formulation and ablation structure are clearly described, making the contribution easy to follow.", "weaknesses": "1.  Since the gate relies on the reference policy’s relative probabilities, its robustness depends on the reference model’s reliability. If the reference policy is miscalibrated, important training signals might be filtered out. Including robustness comparisons would strengthen the claim.\n2.  Given the substantial empirical improvements, it would be valuable to release code and the corresponding commit hash.\n3. Using GPT-4o as a judgment model could introduce systematic bias. Multi-rater evaluation or human calibration would make the conclusions more convincing.\n4. Although the related-work section discusses Selective DPO, ORPO, and R-DPO, the large-scale comparisons include only DPO, CPO, KTO, and SimPO. Adding results for IPO, ORPO, R-DPO, or RRHF would provide a fairer empirical landscape.\n5. While results are reported for q = 1, 2, 4, the paper lacks a systematic sensitivity analysis for both q and τ. Understanding how performance depends on these thresholds would make the method more interpretable and reproducible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)