RE-PO: Robust Enhanced Policy Optimization as a General Framework for LLM Alignment
Authors: Xiaoyang Cao
Zelai Xu
Mo Guang
Kaiwen Long
Michiel A. Bakker
Yu Wang
Chao Yu
Venue: ICLR 2026 Poster (2026)
Keywords: large language model
alignment
robustness
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: RE-PO is an EM-based “robustification layer” for LLM preference alignment that infers per-example (and per-annotator) label reliability and uses it to reweight training, improving robustness to noisy feedback.

Abstract: Standard human preference-based alignment methods, such as Reinforcement Learning from Human Feedback (RLHF), are a cornerstone technology for aligning Large Language Models (LLMs) with human values. However, these methods are all underpinned by a strong assumption that the collected preference data is clean and that all observed labels are equally reliable. In reality, large-scale preference datasets contain substantial label noise due to annotator errors, inconsistent instructions, varying expertise, and even adversarial or low-effort feedback. This creates a discrepancy between the recorded data and the ground-truth preferences, which can misguide the model and degrade its performance. To address this challenge, we introduce Robust Enhanced Policy Optimization (RE-PO). RE-PO employs an Expectation-Maximization algorithm to infer the posterior probability of each label’s correctness, which is used to adaptively re-weigh each data point in the training loss to mitigate noise. We further generalize this approach by linking a broad class of preference losses to induced probabilistic models. This enables systematic robustification of existing alignment algorithms while preserving exact probabilistic equivalence for likelihood-style losses. Theoretically, under perfect calibration and a population/full-batch setting, we show that RE-PO recovers the true annotator reliability. Our experiments demonstrate RE-PO’s effectiveness as a general framework, generally enhancing four state-of-the-art alignment algorithms (DPO, IPO, SimPO, and CPO) against their corresponding standard versions. When applied to Mistral and Llama 3 models, the RE-PO-enhanced methods improve AlpacaEval 2 win rates by up to 7.0% over their respective baselines.

Published as a conference paper at ICLR 2026
RE-PO: ROBUST ENHANCED POLICY OPTIMIZATION AS A GENERAL FRAMEWORK FOR LLM ALIGNMENT
Xiaoyang Cao'; Zelai X Mo Guang", Kaiwen Long', Michiel A. Bakker', Yu Wang?, Chao Yu
1 IDSS, Massachusetts Institute of Technology, 2EE, Tsinghua University, 'Li Auto Inc.,
SIGS, Tsinghua University
ABSTRACT
Standard human preference-bane based alignment methods, such as Reinforcement Learning from Human Feedback (RLHF), are cornerstone technology for align- ing Large Language Models (LLMs) with human values. However, these methods are all underpinned by a strong assumption that the collected preference data is clean and that all observed labels are equally reliable. In reality, large-scale prefer- ence datasets contain substantial label noise due to annotator errors, inconsistent instructions, varying expertise, and even adversarial creates discrepancy between the recorded data and the ground-truth preferences, which can misguide the model and degrade its performance. To address this chal- lenge, we introduce Robust Enhanced Policy Optimization (RE-PO), RE-PO employs an SoppetttiionMaxxmizallle
low-effort feedback. This
algorithm to infer the posterior probability of each label's correctness, which is used to adaptively re- re-weigh each data point in the training loss to mitigate noise. We further generalize this approach by linking a broad class of preference losses to induced probabilistic models. This enables systematic robustification of existing alignment algorithms while preserving exact probabilistic equivalence for likelihood-style losses. Theoretically, under perfect calibration and a population/full-bette butthh setting, we show that RE-PO recovers the true annotator reliability. Our experiments demonstrate RE-PO's effectiveness general framework, generally enhancing four state-of state-of-the-art alignment algorithms (DPO, IPO, SimPO, and CPÔ) against their corresponding standard versions. When applied to Mistral and Llama 3 models, the RE-PO-enhanced methods improve AlpacaEval 2 win rates by up to 7.0% over their respective baselines. Our code and other resources are available at rrppo-roallle wwwggnmeetgggiiiiiee
INTRODUCTION
Aligning Large Language Models (LLMs) with human values a critical prerequisite for developing safe and reliable Al systems. Reinforcement Learning from Human Feedback (RLHF) has emerged as the dominant paradigm for this task (Christiano al. 2017 Ziegler et al., 2019: Ouyang et al. 2022). To mitigate the complexity and instability of the traditional RLHF pipeline, simpler and more direct methods such as Direct Preference Optimization (DPO) (Rafailov et al. 2023] have been developed, which reframe alignment as classification-like problem.
However, these alignment methods implicitly assume that preference datasets provide clean and reliable approximation of single ground-truth preference signal. In practice, this assumption is often violated. Large-scale preference datasets are typically aggregated from multiple crowdworkers or teacher models, and are therefore subject to substantial label noise arising from inattention, misunderstanding, or systematic bias (Frénay & Verleysen, 2013 Gao et al.) 2024). Empirical analyses suggest that significant fraction (often between 20% and 40%) of preference pairs in modern alignment datasets may be corrupted or inconsistent (Gao et al. 2024). Classic work on learning
with noisy labels shows that standard loss functions overfit such corrupted supervision and suffer
"quual contribution: xycao@mit. zelai selai.eecs$gmail con.. com.
Corresponding authors: yuchaolsz. tsineesttttttiitaaaaaaa cn.

---

Published as a conference paper at ICLR 2026
E-Step
Preference Data
Freeze Param
Confidence Score
Language Model
Rasponse
Binary Preference
p
y
yP
VEU
Rellabillty
Model Parameter
8 1
(t)
wiO p(y V1lY" 2 )(
Confidence Score
Prompt
Resporse
?
M-Step
Robust Loss Lx
Robust Signal
Model
log p(y, > y) wi(
θ (
4.0%
60%
Yw
Annotator Reliability (t+1)
1
logp x' yu.100) y (1 w()
Figure 1: Overview of the Robust Enhanced Policy Optimization (RE-PO) framework. Starting from noisy pairwise feedback, RE-PO uses Expectation- iatttionnMammizattion (EM) procedure to jointly refine label confidences and the policy. each iteration, the E-step estimates a confidence score for every observed preference by inferring the posterior probability that the label is correct under the current model and annotator reliabilities. The M-step then uses these scores as adaptive weights to update both the LLM policy and the annotator reliability parameters, progressively down-weighting likely corrupted labels and emphasizing reliable supervision.
severe degradation in generalization performance (Natarajan et al. 2013). Similar observations are reported the label-noise survey Frénay & Verleysen (2013). the context of L.M alignment, Gao et al. (2024) further demonstrate that even a 10 percentage point increase in the label-noise rate can lead to drops of tens of percentage points downstream win rates, highlighting the practical importance of robustness to noisy preference data.
To address this challenge, we propose Robust Enhanced Policy Optimization (RE-PO). Instead of assuming that every observed label is fixed ground truth, approach aims to learn preference model that remains accurate and stable even when the training data contains substantial noise. The innovation of RE-PO is its departure from the hard labels used in traditional RLHF. Rather than committing to binary supervision, we treat the correctness each observed preference as a latent variable and compute soft confidence weights over labels, that highly reliable feedback contributes more strongly while suspicious pairs are down-weighted. Building on Expectation- Maximization-style approaches learning from unreliable annotators crowdsourcing (Dawid & Soppetation-Maximizattee Maximization (EM) framework that simultancously models annotator reliability while optimizing the LLM. In the E-step. infers the posterior probability that each annotated label is correct, effectively estimating annotator reliability. In the M-step, it uses these probabilities as adaptive weights to update the LLM,, thereby learning from a dynamically re-weighted preference signal.
Skene 1979; Chen et al.. 2013). RE-PO employs an Expectation-
Our experiments validate RE-PO as an effective general framework. We show that applying RE-PO generally enhances four state-of-the-art alignment algorithms (DPO, IPO, SimPO, and CPO) across two different base models (Mistral-7B and Llama-3-8B) on the AlpacaEval benchmark (Table 2}. In our main results, RE-PPO-enannee methods achieve substantial win-rate gains on AlpacaEval 2. with improvements of up to 7.0 percentage points over their standard counterparts. Furthermore, we theoretically analyze when RE-PO recover annotator reliability (Theorem 4.1) and empirically verify this behavior in controlled experiments (Section 5.5
In summary, contributions
follows:
We propose Robust Enhanced Policy Optimization (RE-PO), principled EM-based algorithm that treats the correctness of each preference label latent variable, jointly infers per-label
2

---

Published as a conference paper at ICLR 2026
(and per-annotator) reliabilities, and uses them adaptive weights in the training loss, yielding LLM alignment that is substantially more robust to noisy and inconsistent feedback.
We theoretically establish generalized RE-PO framework by using the Gibbs construction to connect a broad class of preference losses to induced probabilistic models. This lifts RE-PO from a single algorithm to a general framework, enabling standard methods such as DPO, IPO, SimPO, and CPO to be systematically transformed into robust counterparts; for non-likelihood losses, the induced objective can differ from the original one.
We conduct extensive experiments demonstrating the practical effectiveness and versatility of RE-PO.. Across four alignment algorithms, two base models (Mistral-7B and Llama-3-8B), and AlpacaEval 2, RE-PO delivers consistent win-rate improvements of up to 7.0 percentage points, and further shows clear gains on a real multi-annotator dataset (MultiPref), along with qualitative and visual analyses of how it down-weights low-confidence, noisy labels.
2 RELATED WORK
LLM alignment with hard preference labels. The standard paradigm for aligning Large Language models (LLMs) with human values is Reinforcement Learning from Human Feedback (RLHF). which involves training a reward model and then fine-tuning the policy against it (Christiano al., 2017
Ouyang et al. [022)). To mitigate the complexity and instability of this multi- process. family ly of simpler, direct alignment algorithms has emerged (Rafailov et al. 2023 Azar et al. 2023 Meng et al. 2024; Hong et al. 2024). These methods bypass the explicit reward modeling stage by optimizing direct classification style loss on the preference data. However, a critical limitation shared by these methods is their reliance on hard preference labels. This approach models human feedback definitive, binary choice, treating every label with equal and absolute confidence. Consequently, it is highly vulnerable to the significant label noise present in real-world datasets, as standard loss functions can lead models to overfit to corrupted labels (Natarajan et al.) 2013] ZZhang & Sabuncu 2018 Frénay & Verleysen 2013). A simple annotation error, such as an accidental misclick, is given the same weight as deliberate, high-quality judgment. This inability to distinguish between reliable feedback and noise means that the model's 's performance degrades significantly as the error rate increases (Frénay & Verleysen 2013 Gao et al. 2024). In contrast, soft-label approaches that represent preferences probabilisticaly can better accommodate uncertainty in feedback by assigning confidence scores or weights to individual labels (Müller et al. 2019; Song et al. (024). By allowing the learning algorithm to rely on high-quality signals while down- wwiihtingg noise, such approaches provide natural path toward robust preference alignment. This is precisely the perspective adopted by our RE-PO framework, which replaces hard labels with EM-estimated soft confidences.
Learning from noisy feedback. The vulnerability to label noise situates preference alignment within the classic machine learning problem of Learning with Noisy Labels (LNL) (Natarajan et al.
2013 Frénay& & Verleysen 2013). Foundational work in this area, such the Dawid Skene model DDwid & Skene 1979), uses an EM algorithm to simultancously infer true latent labels while estimating annotator reliability. This principle was later extended to pairwise comparisons in the Crowd-BT model (Chen et al. (213)), which jointly estimates item scores and annotator-specific reliability parameters in crowdsourced ranking tasks. In modern LLM alignment, several methods have been proposed to improve robustness to noisy preference data. These can be broadly divided into loss-centric approaches and data-centric filtering strategies. In the first category, ROPO (Liang et al. 2024) proposes an iterative robust preference optimization procedure that jointly applies a tolse-tolerant loss and down-weights (or discards) highly uncertain samples, without relying on external teacher models. rDPO (Chowdhury et al., 2024) constructs an unbiased estimator of the true loss but requires the global noise rate to be known a priori. Hölder-DPO (Fujisawa et al. 2025) introduces loss with "redescending" property, which inherently nullifies the influence of extreme outliers without needing a known noise rate. In the second category, Selective DPO (Gao et al. 2025)
proposes filtering examples based on their difficulty relative to the model's capacity- concept orthogonal to label correctness-usee validation loss as a proxy.
Our proposed RE-PO framework is complementary to these methods. Rather than only modifying
the loss shape or discarding high-loss points, RE-PO explicitly models the data satt-ggenratting process by treating annotator reliability and label correctness latent variables to be inferred. This allows
3

---

Published as a conference paper at ICLR 2026
Table 1: Formulations of the preference loss (Cpret) for prominent alignment algorithms.
Method
Preference Loss Lpret(2; you >- 311
DPO
Rafailov al.
2023
-log
log [a(Mu]z)
log
(a
KeellBa m)
(log mo no(uwlz) - log mo(ue ix) Fvrlyul(x)
IPO
Azar et al. 2023
Forl
2/
SimPO
Meng et al.
2024
log a [Mwl log To (U (x)
liel
log To (o (x) - 7)
CPO Xu et al.
2024
log (B log To (y log Te (e( (r)) log To (2/l
RE-PO assign fine- tiee-graindd, example-specific weights based on a posterior confidence, providing principled way to separate signal from noise.
3 METHODOLOGY
This section details our proposed RE-PO algorithm. We first review the standard DPO framework in Section 3.1 In Section 3.2 introduce a latent-variable model that explicitly distinguishes clean and corrupted preference labels. Section 3.3 then derives the corresponding EM-based update rules for RE-PO. Section 3.4 presents practical mini-bater implementation for RE-PO.
3.1 PRELIMINARIES: DIRECT PREFERENCE OPTIMIZATION
The goal of preference alignment is to fine-tune a language model policy, To, using a dataset of preferences D {ar, Yw 3t)) f=11 ) Direct Preference Optimization (DPO) (Rafailov et al. 2023) offers a simple and effective method for this, bypassing the complex multi-stage pipeline of traditional RLHF (Christiano al. 2017
where response Yw is preferred over t for a given prompt
Ouyang et al. 2022). DPO directly optimizes the policy by minimizing a simple classífication loss:
Tae To (xr
COPO(TA, Tret) = - E ww.))))))))
log a
[Blog
B log
(1)
(Te (r)
Tue xx)
where o.) is the sigmoid function, are a fixed reference policy and is a scaling hyperparameter:
3.2 RE-PO FRAMEWORK: CORE ASSUMPTIONS
critical limitation of DPO is its implicit assumption that all observed preferences in DD correct. In practice, this data is often noisy. To address this, we propose Robust Enhanced Policy Optimization (RE-PO), which is built upon two core assumptions that reframe the problem.
Assumption 1: A Latent noise-free preference. We assume that for each training example (I+, Yw,ss st.s) there exists an underlying moise-free preference, denoted Vie > yt which represents the label we would obtain in the absence of annotation errors. The observed preference Yu,i k M1 potentially corrupted observation of this ground truth. To model this, we introduce binary latent variable E {0, }} for each data point, where z; if the observed label matches the latent noise-free preference and z; otherwise. The reliability of annotator is parameterized by 7/ 2 p(z; = k; = k). Here k; E {l,...,
(provided by annotator k;) treated
.,,K K) denotes the index of the annotator who provided the i-th label, and K is the total number of annotators in the dataset.
Assumption 2: general probabilistic model for preferences. Building on this latent variable model, we must also define the probability of the noise-free preference itself, p(y > y zz. 0). To accommodate preference losses beyond DPO (c.g.. IPO (Azar et al.) 2023)). we instantiate RE-PO with broad class of loss functions, Lpret- For likelihood-style losses, this recovers the original probabilistic interpretation; for non-likelihood losses, it should viewed as an induced robust surrogate. Table provides representative examples used in prominent alignment algorithms.
To connect these diverse loss functions to unified probabilistic view, we draw inspiration from the Boltzmann distribution (Luce. 1959). We define induced preference probability where each
4

---

Published as a conference paper at ICLR 2026
ordering is scored by exp(--ppeil This yields the following definition for the noise-free preference probability:
P(U >- 3z. 0) = a (Cerer( (z, > Vw: 0) (Tr Vw > y1; 8))
where a--) is the sigmoid function. This construction induces well-defined binary probability model for each loss. For instance. with the standard DPO loss, this equation recovers the Bradley-Tere model (Bradley & Terry] 1952) (see Sections A and for derivations). For likelihhoddd losses, this induced probability may differ from the exact original training objective.
3.3
THE RE-PO ALGORITHM VIA EXPECTATION-MAXIIAAI
Based on these core assumptions, we aim find the parameters and 77 that maximize the marginal log-likelihood of the observed data. The probability of a single observed preference is obtained by marginalizing over the latent variable 2;:
p > k; 31 de, 0, n) = plym,i > y1 [T; 0)M, p(u. ywil 0) - -
Directly maximizing E; p(y.( >k; M1.s) is feasible but coupled and non-convex due to the log-sum form. We therefore employ the EM algorithm (see details in Section C.. which provides stable alternating updates with closed-form reliability updates. In this iterative process, the superscript (1) will denote the values of parameters at iteration
E-Step: Inferring label correctness. In the E-step, given current parameters ((0) and () we compute the posterior probability w; that the i-th observed label is correct. This value wy acts as "soft label" or the model's confidence in the data point.
plyws >* ye,lo 010) ((1))
p(y x* Vi M1,l,,, mm? where p(u yt 12, 0(0) and p(u, >" Vw,l,,,,00 0()) can be computed according to equation 2
(t) + p(y1. >" 2, (()) Th
M-Step: weighted parameter update. In the M-step, we update the policy parameters 0 and reliabilities 7) using the confidences w/th computed in the E-step. This step conveniently separates into two independent updates.
First, the policy is updated by minimizing a weighted loss function. As established Assumption 2, our probabilistic model for p(y > 31) can be instantiated with multiple preference losses Speet- making RE-PO practical meta-framewort across common objectives. The general RE-PO loss is:
(f) = Σ
(t) log p(u.. > vz. 0) (1 -u(") log p(u, > ve,,s 0) wi
(5)
Second, the reliability 77 for each annotator is updated to the average confidence of all labels they provided. This has a simple and efficient closed-form solution:
LieT, w(")
((+1) T
(6)
N₂.
Here we define the index set of labeled pairs Ig {i ki = k. and the number of labels as N
3.4 PRACTICAL IMPLEMENTATION WITH MINI-BATCH TRAINING
While the exact M-step updates are clear, performing a full iteration over the entire dataset to re- calculate the annotator reliabilities v after each policy update step can be computationally expensive. To balance computational efficiency and performance, we introduce a more practical online update for 77 using Exponential Moving Average (EMA). Instead of hard assignment, perform soft update based on the statistics from the current mini batch B:
Wh + (1 a)mk + a ΣieBnz,
Nr.5
Here, Ne.8 is the number of examples from annotator & in the current mini-batch, and (0, a momentum hyperparameter. The complete training procedure for RE-PO summarized in Algorithm [
5

---

Published as a conference paper at ICLR 2026
Algorithm 1: Mini-batch Implementation of Robust Enhanced Policy Optimization (RE-PO)
Input: Dataset D { Vur,á Vl,i, { Base policy To. reference policy Ter Preference
loss Cerett Hyperparameters: learning rate X, epochs E, EMA momentum a, initial annotator reliabilities 7) E [0.5, 1] for all & E ( (1....,K)
for epoch= =
E do for batch B do
For each sample i E B, compute wy using current and 7k. via equation 4
Compute the weighted loss (o)- for the batch via equation 5
Update parameters 0 using an optimizer (e.g., A (Loshchilov & Hutter 2019)):
for each annotator present in the batch do
Update T7 via equation 7
end end
10 end
THEORETICAL ANALYSIS OF RE-PO
The robustness of RE-PO stems from its adaptive weighting mechanism. This section first provides an intuitive analysis of these training dynamics and then formalizes this intuition with theoretical guarantees, demonstrating that the RE-PO framework can recover the true reliability of annotators.
At the start of training, when the language model is not yet well-optimized, its predictions are uncer- tain, and the probabilities p(y, >" yn 0) are close to 0.5. The confidence score wy approximates the annotator's 's reliability, 7/;,- The loss then acts a form of label smoothing, preventing the model from being severely misled by incorrect labels early on. As the policy improves, its behavior adapts. For high-quality label, the model predicts high probability for the winning response, and wy approaches 1, causing the loss to function like a standard preference optimization objective. Conversely, w; approaches 0 for noisy label. The loss is then dominated by the (1 w;) term, which flips the optimization direction toward the true preference.
We now formalize the intuition that RE-PO recover the true reliability of annotators. We provide this analysis under an idealized setting: full-batch training where the M-step for the policy parameters 0 is assumed to have converged perfectly. While practical implementation in Algorithm | Juses mini-batch gradient updates (a form of Generalized EM), this idcalized analysis provides strong theoretical justification for our framework.
Consider the update rule in equation 6 defined as an operator (T).). The following theorem is stated the populationn/expecttiio level and shows that iterating this operator recovers the true annotator reliability under the idealized assumptions.
Theorem 4.1 (Identification and convergence of RE-PO). Let 0* be a perfectly calibrated parameter such that the model distribution matches the ground-truth preference distribution. Assume that under annotator-k k'ss data distribution, the model probabiliries are not almost surely equal to 1
Consider the sequence of reliability estimates {Ne 1:20 generated by the population operator "h
(0) (0, the iterates converge the true (++1) Tw (1) (0)). Then, for any initialization Vh
reliability me [2 ki k):
lim 7/
(8)
nh-
The proof is provided in Appendix D finite-sample mini-batch procedure closely tracks this population-level behavior.
In section 5.5 we empirically corroborate that the practical
Practical implications and limitations. The assumption of a perfectly calibrated model in Theorem 4.1 is intentionally idealized: in practice, we apply RE-PO to base models that are not exactly calibrated to the gruund-truth preference distribution. In our experiments, always start from strong instruction-tuned LLMs Wwistral-7B-Instruu Meta-Llama-- -------nBttt-u
and which already display good zero-shot preference behavior. Empirically, do not observe the failure mode suggested by extremely misaligned initialization:

---

In this section, conduct comprehensive set of experiments evaluate the performance of RE-PO. We begin in section3.TTb detailing our experimental setup, including the models, datasets, evaluation benchmarks, and baseline algorithms. In Section [.2 we present our main results. Section [..3 reports additional experiments to evaluate RE-PO's performance on realistic multi-annotator datasets. We then conduct an ablation study in Section 5.4 to analyze the framework's sensitivity its key we provide an empirical verification of our theoretical claims of Published as a conference paper at ICLR 2026
across the broad range of hyperparameters explored in Section 5.4 the learned ni 's remain stable and the downstream performance consistently improves over the corresponding base methods. Fur- thermore, the controlled experiments in Section [.5] where inject substantial synthetic noise into the data, show that RE-POO's estimated reliabilities closely track the ground-truth values, suggesting robustness imperfect calibration in practice. If the base L.M were initialized in highly misaligned regime, the E-step could assign misleadingly high confidence to incorrect labels and RE-PO might fail to effectively denoise the supervision.
5 EXPERIMENTS
hyperparameters. In section 5.5 Theorem 41
5.1 EXPERIMENTAL SETUP
Models and training settings. We use two state-of-the-ar open-source large language models as base models: Mistral-7B Instruct-vD vv.2 .2 and Meta-Llama 83 Instruct. For fine-tuning, we utilize two datasets from the SimPO paper (Meng et al.] 2024), which were generated via on-policy sampling using prompts from the UltraFeedback dataset (Cui et al. (024). The specific datasets are miistra--isstruttt-e-.
cbbck for the Mistral model and for the Llama-3 model As these datasets do not provide annotator-specific in information, we model the preferences as they originate from a single, virtual datasets, we further evaluate RE-
annotator (K 1> In addition to these UttraFeedback-basee PO on the real-world MultiPref multi-annotator preference dataset (Miranda et al., 2024). where per-annotator reliabilities can be explicitly modeled (Section 5.3).
Evaluation benchmarks. We assess model performance on two widely recognized evaluation benchmarks. The first is AlpacaEval (Dubois et al. 2024), an automatic, LLM--ased evaluator that measures model performance by computing the win rate against reference outputs. provides both raw Win Rate (WR) and a Length-Controlled (LC) Win Rate to account for verbosity bias. The second is Arena-Hard (Li et al. 2024), challenging benehmark composed of difficult prompts crowdsourced from the LMSYS Chatbot Arena. It is designed to differentiate tigh-performing models by testing them on complex, real-world user queries. Performance reported as the win rate against suite of other models.
Baseline algorithms. To demonstrate that RE-PO operates versatile meta-framework, benchmark it against four popular direct preference alignment methods: DPO (Rafailov et al.) 2023): IPO (Azar al., 2023), which uses squared hinge loss to optimize preferences; SimPO (Meng et al. 2024), which proposes a simplified, reference-free reward formulation normalized by sequence length; and CPO (u al. 2024). which adds a term directly maximize the likelihood of the preferred response. For each of these baselines, we compare the original algorithm to its RE-PO-enn P---ennhnncedd counterpart (e.g., DPO vs. RE-DPO). In addition, we include robustness-oriented baselines rDPO (Chowdhury et al. 2024) and Hölder-DPO (Fujisawa et al. 2025). as well as simple label-smoothing variants for each method. The results are shown in Table
See Appendix for links to models and datasets. "This is a reasonable simplification. For instance, pool of two annotators with reliabilities and appearing with frequencies PA and pa respectively, can be modeled as a single annotator with an effective
reliability Munited PAлA + PB
7

---

Published as a conference paper at ICLR 2026
Table 2: Performance comparison on AlpacaEval 2 for Mistral-7B- -nn .22 and Meta-Llame - fine-tuned on UltraFeedback-baaee preference datasets. Metrics reported are LC (Length-Controle Win Rate) and WR (Raw Win Rate), both in percentage points. The table presents reference Baselines (bottom) alongside four algorithm families (DPO, IPO, SimPO, CPO). For each family, we compare the Standard implementation, the variant with Label Smoothing (w/ LS), and RE-PO (w/ RE-PO). Bold denotes the best result within each family for given backbone.
Mistral-7B-Instruu
Method
Standard
w/ LS
w/ RE-PO
Standard
w/ LS
w/ RE-PO
DPO
28.5 28.6
29.7 27.5
35.5 33.0
40.8 / 42.9
41.3 / 42.6
44.1 / 46.2
IPO
30.8 28.0
29.7 28.7
/2.9/ 30.5
43.6 41.6
40.3 / 38.2
48.3 / 48.6
SimPO
28.3
26.5
30.4/32.9
14.5/37.1
48.1 38.77
46.97 / 39.4
CPO
26.3 26.44
28.5 / 28.8
27.6 27.8
35.9 40.3
35.3 / 34.8
40.1 / 43.8
Base Model
21.1 16.55
29.7 / 29.9
rDPO
28.1 29.1
37.3 35.4
Höldcr-DPO
30.1 28.6
39.3 / 38.2
Table Performance of DPO and RE-DPO on AlpacaEval 2 when trained on the MultiPref dataset MMranda et al. 2024). Results reported I.C WR (%) for Mi stral Setttraa-----nssuuet-------
and Meta-ll -I, ama 3 -B- Inst.uct.
Method
Mistral-7B- -nnttruet
Llama-3-8B-Instrue
DPO
28.8 / 26.4
36.7 / 39.3
RE-DPO (Ours)
31.8/ / 28.8
41.1 44.4
5.2 MAIN RESULTS
As shown Table 2 our experimental results provide strong evidence that RE-PO generally improves preference- based alignment across objectives, model scales, and datasets. Below we highlight the main empirical findings.
RE-PO general framework. A first observation is that RE-PO behaves as generally effective plug-in robustness layer for a wide range of alignment losses. Across all four objective families (DPO, IPO, SimPO, CPO) and both backbones (Mistral-7B and Llama-3-8BB) the REE-PO-enhanced variant generally matches or outperforms the corresponding standard implementation on AlpacaEval For example, on Mistral-7B, RE-DPO improves LC / WR from 28.5 28.5/28, 28..6 6 35.5/ 35.5/33. /33.0 gain of +7.0 and +4.4 points, respectively). and on Llama-3-8B, RE-IPO improves LC / WR from 43.6/ 43.6/41.6 6/41.6 48.3/ 48.3/48.6 /48.6 (a gain of +4.7 and +7.0 points, respectively). These trends indicate that RE-PO an effective robustness layer for existing preference objectives.
Comparison with label smoothing and robust baselines. Table also compares RE-PO to two natural robustness baselines: label smoothing applied to each preference loss and the recently proposed robust objectives rDPO (Chowdhury et al. 2024) and Hölder-DPO (Fujisawa et al. 2025). Label smoothing sometimes yields modest gains over the standard objective (e.g.. SimPO w/ LS on Llama-3-8B improves LC from 44.5 to 48.1), 1,, but RE-PO is competitive and often achieves the best performance within each family and backbone. For instance, in the DPO family, RE-DPO outperforms both label smoothing and the specialized robust baselines: on Llama-3-8P RE-DPO reaches 44.1 44.1/4.. on AlpacaEval compared to 41.3/ 3/42.6 6 for DPO w/ LS. 37. 37.3/35.4 for rDPO. and 39.3/38. 3/2.2 for Hölder-DPO. These results suggest that explicitly modeling noisy supervision via RE-PO more effective than purely loss-level modifications or global noise-correction schemes.
Qualitative analysis of noisy labels. Beyond aggregate metries, also perform qualitative analysis of the learned confidence scores. In Appendix[F] we present case studies of preference pairs with very low posterior confidence w, RE-PO assigns low confidence to annotations that off-task,
8

---

Published as a conference paper at ICLR 2026
Table 4: Ablation study on the initial annotator reliability (17) and the EMA momentum (a). Results are reported for RE-DPO on Mistral-7B-Inst ruct-vv trained on UltraFeedback-basee data, evaluated on AlpacaEval 2 (LC WR) and Arena-Hard (WR), all in percentage points. The best-performing settings used in our main experiments are highlighted.
Initial 7jo
EMA a
Metric
0.99
(Ours)
0.75
0.55
0.001
0.01
(Ours)
1.0
AlpacaEval2 LCC (%)
30.9
35,5
31.1
31.4
30.9
30.1
35.5
33.4
31.1
AlpacaEval2 WR (%)
31.7
33.0
33.3
32.0
27.8
27.2
33.0
34.8
28.9
Arena-Hard WR (%)
12.3
14.7
12.4
11.8
12.9
13.6
14.7
14.0
12.8
inconsistent with the prompt, or at odds with more plausible alternative response. Together with the quantitative gains in Tables 2 these examples illustrate that RE-PO not only improves benchmark performance but also identifies and down-weights noisy supervision at the example level.
5.3 MULTI-ANNOTATOR EXPERIMENTS ON MULTIPREF
To further evaluate RE-PO under realistic multi-annoten disagreement, we conduct additional experiments on the MultiPref dataset (Miranda et al., (024). large-scale human preference dataset with genuine rater disagreement. The official training split contains 227 unique human annotators. Unlike the SitraaFeddbcck-basee datasets used in our main experiments, MultiPref provides annotator identifiers, allowing us to instantiate an individual reliability parameter 17s for each annotator and to update these parameters via our EM-style scheme.
We train vanilla DPO and our RE-DPO on MultiPref for both Mi InstrInttrutttt. and evaluate the resulting models AlpacaEval 2. As and Met ama -- 8- -
-v
summarized Table 3 RE-DPO consistently outperforms vanilla DPO under this multi-annotator setup: for Llama-3-8B the AlpacaEval LC improves from 36.7 to 41.1 1 and WR from 39.3 to 44.4; for Mistral-7B, LC improves from 28.8 to 31.8 and WR from 26.4 to 28.8. These gains mirror the trends observed in UltraFeedback experiments and show that RE-PO remains beneficial when trained on data with heterogeneous, potentially noisy annotators, rather than a single virtual annotator.
In Appendix E we visualize the learned annotator reliabilities distributions on MultiPref. Experiment results indicate that RE-PO identifies high-reliability majority and a nontrivial tail of downweighted annotators, and that this pattern is robust across different prior settings and backbones. Moreover, to probe the impact of the choice of automatic judge, we repeat the MultiPref evaluation using different LLM evaluator; Appendix G reports these results and shows that the performance gains from RE-DPO are stable across judge models.
5.4 ABLATION STUDY
We conduct ablation studies to analyze the sensitivity of RE-PO to two key hyperparameters: the initial annotator reliability no. and the EMA momentum parameter a. All experiments performed using RE-DPO on the Mistral-7B-Instmutt-0022 model. The results are summarized in Table
Efffet of initial r7o- The initial reliability Th sets the model's prior belief about the correctness of the labels in the dataset. As shown in Table the model's performance is best when 7Jo is set to 0.9, which was the value used in our main experiments. An overly optimistic initialization (e.g., 7/u 0.99) can cause the model to trust noisy labels too strongly at the beginning of training, hindering the denoising process. Conversely, a pessimistic initialization (e.g.. N7u 0.55) treats the data as highly unreliable from the outset, which can slow down the model's ability to learn the underlying noise-free preference. An initial value of 0.9 appears strike the right balance, starting with reasonable assumption of data quality.
Effect of EMA parameter a. The EMA parameter a governs the update rate of the annotator reliability scores. balancing the influence of historical estimates against new information from the current mini- minh. Our experiments confirm that the optimal performance is achieved with a = 0.1. The model shows considerable sensitivity this parameter. A very small a (e.g., 0.001) makes the

---

Published as a conference paper at ICLR 2026
1.0
(GPT-4) n (RE-FO)
Annotator Rellabiitt
0.8
Rellability
0.6
0.6
0.4
Annotator 1 (GPT-4o)
0.7
Annotator FO)
Annotator 2 (GP1-40)
Annotator n (RE PO)
0.0
0.2
0.2 Noise Level Applied Annotator
0.8
Noise Level
(a) Single-annotator setting.
(b) Two-annotator setting.
Figure 2: Empirical verification of annotator reliability estimation under controlled synthetic noise. Ground-truit reliability (17 GPT-4o) is established using GPT-4o's labels on UltraFeedbackdder preference pairs, and different reliability levels are simulated by injecting synthetic noise into copies of the dataset. In the single-annotator setting (a), a single annotator's 's dataset is perturbed with varying noise rates. In the two-annotator setting (b), Annotator uses the original data with no added noise, while noise is progressively added to Annotator 2's data. The plots compare ground-truth reliabilities (solid lines) with RE-PO-estimated reliabilities (dashed lines), showing that RE-PO closely tracks the true reliability in both scenarios.
reliability updates exceedingly slow, preventing the estimates from adapting to the model's evolving understanding of the data. On the other hand, very large (e.g., 1.0) makes the updates highly volatile, as the reliability score becomes dependent solely on the samples the current mini-batch.
5.5 EMPIRICAL VERIFICATION OF THEOREM
We conduct controlled experiments to verify Theorem Our setup is designed align with the theorem's ' assumption of a perfectly calibrated model, for which we use a -call-scale base model, Owen2. 5-0. 5B- Instruct. to ensure fast convergence. To simulate annotators with varying levels of reliability, we create distinct copies of the UltraFeddbcck dataset (Cui et al. 2024) for each annotator and inject controlled degree of synthetic noise into their respective dataset.
We test two scenarios, with results presented in Figure 2 (a) Single Annotator: A single annotator whose dataset is modified with a synthetically controlled noise rate. (b) Two Annotators: A scenario with two annotators, where Annotator serves as a baseline using the original data without added noise, while the dataset for Annotator is injected with progressively increasing noise levels.
The results in Figure show that the estimated reliability (RE-PO) closely tracks the ground-truth (GPT-4o) in both single-annotator (Figure 2ab and two-annotator (Figure 2b) settings. Notably, in the two-annotator experiment, RE-PO successfully identifies the stable reliability of the baseline annotator while accurately tracking the declining reliability of the noisy one. Although the theorem assumes a perfectly calibrated model, these experiments demonstrate that RE-PO's reliability estimates remain accurate and stable even when the underlying model only approximately calibrated and trained under realistic noise patterns, mitigating concerns that early miscalibration would systematically doighweight correct labels.
CONCLUSION AND FUTURE WORK
In this paper, we introduce Robust Enhanced Policy Optimization (RE-PO), a framework designed to address the critical challenge of aligning LLMs with noisy human preference data. Our approach employs an Expectation-Maxiine
algorithm to infer the reliability each preference pair, treating labels as soft, dynamic weights rather than fixed ground truths. Across the evaluated objectives and backbones, RE-PO improves multiple of alignment algorithms, achieving substantial gains over their corresponding standard versions. A natural limitation of our current theory is the assumption of a perfectly calibrated model: extending convergence guarantees to settings where the base model is significantly misaligned remains important future work.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This research was supported by National Natural Science Foundation of China 02240615,,623254444 Zhongguancun Project C20250301, Tsinghua University Initia- tive Scientific Research Program, TsinghuaEfort Joint Research Center for EAI Computation and Perception, Beijing National Research Center for Information Science, Technology (BNRist), Beijing Innovation Center for Future Chips, and State Key laboratory of Space Network and Communications.
REPRODUCIBILITY STATEMENT
To facilitate reproducibility, we provide (i) full methodological details and derivations in Sec- tions and 4 with additional derivations and proofs in Appendix A (ii) experimental setup, implementation details, and additional empirical results Section and the appendices (includ-
ing runtime/resources in Appendix H and I (iii) public artifacts, including the project webpage (repo-alignment.gituu github.io) and the code repository [gthhub.om//KiooyggoollllLLL

---

Published as a conference paper at ICLR 2026
REFERENCES
Mohammad Gheshlaghi Azar, Georg Ostrovski, Remi Munos, and Bernardo Pires. A general theo- retical paradigm for preference-basee reinforcement learning. arXiv preprint arXiv:2310. /2036, 2023.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs. i. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
Xi Chen, Paul N Bennett, Kevyn Collins- Collins-Thompson, and Eric Horvitz. Pairwise ranking aggregation in a crowdsourced setting. In Proceedings of the sixth ACM international conference on Web search and data mining, pp. 193-202, 2013.
Sayak Ray Chowdhury, Anush Kini, and Nagarajan Natarajan. Provably robust dpo: Aligning language models with noisy feedback. arXiv preprint arXiv:2403.0044 2024.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems 30, 2017.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with scaled ai feedback, 2024.
2211 O 01377
A Philip Dawid and Allan M Skene. Maximum likelihood estimation of observer error-rates using the em algorithm. Journal of the Royal Statistical Society: Series C (Applied Statistics), 28(1): 20-28, 1979.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint M.Xiv:2404.0447555 2024.
Benoît Frénay and Michel Verleysen. Classification in the presence of label noise: a survey. IEEE transactions neural networks and learning systems, 25(5)8445-869, 2013.
Masahiro Fujisawa, Masaki Adachi, and Michael A Osborne. Scalable valuation of human feedback through provably robust model alignment. arXiv preprint arXiv:2505. 17859, 2025.
Chengqian Gao, Haonan Li, Liu Liu, Zeke Xie, Peilin Zhao, and Zhiqiang Xu. Principled data selection for alignment: The hidden risks of difficult examples. arXiv preprint arXiv:2502.09650. 2025.
Yang Gao, Dana Alon, and Donald Metzler. Impact of preference noise on the alignment performance of generative language models. arXiv preprint arXiv:2404.09824 2024.
Jiwoo Hong, Noah Lee, and James Thome. Orpo: Monolithic preference optimization without reference model. arXiv preprint arXiv:2403.07691, 2024.
Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From crowdsourced data high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprini arXiv:2406. 1939, 2024.
Xize Liang, Chao Chen, Shuang Qiu. Jie Wang, Yue Wu, Zhihang Fu, Zhihao Shi, Feng Wu, and Jieping Ye. Ropo: Robust preference optimization for large language models. arXiv preprint arXiv:2404.04102. 2024.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations (ICLR), 2019.
R Duncan Luce. Individual choice behavior: A theoretical analysis. John Wiley & Sons, 1959.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-
free reward. Advances in Newral Information Processing Systems, 37:124198-1244 2024.
12

---

Published as a conference paper at ICLR 2026
Lester James V. Miranda, Yizhong Wang. Yanai Elazar, Sachin Kumar, Valentina Pyatkin, Faeze Brahman, Noah A. Smith, Hannanch Hajishirzi, and Pradeep Dasigi. Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedback. arXiv, abs/2410.19133, October 2024.
Rafael Müller, Simon Korblith, and Geoffrey E Hinton. When does label smoothing help? Advances in neural information processing systems, 32, 2019.
Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. In Advances in neural information processing systems 26, 2013.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems 35, pp- 27730-27744, 2022.
Rafacl Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems 36, 2023.
Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. Preference ranking optimization for human alignment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 18990-18998, 2024.
Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of 11m performance in machine translation. arXiv preprint arXiv:2401.0844 004417. 2024.
Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems, 31, 2018.
Daniel M Ziegler, Nissan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint Xrviv: 1909 000099........... 2019.
A DERIVATION OF GENERAL PROBABILISTIC MODEL.
Here we provide the detailed derivation for equation 2 For a given prompt x and candidate responses Yw Yl we assume the probability of the ground-trut preference yo >* yi is proportional to exp(-Eper(2, Vie > 31)). That is:
ply
in (r, exp(--prer(x,, yhe v1)
(8)
Similarly, for the inverse preference:
p(u yo 0) exp(- y > yw))
99)
Since yo >* y and z/ >* you are the only two mutually exclusive outcomes for a binary preference, their probabilities must sum to 1. Using the property of normalized probabilities from proportional relationship, we have:
EppC-Cper(2, Bhe > 91))
ply >* 11/2. 0) exp(- Coret Uu > 91)) exp(pp (xxp(-Cpect(ae
(2, y1 yw))
1+
> Bw) Epror (I, yw > 21))
= a (Eperlz, (z, > yw) - Epeerlx, Bw / 11))
The last line is the General Probabilistic Model in equation 2
13

---

Published as a conference paper at ICLR 2026
B CONSISTENCY WITH BRADLEY-TERK' MODEL FOR DPO
We show that equation is consistent with the Bradley-Tery model when applied to DPO. The DPO loss for preferred pair (Yw: yr) given prompt is:
To (z)
(a [x) Tiee (Br [1)
(x, > yt) = log Glo
3 log
(10)
Tret (yw
Let (ar, Uw w) = ,log na (b(ul))) FaelUa m) 3 log Red (an((n)))
(x)
Then, we can write:
LDPO yhe > W) = log a (S ywo v1)
CDPO (2r, Mr yiu log (S y1 Vo)) log (0-S((z, Mers u1)
Substituting these into our general probabilistic model (equation 2
plyn >- y1lor, 0) (LDoo y1 yve LDPO (z;, yhe > v1)
= a (log o (S((,, Fw 31)) log (- Yues 81 U1)))
o(S(x,, Mors 21))
log 1 oS Mer, 91))
(SS( yw-)1))
= =o Log (Ty (x)
T0(31 (x)
3 log (Tre(y) (x)
Tre((ywllz)
This resulting probability exactly matches the form of the Bradley-Terry model (Bradley & Terry 1952 for preferences, where the implicit reward of response y r(z, y) B log x.
m
C DERIVATION OF THE RE-PO EM ALGORITHM
The primary objective of Robust Enhanced Policy Optimization (RE-PO) is to find the model parameters 9 and the vector annotator reliabilities r that maximize the log-likelihood of the observed data. The observed data consists of prompts, chosen and rejected responses, and the annotator's 's index, denoted as X = D = { Mooii Ut,i, k:))
The likeliiel function is given by:
C(0. n) = Σ log [p(uw,s >* 11,, 0)T,. + plur, >- Ver 1 0) 7k.)
(11)
There is sum inside the logarithm, which makes direct optimization coupled and non-convex. The Dnpectation-Maximzaaa (EM) algorithm is an iterative procedure designed to solve such maximum likelihood problems with latent variables by alternating between an Expectation (E) step and a Maximization (M) step.
C.1 DERIVATION OF THE Q-FUNCTION (THE E-STEP)
The EM algorithm simplifies the problem by working with the complete data, (X, 2), where Z = ( the set of all latent variables.
The complete-data log-likelihood, L., assumes that we know the values of all latent variables z4:
C.(0, ((00m,X,2))
Σ (z: log (p(ww,s >" y1,s/20, 0ymm,1 + (1 - 2:) log [p(y,, yo, 20 0) - T7,,
(12)
This form is tractable because the logarithm acts on products, which can be separated into sums.
The core idea of EM is iteratively maximize the expectation of the complete-data log- likelihooodd This expectation, known as the Q-function, is taken with respect to the posterior distribution of the latent variables Z, given the observed data X and the parameter estimates from the current iteration, (0(), (()).
14

---

Published as a conference paper at ICLR 2026
Q(e. mlo 17))) ,0 (C,(0, 77: X)]
(13)
To compute this expectation, we push the expectation operator inside the summation. The only random variables Le are the 24-
Q(0, ml0 n10) = [B[zs] log (p >" v1,10)mm,] + (1 - E[zi]) log [p(u,, >" you (0) T/s;))
)))
(14)
The term [[2,] is the expectation of the binary variable 2, which is its posterior probability of being This probability is conditioned on the observed data and the parameters from the current iteration t.
We denote this posterior probability w!") which is computed in the B-Step:
m (1) E(2, X; fit) (0
pp(= = 18 bh; 31 T; (()) 17 (0)
p(Uwi > ke M4 = 1, Fij 0(0) )p(2; 2; = 1 his, (is, (1)
plyaa bh 31,4 F4 ((t), 17(1))
p >" Vt 1x,, O10)
}71s; (1)) (e)
(15)
p(u >n* 31. a 000) ym) (1) p(y1, ,w (1)) Tik
(0)
Substituting /! into the expression yields the final form of the Q-function:
Q ml0(1) m)0))) )))0
(t) log(p(yw, >" M1,i00mm. + (1 (1-- log(p(un >" vw,il0)(1 nT.)))
(16)
C.2 DERIVING THE RE-PO FRAMEWORK (THE M-STEP)
The goal of the M-Step is to find the parameters for the next iteration, (0(++1), (t+1)), by maximizing the Q-function that was constructed using the parameters from the current iteration t.
(0(++)) (t+1)) = arg max Q( m)((1) (*)
(17)
0.7
To perform this maximization, we can first expand the Q-function by separating the terms involving the policy 0 from those involving the annotator reliabilities 17.
N
Q(0, (0, n ())
w log p(ue, >" 01,100) + (1 w. (t) log,( >* Bur,i(0) 10)
=
Depends only on
w (t) log T7k + + (og Tha))
(18)
Depends v
Because the Q-fucction is separable into two independent parts, we can maximize each part separately find the new parameters.
To find the optimal ((++1), hold n fixed and maximize the terms in the Q-function that depend 0:
g(++1) arg max
(t) log p(uees X "1,,00) + (1 -w: lol)loopuus >" 1w,:
w (e) log Llves X" 10,10) + (1 we(") pluns x" (on:1011 (19)
arg min
15

---

Published as a conference paper at ICLR 2026
The expression inside the arg min is precisely the weighted RE-PO loss function, (8). This establishes that the M-step for the policy parameters is equivalent to minimizing this weighted loss,
using weights w
(1) from the E-step.
((+1) for a specific annotator k we hold fixed and maximize the terms in the To find the optimal 7h
Q-function relevant 7Jk- These terms only involve samples labeled by annotator & (where K; k):
1/1
(1+1) arg max Th 1
(9) W
log T/ (1 - w log(1 m))
(20)
To find the maximum, take the derivative with respect y/s and set it to zero:
log Th + (1 - w log(1 176)) 0 w
(21)
OTT skk=
(0)
1
w!!
(22)
1
7K
1
Σ
w!t)
1
(1- (1)
(23)
i:de
1
W
N
w
(t)
(24)
Tk
k
where N is the total number of annotations provided by annotator k. Cross- Cross-multiplying gives:
(1 - mk) Σ
Wi
(t)
1k
Nk Σ
(1) Wi
(25)
i: =
iske =
Σ =ky=
w
7k
W (t)
N 7k
W (t)
(26)
(t)
N
(27)
This yields the intuitive and closed-form update rule for the reliability at iteration + 1:
((+1) Eih.
w
U
(28)
N
This shows that the updated reliability for annotator simply the average posterior probability (or confidence) from the previous iteration that their labels were correct.
D PROOF OF THEOREM 4.
In this section, we prove Theorem 4.1 at the population level under the idcalized full-batch setting.
Definition of the Update Operator. Recall the M-step update for annotator reliability (Eq. 6):
Mh No EiEr, (T)... For theory, we define the corresponding population operator
p"1
T (T( A E
k; k
p
p1 1))
where p* = p(yn > y1 .:) for a random sample from annotator-k: ss data distribution.
Step 1: Fixed Point Property. Let v - E|z k; = k] be the true annotator reliability and let obs, A (bwes b Vt, (x)) denote the observed preference event. At the true parameters,
w,(nt) P(2; = 1 obs;, 0* mTa E(2; obs:]-
Applying T& and using the law of total expectation:
To (1)) E|Es2 obs;] = k] = [2 h; k] T/p-
Hence, 2/6 is fixed point of the population operator.
16

---

Published as a conference paper at ICLR 2026
Step 2: Global Convergence. (1) = 18 (og(pppp")) + (1 p-pppp' (1 -1)) h; k]
Consider the population observed-data log-likelinn likelihood
Differentiation gives
(())) 771 11)
(Th 1)
stationary points of & exactly fixed points of Tи. The second derivative is
(2p" - -1)a
(n) - E
k: = k
(p + (1 (0-pp)(L (1 - n))
If pz A 0.5 with nonzero probability, then (n) < for all ry E (0, so la is strictly concave and
has a unique stationary point By Step I, th is stationary point; by uniqueness, m%. Therefore, the full-batch EM iterates converge to the true reliability:
lim 1/K (-000
E VISUALIZATION OF ANNOTATOR RELIABILITY
To better understand how RE-PO behaves on truly multi-annotator dataset, we analyze the distri- bution of the learned annotator reliabilities {fk ]k= on MultiPref. For each annotator RE-PO maintains posterior estimate îi after EM-style updates over the full training run. Figure summa- rizes these posterior reliabilities for different backbones and prior settings.
The figure is organized row), and columns correspond to different choices of the prior mean no E {0.80, 0.90, 0.95, 0.99). Within each panel, we plot a histogram of the posterior means fię and report the empirical mean μ and standard deviation of the în values across all 227 annotators.
grid: rows correspond to the base llms on the top row and Llama-33 -aa-a------Insttreee on the bottom
Several consistent patterns emerge across subplots. First, in all settings the mass of the distribution concentrated near high reliability (îj close to 1), but there is persistent tail of annotators with substantially lower îik- This tail appears in every column, indicating that RE-PO is not simply reproducing the prior: labels are systematically inconsistent with the model's evolving preferences are pulled down and assigned clearly lower reliability.
when the prior mean no is large (e.g., 0.95 0.99), annotators whose
Second, moving from left to right across columns (increasing 17) mainly affects the concentration of the bulk mass rather than eliminating the low-reliability tail. As VJo increases, the main peak of the histogram shifts closer to and becomes narrower (smaller σ), reflecting a stronger prior belief that most annotators are competent. However, the tail of low-îı; annotators remains visible, showing that the data is still informative enough for RE-PO to downweight noisy annotators even under confident prior.
Third, comparing the two rows reveals mild backbone effect. For the same prior 7/u the Llama-3-8B panels (bottom row) typically exhibit a more peaked distribution with slightly smaller spread than the corresponding Mistral-7B panels. This suggests that, on MultiPref, the Llama-based models induce slightly more internally consistent preference signal: annotators are more cleanly separated into high-reliability majority and smaller group of downweighted raters.
Overall, these histograms support our qualitative claim about RE-PO on multi-annotator data: (i) the method does not collapse all annotators to uniform reliability level, but instead identifies and downweights nontrivial fraction noisy annotators; and (ii) this behavior is robust across reasonable choices of the prior mean 7)u and across different backbones. These observations complement the quantitative gains reported in Table 2, providing direct evidence that RE-PO is exploiting genuine multi-annotator disagreement rather than overfitting a particular prior model.
17

---

Published as a conference paper at ICLR 2026
= 0.80
0g = 0.90
0,- 0.95
= 0.99
0.813
μ. 0.868
0.933
μ- 0.970
75
0.057
18 Pentrren
50
Annelators
75
j. 0.811
p 0 901
j 0.944
=.= 98
98 E Luume
Annotator reliability
Figure Histograms of posterior annotator reliabilities 7)e on the MultiPref training split. Rows correspond to backbones dffssral-7B-Instruutt--... top; LJamm-3-8B-Instruu correspond to different choices the prior mean JJo E (0.80, 0.90, 0.95, 0.99) (from left to right). Each panel reports the empirical mean and standard deviation of {1 )
bottom). Columns
1==1
F QUALITATIVE ANALYSIS OF NOISY PREFERENCE LABEL
In this appendix, we present qualitative case studies of preference pairs that our Robust Enhanced Policy Optimization (RE-PO) model assigns very low confidence to. These examples illustrate the kinds of inconsistent, noisy, even reversed labels that appear in real-world preference datasets, and how RE-PO effectively downweights them during training.
We use the mi mistral .2 fine-tuned with RE-DPO on this dataset.
feedback dataset, and the model is
F.1 EXAMPLE: MISALIGNED LABEL A TOPIC CLASSIFICATION TASK
T a representative example from topic-classffictioon prompt. The task specification is extremely constrained: the model must output single integer between and 14, corresponding to specific category, and must not produce any additional text.
The chosen response in the dataset begins with the correct label (13), but then continues with long sequence of additional Problem/Solution pairs and explanations, many of which are (i) clearly outside the scope the original prompt and (ii) factually or categorically incorrect (for example, misclassifying buildings companies or natural places). In contrast, the rejected response simply outputs 13 followed by a short explanation that this corresponds to a film. Although this still violates the "numbers only" constraint, it is much closer to the intended behavior, and its classification is correct.
Despite the dataset labeling the first response as the preferred one, RE-PO assigns low posterior confidence to this label (approximately w; = 0.037 in this case), treating as a highly suspicious or potentially reversed preference.
F.2 EXAMPLE: NOISY LABEL IN A SHONOUN-PHRARE IDENTIFICATION TASK
Table shows an example from a pronoun phrasse identification task, where the model must output noun phrase corresponding to given pronoun in the sentence. The prompt ends with the sentence "the man i fifted the boy onto his shoulders.. and expects a single noun phrase referring to the pronoun "his."
The dataset marks as chosen a response that starts with the plausible answer "the man't shoulders but then continues with long list of additional, mostly unrelated sentence-phrass pairs that go far beyond the requested output format. contrast, the rejected response simply re- turns concise noun phrase: "the man's s shoulders (or the boy onto the man' shoulders). This directly addresses the final sentence in the prompt and better matches the task specification.
18

---

Published as a conference paper at ICLR 2026
Table representative low-confidence preference pair from a topic classification task. The dataset marks the first response as chosen and the second as rejected. RE-PO assigns a very low confidence this label, indicating that the observed preference is likely incorrect.
Prompt ID
Prompt
which is the body of docurent.. Your job is to classify fy the topic of the document into these categories: Company, 2) Educatiional Institution, Artiis, Athlete,, 5)0ffice Holder, 6)Ma of transportation, Buildin, NaNutual place, 9)village, Anima,, 11)Plant, Album, Fimmi writtitten work. Don't generate anything apart from numbers 1-14.
Detailed Instructions: this task, you are given
See one example below:
Problem: Text: Abbott Farnham Abbott Limited was British coachbuilding business based in Farnham surrey trading under that name from 1929. A majer part of their output was under sub-contract to motor vehicle manufacturers. Their business closed in 1972.
solution:
Explanation: Here, the given text is about British coachbuilding business company. Hence, it is correct answer.
Problem: Text: The Concubine (RR: Hugoong: Jewangui Chub; literal Royal Concubine: Concubine to the King) is a 2012 South Korean historical film directed by Xim Dae-seung, Set in the Joscon Dynasty centers around Hwa-yeon (Jo Yeo-jeong), who becomes royal concubine against her will, Xwon-yoo (Kim Min-joon), a man torn between love and revenge, and Prince Sung-won (Kim Dong-wook), who has his heart set on Hwa-yeon despite the many women avai lable to him. Solution:
title:
Chosen response
#xplanation: The text describes a South Korean historical film. Wherefore, it is film.
Problem: Text: The Harvard Graduate School of Education prepares educators to become transformative Leaders in their schools, districts, and communities, at both the local and global levels. Its research, taught in an engaging, student-centered environment, advances the understanding of teaching and learning better prepare students for success a diverse and interconnected world. Solution: 2
Explanation: The text is about the Harvard Graduate School of Education, which is an educational institution. (truncated; additional Problem/Solution pairs and explanations, several of which are risclassified unrelated to the original prompt)
Rejected response
13
The text describes a South Korcan historical film. Therefore, it falls under the Film category.
Explanation:
RE-PO posterior confidence for the observed label
~ 0.037
RE-PO assigns a very low posterior confidence to the observed label (w, se 0.011), again indicating that the dataset's preference is likely noisy reversed and should be heavily downweighted.
19

---

Published as a conference paper at ICLR 2026
Table A representative low-confidence preference pair from a pronoun-phrase identification task. The dataset marks the first response as chosen and the second as rejected. RE-PO assigns very low confidence this label, indicating that the observed preference likely incorrect.
Prompt ID
Prompt
Detailed Instructions Indicate noun phrase from viven statement wh ch mentions a pronoun.
the actress used named terps chore, but. she changed to tina few years ago, because she figured was too hard to pronounce. it was too hard
sam tried to paint a picture of snepherds with sheep, but they ended up looking more like golfers. they ended up looking
like golfers.
the 11 lifted the boy onto his shoul ders.
Chosen response
the man's shoulders the singer's s voice hoarse from all the performing ng, but he gave lent concert nonetheless.
the singer's s hearse voice
the professor explained the concept relativity the class in great detail.
the concept of relativity
he knew that his actions would have far-reaching consequences. the consequences of his actions
she couldn't believe her eyes when she the elephant in the circus.
the elephant the circus the sun began to set over the calm and tranquil lake. the setting over the Lake
(truncated; the full response cont continues nues with many additional, most.ly unrela unrelated sentence--phrase pairs)
Rejected response
the man's s shoulders ders (or the boy onto the man's s shoul ders)
RE-PO posterior confidence for the observed label
RJ 0.011
G ADDITIONAL RESULTS ON MULTIPREF
evaluated RE-DPO on the MultiPref dataset (Miranda al. 2024) using Al- pacaEval 2 as the automatic judge. For completeness, Table reports updated results when using DeepSeek v3. 2-Exp the evaluator. The trends match main findings: RE-DPO consistently improves over vanilla DPO on both backbones when trained on genuine multi-annotator preference data.
In Section 5.3
H RUNTIME OVERHEAD OF RE-PO
We additionally measure the computational overhead introduced by RE-PO's EM-style reliabil- ity updates. For this purpose, compare the wall-clock training time of each base prefer- ence objective with its RE-PO-enhaneed variant on both Mistral- Ins-I--tttttttto and Meta-I. ama - - Instructt
20

---

Published as a conference paper at ICLR 2026
Table Performance of DPO and RE-DPO on AlpacaEval 2 when trained on the Multi- Pref dataset (Miranda et al., 2024) and evaluated with DeepSeck-v3.2-Exg as the judge model. Results are reported LC / WR (%) for Mistral-7B-Instrueeee Meta- ama 3 IB-Instruct.
and
Method
Mistral-7B-Instrucc
Llama-3-8B-Innse
DPO
/
36.3 / 38.5
RE-DPO (Ours)
32.9 30.3
40.4 / 42.7
Experimental setup. All runs conducted single machine equipped with 8x NVIDIA A800-SXM4-40GB GPUs, using the same software stack and with no other jobs running concurrently. For each backbone and each preference objective (DPO, IPO, SimPO, CPO), train both the base method and its RE-PO-enhanced counterpart on the UlrraFeedbcck-bace preference datasets described Section S.T To isolate the cost of EM-based reliability updates, we keep all optimization hyperparameters fixed across base vs. RE-PO runs (optimizer, learning-raie rate schedule, global batch size, gradient accumulation, and number of training steps).
Runtime overhead. Table T reports wall-clock training time in seconds (mean + standard deviation over three seeds), where each cell shows "Base / RE-PO" for a given all eight configurations, the average slowdown about 11%, but overhead varies by objective; SimPO has the largest overhead (about 32%-40%). For example, on Llama-3-8B, IPO takes 8571 +: 20 seconds vs. 9747 18 seconds with RE-PO; on Mistral-7B, SimPO takes 5383 10 vs. 7557 + 23 seconds with RE-PO. In few configurations (e.g., DPO and CPO on some backbones), the measured wall-clock time of the RE-PO variant is slightly lower than that of the base method, which we attribute to run-to-run system variance rather than an intrinsic speedup, since RE-PO only adds lightweight scalar reliability updates on top of the base objective.
pair. Across
Table 8: Wall-clock training time (in seconds) on UltraFeedback-bne based preference datasets for base preference objectives and their RE-PO-enhanced variants. Each cell reports mean # standard deviation over three runs, formatted as "Base / RE-PO". All runs use the same 8xNVIDIA A800- SXM4-40GB hardware and identical optimization hyperparameters: only the objective (base vs. RE-PO) differs.
Method
Mistral-7B (Base / RE-POO
Llama-3-8B (Base / RE-PO)
DPO
7138 + 21 6587.8 + 2.2
7089 + 12 6837 + 21
IPO
7999 = 10 9043.0 = 2.8
8571 + 20 / 9747 = 18
SimPO
5383 + 10 / 7557 + 23
5384.2 + 9.9 / 7117 + 16
CPO
5868 12 / 5862 + 20
6503.4 + 8.2 6337 11
I RESOURCES
Models:
Mistral--B--Inttrudt-e
http:///pggiiiree.oooooottaaaaaa Liama-3-BB-Instruuct:
Datasets:
m mistral-instros--ubaafeeeeallll
21

---

Published as a conference paper at ICLR 2026
BttSs///aaaallcceeee///aaaa///ep multipref: httpss//wwgggnnneee....//tttttta |/tp://
THE USE OF LARGE LANGUAGE MODELS
We employed large language models (LLMs) assistive tool during the preparation of this work. Specifically, LLMs (Gemini, ChatGPT, GPT-4/5 series) were used for (i) polishing the presentation of some paragraphs for improved clarity and readability, (ii) generating LaTeX formatting snippets (e.g., table/figure environments), and (ii)) providing feedback on alternative phrasings of technical explanations. The core research contributions, including problem formulation, algorithm design, theoretical analysis, and all experiments. were fully developed and conducted by the authors without the use of I.L.Ms. The LLM usage was limited editing support and did not influence the research ideas, methodology, or results.
22


Reviews:
[{"review_id": "Eyr2VZGuLv", "reviewer": "Reviewer_qRQj", "summary": "This paper introduces Robust Preference Optimization (RPO), a general framework for aligning large language models (LLMs) with noisy human preference data. Unlike standard RLHF or DPO methods that assume all feedback is correct, RPO explicitly models the correctness of each annotation as a latent variable. Using an Expectation–Maximization (EM) approach, RPO jointly estimates annotator reliability and model parameters. In the E-step, it computes sample-wise confidence weights reflecting annotation trustworthiness; in the M-step, it updates model parameters and annotator reliabilities via weighted loss optimization.", "questions": "See Weakness", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. RPO provides an unification of robust preference optimization via probabilistic modeling and EM inference. \n\n2. The paper includes convergence proofs for the EM updates and demonstrates that annotator reliability can be recovered under realistic assumptions. \n\n3. Experiments on multiple LLMs (Mistral, Llama-3) and datasets (AlpacaEval 2, Arena-Hard) show RPO outperforming baselines.", "weaknesses": "1. Although RPO models annotator reliability, the main experiments assume a single virtual annotator (K=1). The paper would be more convincing if it demonstrated effectiveness on datasets with genuine multi-annotator disagreement.\n\n2. The EM-based procedure may introduce extra computational costs and slower convergence. While the paper mentions using EMA for efficiency, it does not quantify runtime overhead or training stability compared to standard DPO.\n\n3. While related work such as rDPO and Hölder-DPO is cited, these methods are not included in the empirical comparison. Adding such baselines would clarify the relative robustness advantages of RPO.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uR5iZN9kYV", "reviewer": "Reviewer_Q5V8", "summary": "This paper introduces Robust Preference Optimization (RPO), a general framework designed to address preference noise commonly encountered in large language model (LLM) alignment. Existing preference-learning methods such as DPO and IPO typically assume clean and consistent preference labels, which rarely hold in real-world settings where human feedback is diverse and occasionally erroneous. RPO treats preference labels as latent variables and employs an Expectation–Maximization (EM) procedure to jointly estimate the “soft label” confidence and annotator reliability during training. This enables adaptive reweighting of noisy data. Furthermore, the authors propose a unified probabilistic formulation that generalizes multiple preference objectives under a single likelihood-based framework, thereby enhancing methodological coherence and extensibility.", "questions": "see the Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper tackles a realistic and important issue—label noise in preference alignment—that directly affects the reliability of RLHF-style training.\n2. Modeling annotation noise via latent variables and integrating EM into preference optimization is a well-grounded and elegant idea.\n3. The probabilistic unification of different preference losses provides conceptual clarity and could potentially bridge diverse alignment approaches under one framework.\n4. The experiments demonstrate consistent robustness improvements across several noisy-preference setups, validating the framework’s effectiveness.\n5. The paper is generally well written, with clear mathematical derivations and a reasonable balance between theory and empirical validation.", "weaknesses": "1. All experiments are conducted under a single-annotator noise setting. Incorporating multi-annotator scenarios would make the robustness claims more realistic.\n2. Methods such as ORPO and rDPO share similar robustness motivations but are not included in the empirical evaluation. Adding these baselines would strengthen the comparison.\n3. Using GPT-4o preferences as ground truth could introduce systemic bias. It would be better to include multiple evaluators or human assessments for verification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2FI1KlhhFh", "reviewer": "Reviewer_5BHp", "summary": "This paper introduces RPO, a new framework for aligning LLMs that is designed to be resilient to noisy preference feedback. RPO addresses this by reformulating the alignment problem using an Expectation-Maximization. The core idea is to treat the \"true\" underlying preference as a latent variable. In the E-step, the framework uses the current model parameters to infer the posterior probability (a \"confidence score\" $w_i$) that each observed preference label is correct. In the M-step, these confidence scores are used as adaptive weights in the loss function to update both the LLM policy and a set of learned annotator reliability parameters $\\eta_k$. The authors present RPO as a \"meta-framework\" capable of making existing alignment algorithms (like DPO, IPO, SimPO, and CPO) more robust. They provide a theoretical proof that RPO can converge to the true annotator reliability under the assumption of a perfectly calibrated model.", "questions": "- Could the authors please provide experimental comparisons against the highly relevant baselines mentioned in their own related work, specifically rDPO and Selective DPO? Furthermore, how does RPO compare against simpler robustness-enhancing techniques like applying label smoothing to the standard DPO/IPO/SimPO/CPO losses? \n\n- Can the authors provide 2-3 qualitative examples of preference pairs from the UltraFeedback dataset that were assigned very low confidence scores $w_i$ by the RPO-DPO model?\n\n- I think the authors should add discussions around poor alignment of the initial model", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- I like the use of EM to jointly infer label correctness and optimize the model. I find it an elegant and principled approach.\n\n- The authors provide a generalization of RPO into a meta-framework. The formulation (Eq. 2) that connects arbitrary preference loss functions $\\mathcal{L}\\_{pref}$ to a unified probabilistic model allows RPO to be easily applied to a wide range of existing methods\n\n- The experiments show that RPO consistently improves performance across different PO algorithms and two different base models. The reported gains are substantial. \n\n- The paper provides theoretical analysis and empirically show that the RPO framework can accurately estimate the true reliability of different datapoints", "weaknesses": "- The paper positions RPO as a solution for robust alignment but doesn't compare it against other methods designed for the exact same problem. The related work section explicitly mentions rDPO and Selective DPO, but they are absent from the main results in Table 2. The paper also misses simpler baselines. For instance, how does RPO compare to standard DPO (or IPO, etc.) trained with label smoothing? Without these comparisons, it is impossible to conclude that the added complexity of the EM framework is justified. \n\n- The paper claims to address both simple annotator error and the \"inherent pluralism of human preferences\". However, it also states that these disagreements are \"functionally equivalent to noise\" and proceeds to learn a single \"latent collective preference\". This is a big oversimplification. The method does not truly deal with preference pluralism (e.g., by learning a distribution over rewards) but instead treats all non-majority opinions as incorrect data to be down-weighted. The related work on diverse preferences is not adequately addressed (e.g. [1]).\n\n- The central theoretical convergence guarantee (Theorem 4.2) relies on the assumption of a \"perfectly calibrated model\". The assumption that the model already matches the ground-truth preference distribution is at odds with the goal of training, which is to find such a model. This weakens the applicability of the theoretical claims to the practical training process, which generally starts from a poorly calibrated state. I understand that, in practice and in most cases, the starting model is already good enough to provide decent information, but this needs to be addressed and discussed somewhere in the paper. The EM algorithm's E-step depends on the model's current (and potentially poor) estimates of preference probabilities $p(y_w >^* y_l | x, \\theta^{(t)})$. If the initial model is badly aligned, its confidence estimates $w_i$ will be unreliable, potentially leading the M-step to down-weight correct labels and up-weight noise, hindering or preventing convergence. The paper does not investigate or discuss this. \n\n- The paper is missing a qualitative analysis of what its model identifies as noise. What kind of preference pairs receive a low confidence score $w_i$? Are they obvious annotator mistakes (e.g., typos), subjective disagreements (i.e., pluralism) or something else? \n\n- The writing could be improved for clarity. For instance, $K$ is used in Algorithm 1 but is never explicitly defined in the main text, only appearing in a footnote on page 7\n\n[1] A Minimaximalist Approach to Reinforcement Learning from Human Feedback, https://arxiv.org/abs/2401.04056", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MN7cOlxC3K", "reviewer": "Reviewer_tvbG", "summary": "This paper introduces Robust Preference Optimization (RPO), a novel method designed to address the challenge of label noise in Large Language Model (LLM) alignment. The authors correctly identify that existing alignment methods like DPO, IPO, and others operate under the strong and often flawed assumption that all human preference labels are correct. RPO reframes the problem by treating the true preference as a latent variable and employs an Expectation-Maximization (EM) framework to jointly infer the correctness of labels (E-step) and update the model policy along with annotator reliability parameters (M-step). A key strength is its design as a meta-framework, making it compatible with various existing preference loss functions. Comprehensive experiments on models like Mistral-7B and Llama-3-8B across benchmarks like AlpacaEval 2 and ArenaHard show consistent and significant performance gains when RPO is applied to several base algorithms (DPO, IPO, SimPO, CPO). The paper also provides theoretical guarantees, proving that under idealized conditions, RPO can recover the true reliability of annotators.", "questions": "1. How would RPO perform in a setting with a dataset containing multiple annotators with widely different and unknown reliabilities? Are there plans to validate the method on a dataset with genuine, traceable multi-annotator labels (e.g., from a crowdsourcing platform)?\n\n2. Theorem 4.2 assumes full-batch updates and a perfectly calibrated model. In the practical mini-batch setting with stochastic gradient descent, have you observed any issues with the convergence or stability of the EM procedure, particularly for the annotator reliability estimates?\n\n3. The experiments are conducted on 7B and 8B parameter models. Do you anticipate any challenges (e.g., memory, convergence) in scaling RPO to much larger models (e.g., 70B+ parameters), and have you conducted any preliminary experiments in this direction?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The core idea of applying an EM-based, soft-labeling approach to LLM preference alignment is highly innovative. Moving from hard, assumed-correct labels to a probabilistic model that explicitly accounts for noise is a significant conceptual and practical advancement.\n\n2. The formulation of RPO as a wrapper that can enhance a wide range of existing alignment methods (demonstrated with DPO, IPO, SimPO, CPO) is a major strength. This greatly increases the potential impact and applicability of the work.\n\n3. The paper is not merely empirical; it provides a solid theoretical foundation. The derivation of the EM algorithm for this context and, crucially, the proof of convergence to the true annotator reliability (Theorem 4.2) lend significant credibility to the method.\n\n4. The experimental section is thorough, validating the method's effectiveness across two different base models, two challenging benchmarks, and four distinct alignment algorithms. The consistent improvements are compelling evidence for RPO's utility. The authors thoughtfully address the computational challenges of a full-batch EM algorithm by proposing a practical mini-batch training scheme with an Exponential Moving Average (EMA) for updating annotator reliabilities, making the method feasible for large-scale training.", "weaknesses": "1.  A notable weakness is the experimental setup where all preference data is modeled as coming from a single, virtual annotator (K=1). While the justification in a footnote is reasonable, it undermines the validation of one of RPO's core capabilities: distinguishing between multiple annotators of varying reliability. The empirical results primarily demonstrate noise robustness in general, but not necessarily the multi-annotator aspect.\n\n2. The ablation study reveals that performance is sensitive to the initial annotator reliability (η₀) and the EMA momentum (α). This sensitivity could pose a non-trivial tuning burden in practice, especially in real-world scenarios where the optimal values are not known a priori.\n\n3. The experiments rely on existing datasets and synthetic noise injection. While valuable, this does not fully capture the complexity and structure of noise found in real-world, multi-annotator preference data. Validation on a dataset with genuine, traceable multi-annotator labels would be stronger.\n\n4. The EM procedure, with its additional forward passes to compute confidences (w_i) and updates for η, inevitably introduces computational overhead compared to the base algorithms. While likely acceptable given the performance gains, a brief discussion of the training time/ cost increase would be helpful for practitioners.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)