Aligner, Diagnose Thyself: A Meta-Learning Paradigm for Fusing Intrinsic Feedback in Preference Alignment
Authors: Mengyang Li
Pinlong Zhao
Zhong Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Direct Preference Optimization
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: The alignment of Large Language Models (LLMs) with human preferences is critically undermined by noisy labels in training datasets.
Existing robust methods often prove insufficient, as they rely on single, narrow heuristics such as perplexity or loss, failing to address the diverse nature of real-world noise.
We challenge this limited-scope approach by introducing a new paradigm where models learn to diagnose thyself, systematically fusing multiple streams of intrinsic feedback for a holistic reliability assessment of each preference pair.
We instantiate this paradigm through a meta-learning methodology that learns to adaptively reweight samples based on a rich diagnostic vector.
This vector captures three complementary perspectives: preference consistency, learning difficulty, and generation confidence.
Extensive experiments demonstrate that our approach significantly outperforms state-of-the-art methods across various noise conditions.
Crucially, our work provides the first quantitative analysis of these intrinsic diagnostics, revealing that their fusion is essential for overcoming the blind spots inherent in any single heuristic.
This diagnostic-driven paradigm offers a principled path towards developing more robust and trustworthy LLMs.

Published as a conference paper at ICLR 2026
ALIGNER, DIAGNOSE THYSELF: A META-LEARNING PARADIGM FOR FUSING INTRINSIC FEEDBACK IN PREFERENCE ALIGNMENT
Mengyang Li Pinlong Zhao² Zhong Zhang
"Tanjin Key Laboratory of Wireless Mobile Communications and Power Transmission, Tianjin Normal University, Tianjin, China, 300387
School of Cyberspace, Hangzhou Dianzi University, Hangzhou, China, 310018 1 Limengyanget jnu. edu. cn, pinlongzhao@hdu. edu. zhong. zhang8 84 3486 www.mmmll...oommm
ABSTRACT
The alignment of Large Language Models (LLMs) with human preferences is crit- ically undermined by noisy labels in training datasets. Existing robust methods often prove insufficient, as they rely on single, narrow heuristics such as perplex- ity or loss, failing to address the diverse nature real-world noise. We challenge this limited-scope approach by introducing a new paradigm where models learn to diagnose thyself, systematically fusing multiple streams of intrinsic feedback for holistic reliability assessment of each preference pair. We instantiate this paradigm through meta-learning methodology that learns to adaptively reweight samples based rich diagnostic vector. This vector captures three comple- mentary perspectives: preference consistency, learning difficulty, and generation confidence. Extensive experiments demonstrate that our approach significantly outperforms state-of-the-art methods across various noise conditions. Crucially, our work provides the first quantitative analysis of these intrinsic diagnostics, re- vealing that their fusion is essential for overcoming the blind spots inherent in any single heuristic. This diagnostic-driven paradigm offers a principled path towards developing more robust and trustworthy L.MSs.
INTRODUCTION
Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide range of tasks (Brown al. 2020 Touvron et al. )023). A crucial step in realizing their full potential lies in aligning these models with human preferences, ensuring they are helpful, harmless, and honest (Cao al., 202T Bai al. 2022). This alignment process often relies on preference datasets, where humans or Al systems indicate preferred responses among candidate pairs (Christiano et al., 2017; Stiennon et al. 2020 Rafailov et al., 2023). However, these datasets are often plagued by noisy preferences (NPs), where the recorded preference label is incorrect due to annotator disagreement, subjective biases, or errors in Al-based labeling (Gao et al. 2024 Zheng et al. 2023 Baumgärtner al. 2024). NPs can severely degrade alignment quality, leading poor model performance and the reinforcement undesirable behaviors (Gao et al., 2024 Rafailov et al. 2023).
To mitigate the impact of NPs, existing robust alignment methods can be broadly categorized into two groups. The first group employs coarse- -rrineed adjustments, such as modifying the loss func- tion with global noise estimates (Rafailov et al.) 2023 [Coowdhury al.] 2024). While offering some robustness, these methods lack the precision to handle instance-speciffe noise, treating all samples equally regardless of their individual characteristics. The second group leverages single- heuristic criteria to identify and correct or down-weight potentially noisy samples. A prominent example Kong et al. (2024), who utilize the perplexity difference (PPLDiff) between preferred and dispreferred responses as a signal for label inconsistency. While these methods offer instance- level granularity, they rely on a single, often myopic heuristic, neglecting the multifaceted nature
"Corresponding author.

---

Published as a conference paper at ICLR 2026
of preference reliability. For instance, PPLDiff may be misleading when dealing with fluent but factually incorrect responses, or when the model is inherently uncertain about the best response due to subjective or nuanced queries.
To address the limitations of existing approaches, we introduce a new paradigm for robust preference single, potentially flawed signal,
alignment: Aligner, Diagnose Thyself. Instead of relying our paradigm empowers the alignment model to act as its own diagnostician, systematically fusing multiple streams of intrinsic feedback to assess the reliability of each preference pair. We argue that preference reliability is not a monolithic property, but rather a multifaceted construct that can be best understood by considering complementary perspectives derived from the model's internal state. Specifically, we identify three key perspectives that, when combined, provide a more holistic and reliable assessment of preference quality:
Preference Consistency: Does the model's likelihood estimation align with the provided preference label? This captured by the dynamic perplexity difference (PPLDiff) between the preferred and dispreferred responses, reflecting the model's intrinsic assessment of their fluency and plausibility (Kong et al. 2024).
Learning Difficulty: How easily does the model assimilate the preference information? This is quantified by the training loss incurred by the preference pair, reflecting the degree to which the sample aligns with the model's current understanding of the task (Ren et al.) 2018 Shu et al. 2019). High loss values may indicate noisy labels or challenging edge cases.
Generation Confidence: How certain is the model in its generation process? This is estimated by the uncertainty associated with the model's token predictions, reflecting the model's internal confidence in its chosen responses. High uncertainty may suggest that the model is struggling to distinguish between plausible alternatives, potentially indicating subjective or ambiguous preferences.
These three perspectives, while individually informative, are inherently limited. PPLDiff can be misled by fluent misinformation, loss can be high for both noisy and genuinely difficult examples, and uncertainty ment method must be able to intelligently fuse these diverse and sometimes conflicting signals to arrive more informed assessment of preference reliability.
arise from both ambiguity and lack of knowledge. Therefore, robust align-
We operationalize this paradigm with a novel meta-learning methodology that learns to adaptively reweight training samples based rich diagnostic vector. This vector captures the three aforemen- tioned intrinsic feedback signals, allowing the model to dynamically adjust its learning process based on its own self-assessment. By training meta-learner interpret this diagnostic vector and assign appropriate weights, we enable the model to prioritize reliable preferences while down-weighting potentially noisy ones.
The contributions of this work are as follows:
We introduce a new paradigm for robust preference alignment based on fusing multiple intrinsic model diagnostics, empowering models to diagnose thyself instead of relying single, potentially flawed heuristics.
We instantiate this paradigm with a novel meta-learning methodology that learns to weigh samples based on diagnostic vector capturing preference consistency, learning difficulty, and generation confidence.
We provide the first systematic analysis of the interplay and relative importance of these intrinsic diagnostics, revealing that their fusion is essential for overcoming the limitations inherent any single heuristic.
We conduct comprehensive experiments demonstrate method's superiority sfat--of-the-an baselines under various noise conditions, including strong baselines uti- lizing perplexity differences.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
Our work is situated at the intersection of three research areas: robust preference alignment for LLMs, learning with noisy labels, and meta-learning for robustness.
LLM Alignment with Noisy Preferences. Aligning LLMs with human values via preference data is a cornerstone of modern AI safety (Ouyang et al. 2022 Bai et al. (2022), with methods like Rein- forcement Learning from Human Feedback (RLHF) (Christiano al., 2017 Stiennon et 2020) and Direct Preference Optimization (DPO) (Rafailov et al. 2023) being widely adopted. However, the susceptibility these methods to noisy preferences a well-documented challenge (Gao al.) 2024 Zheng et al. 2023). Initial efforts to address this focused on robust loss formulations. For instance, cDPO (Rafailov et al., 2023) and rDPO (Chowdhury et al.) 2024) introduce confidence scaling or label smoothing based on global noise estimate, applying uniform correction across all samples. More recently, methods have shifted towards instance-level hcuristics. A notable ex- ample is PerpCorrect (Kong et al. 2024). which uses the perplexity difference (PPLDifff between responses as a direct signal to detect and flip potentially mislabeled preferences. While effective, these approaches remain confined to single diagnostic perspective.
Learning with Noisy Labels (NLL). The problem of learning from corrupted supervision is long-standing challenge in machine learning (Frénay & Verleysen 2013 Song et al. (2022). Sample reweighting is prominent paradigm within NLL. where the idea down- weight instances that are likely to be mislabeled (Liu & Tao 2015: Jiang et al.) 2018). Various strategies have been proposed determine these weights, often based on heuristics like the training loss of sample-
the intuition being that noisy samples tend to have higher loss values (Han et al. 2018 Shu et al.)
2019 Li al. 2025). Our work adapts this established sample reweighting principle the unique context of LLM preference alignment.
Meta-Learning for Robustness. Meta-learning, or "learning to learn", has proven to be a power- ful technique for designing adaptive training algorithms (Ren et al.) 2018). the context of robust learning. a particularly successful application has been to meta-learn a sample reweighting func- tion (Shu et al., 2019 Jamal et al., 2020). These methods typically train a small neural network (a "meta-net") to map features like training loss to sample weights, with the meta-net's parameters being optimized based on performance on small, clean meta-dataset. Our methodology builds directly upon this meta-learning foundation. Our key novelty lies in what feed into the meta- learner.
3 METHODOLOGY
In this section, we formally introduce our paradigm for robust preference alignment. We begin by defining the problem setup and then detail the construction of our multi-perspective diagnostic vector. Finally, present our meta-leamnnn methodology for learning to fuse these diagnostics to achieve robust alignment. The overall architecture of our approach is illustrated in Figure
3.1 PRELIMINARIES: DIRECT PREFERENCE OPTIMIZATION
We build upon the Direct Preference Optimization (DPO) framework (Ratailov et al. 2023). Let D = ((x(9)) (8) (6) ) be a preference dataset, where 2.(() is a prompt, ya is the preferred response, and y1"h is the dispreferred response. DPO aims to train an LLM policy ne to satisfy these preferences, starting from reference policy कार (typically an SFT model). The DPO loss for single preference pair is given by:
To (t)) (rer(yul
To ())
Loro (o. nett) log
Boog
3 log
(1)
Terr(y (z)
where is hyperparameter that controls the deviation from the reference policy, and (-)) is the lo- gistic function. When the training dataset D contains noisy preferences (i.e., (yee 31) are swapped), directly minimizing this loss can lead the model to learn incorrect behaviors.
3

---

Published as a conference paper at ICLR 2026
Meta-align Model
Language Model
Preference Consistency
Learning Difficully
-
Updata
Meta perference data
Generation Confidence
Meta loss
parameters
Perference
Virtual DPC
Updata virtual parameters 6'
Weighted DPO
Updata parameters
Figure I: overview of our diagnostic-driven meta-learning paradigm.
3.2 CONSTRUCTING THE INTRINSIC DIAGNOSTIC VECTOR
Our core premise is that a model's own internal state provides rich, multi-faceted feedback about the reliability of given preference pair. We capture this feedback in dynamic intrinsic diagnostic vector, z E Rd, computed for each sample at every training step. This vector comprises three key components, each offering a complementary perspective data quality.
Preference Consistency (sppi). A well-aligned model should assign higher likelihood (lower perplexity) a genuinely preferred response. deviation from this expectation strong indicator of potential label-model conflict. We quantify this using the log-perplexity difference (PPLDiff), computed dynamically with the current policy Te, at training step
log PPL( (xr v1 log PPL Ta, Ix (x:); (
ppp
(s) Σ log (Sk (<x)))))) A positive Eppl suggests that the labeled win- ning response less likely under the current model than the losing one, flagging it potential NP.
where PPL(T, s) exp( 1/1
Learning Difficulty (Etss). The magnitude the training loss for a sample reflects how inconsis- tent it is with the model's current parameterization. Noisy samples often present conflicting gradi- ents, resulting in higher loss values. We use the instance-wise DPO loss itself as signal of learning difficulty:
(DPO(TT, Feefi (x) (8) Mi (O))
Jlsss
This provides direct measure of how surprising given preference is to the model.
Generation Confidence (Zuncert). Beyond likelihood, the model's confidence during the genera- tion process offers another valuable signal. model that is uncertain about its token predictions for a given response may be grappling with ambiguity or subjectivity in the prompt, making the corre- sponding preference label less reliable. We measure this confidence using the average token-level entropy of the generated responses. Specifically, for a response y (y1...., Bm)) the uncertainty is:
H H (e.)
Σ Te, (v| v<;) log Te, v<,),
(4)
j=1 VEV
where the vocabulary. High entropy indicates low confidence. We use the uncertainty of the preferred response as our diagnostic signal, = = H (=(+) mo.), as noisy preferences often correspond to less coherent confident generations for the supposed winner.

---

Published as a conference paper at ICLR 2026
Algorithm Meta-Learning for Fusing Intrinsic Diagnostics
Input: Noisy data D, clean meta-data Dmeta. initial 0o. Wo. rates Oe,, ow, steps T. Output: Aligned model parameters 0r.
1: for tort- = 0 to T do
2: Sample mini-batches B, C and Bmeta C. Deeta-
3:
Compute diagnostic vectors {z, Jies using Oz.
Sec. 3.2
Compute weights {u, = (zz,), [e)]seB.
4:
5:
Compute virtual parameters 0y (W,) via 6q.
6:
Compute meta-loss Lmeta (W)) on Beeu using (W,) via Eq.
7:
Update meta-learner: Wt+1 <- W. OW Vww.. Lmen (W))
8: Update main model: Oc+1 +- 02 Org Vo, feeigheed Oz, W2+1).
b Using new weights
9:
end for
10:
return 0r.
The final diagnostic vector for sample i at step is the concatenation of these normalized compo- nents:
(i) [norm( (ppl), norm( ) norm( Puncent)
Zz
3.3 A META-LEARNING FORMULATION FOR FUSING DIAGNOSTICS
Given the diagnostic vector z, our goal is learn a function V(z; W) that maps these diagnostics to non-negative sample weight, where W are the parameters of the meta-learner. We employ meta-learning strategy (Ren et al.) 2018) where
quality of the weights produced by is eval- uated based on the main model's performance on small, clean meta-dataset, Dmeta- This bi-level optimization can be understood as learning an implicit, adaptive weighting scheme, with theoretical guarantees that minimizing the empirical meta-loss leads to good generalization on the true clean data distribution. We provide detailed theoretical analysis in Appendix A
The training proceeds in bi-level optimization loop. At each step sample mini-batch B, from the noisy training set and a mini-batch Bmeta from the clean meta-set D Dmeta-
Inner Loop: Virtual Update. First, we compute the diagnostic vector z! for each sample E B,
using the current policy Te, The meta-learner (-; Wx) then produces weights of = (z Wi). These weights modulate the DPO loss on the training batch:
Eweighedd (8t, We)
11 Loro(To, Tetl J).
(5)
[B
JES,
We then compute hypothetical one-step gradient update for the main model, resulting in virtual parameters (W,):
(,WW,) 0, ogg Vo, weigghted (02, W.).
where og is the learning rate for the main model.
Outer Loop: Meta-Objective and Updates. The quality of the weighting parameters W, as- sessed by evaluating the performance of the virtual model Tw((W.) on the clean meta- batch B This yields the meta-loss:
Leeeta (W,) Bietal
Σ LoPO(Te:(W.)) Feतt k). kEB
The meta-learner's parameters W are then updated by descending the gradient of this meta-loss: W = W, aw V mmeaa (W,). Finally, the main model's parameters 6, are updated using the original training batch B,, but with weights computed from the updated meta-learner V:; W++1):
0,+1 = 02 0g Ve, Lweightea (Or, W++1).
This process, summarized in Algorithm I allows the meta-learner to learn effective, data-driven strategy for fusing the intrinsic diagnostics, guided by the objective of improving performance on clean, reliable data.
5

---

Published as a conference paper at ICLR 2026
4 EXPERIMENTS
We conduct a comprehensive set of experiments to validate our proposed paradigm. Our evaluation is designed to answer three key questions: (1) Does our diagnostic fusion approach outperform state- of-the-art robust alignment baselines across various noise conditions? (2) What is the individual contribution of each intrinsic diagnostic, and is their fusion truly necessary? (3) How do the different diagnostics interact, and what is their relative importance in identifying noisy preferences?
4.1 EXPERIMENTAL SETUP
Datasets and Noise Simulation. Our experiments are conducted on two widely-used public pref- erence datasets: Golden HH (Bai et al. 2022 Ethayarajh et al. 2024), a helpfulness-focusee subset of Anthropic-HH, and OASSTI (Köpf et al. 2023). a multi-turn conversational dataset. To evalu- ate robustness under controlled conditions, we simulate noisy preferences by randomly swapping 'chosen' and 'rejected' labels for a fraction E {0.1, (..1,0.2, 0.3, 0.4] of the training samples, following standard protocols (Kong et al.] [024]] Chowdhury et al. 2024). To validate scalabil- ity and real-world applicability beyond synthetic noise, we additionally conduct experiments on two large-scale datasets: HuggingFace H4 StackExchange Preferences (10.8M samples) containing community-voted technical Q&A with inherent annotation subjectivity, and GPT4AIl (0.8M sam- ples) representing LLM--istilled preferences with generation artifacts. These datasets exhibit natu- rally occurring noise from human disagreement and distillation errors, providing a complementary testbed synthetic label-flipping. For our method, a small, clean meta-dataset is held
from the original training set: M = 100 samples for Golden HH and OASST1, and M = 200 for the large- scale datasets. These sizes were chosen based on sensitivity analysis (see Appendix C which shows that performance saturates in this range, making them practical and effective choices. Regarding the practical acquisition of such data, we explore scalable strategies including aagh-agreement filtering and expert curation in Appendix D.5 Further details on data splits are provided in Appendix B.1
Models and Implementation. We evaluate on suite of open-source LLMs demonstrate broad applicability, including Llama-2-7B (Touvron et al.) 2023). Phi-2 (Javaheripi et al.] (023), and Llama-3-8B for large-scale and real-world evaluations (Tables (+2). Our method and all DPO- based basclines are implemented using the TRL library (von Werra et al., 2020) for consisteney. The meta-learner (-; W) in our approach is a two-layer MLP. All All hypeparametess and implemen- tation details are detailed in Appendix B.2 to ensure full reproducibility.
Baselines. We compare our approach against a strong and diverse set baselines. These include Vanilla DPO (Rafailov et al. (023); robust DPO variants such cDPO (Rafailov al. (023), rDPO (Chowdhury et al.) 2024), and the recent state-of-the-aar DR-DPO (Azar et al. 2024); and prominent heuristic-based methods. For the latter, implement PerpCorrect (Kong et al.) 2024) in two settings: a static version with pre-computed PPLDiff, and stronger dynamic version where PPLDiff is re-computed each step for fairer comparison with our approach.
Evaluation Metrics. Following standard practice (Rafailov et al., 2023 Chowdhury et al. 2024).
our primary automated metric Reward Model Accuracy, where an independently trained reward model assesses alignment on a clean test set. To capture nuances beyond automated scores, we com- plement this with human-proxy proxy evaluation using GPT-4 Win Rate, For this, we compare generations from our final model against the strongest baselines in pairwise fashion, with GPT-4 acting as an impartial judge.
4.2
MAIN RESULTS: STATE-OF-THE-AR ROBUSTNESS
Figure presents primary results, plotting Reward Model Accuracy against the injected noise rate (E) on the Golden HH and OASSTI test sets. Across all datasets, model architectures, and non-zero noise conditions, our full diagnostic fusion method, denoted Ours (Fusion), consistently establishes a new ofate-of-the-an in robust preference alignment.
As expected, the performance of Vanilla DPO degrades sharply as the noise level increases. demon- strating its sensitivity to label corruption. While existing robust methods, including cDPO, rDPO, and DR-DPO, offer substantial improvements, approach consistently maintains a significant

---

Published as a conference paper at ICLR 2026
Golden HH with Llama-2-7B
OASSTI with Llama-2-7B
100
75-
70-
90-
Reward Accuracy (%)
Reward Accuracy (%)
65-
80
60-
70
.5
Vanila DPO
Pepp Lanost (Pysania) (Uhorriirry only
50
Vasilaa DPG
PapCoeree (Dysazscc) Dona tacotaine
(D00
d37o
60
only) (P9LDif wely)
45
PapCOreent (aaid)
(Faiiz))
PepCCunect (Stati))
Ourr avisa)
30
0%
10%
20%
30%
40%
40
0%
10%
2006
30%
41%
Triining Data Naise Ratio (d)
Training Duta Noise Ratio (c)
Golden HH with Phi-2
OASSTI with Phi-2
100
75-
70-
Roward Accuracy (%)
Roward Accuracy (%6)
65-
60
55
70-
50
Vanila DPD
Paptlreet (Dysanel
Vanila DPU
- Poptlomet (Uhreazee)
LEPO
Oua (Unceraizty ondyt only) (P9LDiff vely
45
<DPO
Oura (Uacemante wayy
(lase
Pupeer (Sari)
(Pwino))
Popclonee (Sude))
35
0%
10%
20%
30%
40%
0%
10%
20%
30%
40%
Trininn Data Noise Ratio
Tmining Data Noise Ratio
Figure 2: Reward Accuracy (%) versus Training Noise Ratio (c) Golden HH (left) and OASSTI (right) datasets.
performance margin them. Notably, our method also outperforms the strong heuristic-baid based baselines. Even when compared against PerpCorrect (Dynamic) which also leverages a dynamic, instance-level signal-our method's ability fuse multiple complementary diagnostics provides clear and decisive advantage. This performance gap widens in high-noise regimes (e 2 0.3), where relying on a single heuristic becomes increasingly insufficient.
To assess the practical impact on generation quality, Figure 3 presents the results of our pairwise comparison judged by GPT-4. When pitted against the strongest baselines on the Golden HH dataset with 30% noise, our method achieves a decisive win rate. For instance, against DR-DPO, Ours (Fusion) is preferred in 62.5% cases, underscoring that the im- provements measured by reward model accu- racy translate into tangible gains in conversa- tional quality and helpfulness. This suggests that our method does not merely overfit to the reward model but learns a more genuinely robust and helpful policy.
Win
Tic
PerpCorrect (Dynemie)
DR-DPO
20
40
60
80
100
Percentage (%)
Figure 3: Win rates of Ours (Fusion) vs. baselines on Golden HH (c 0.3).
4.3 ABLATION STUDY: THE NECESSITY OF FUSING MULTIPLE DIAGNOSTICS
Having established the overall superiority of our fusion-based approach, we now conduct a targeted ablation study to disentangle the contributions of its core components. The central question ad- dress is: is the fusion of multiple diagnostics truly necessary, or is the performance gain primarily driven by single, dominant diagnostic like PPLDiff? To investigate this, we evaluate several vari- ants of our method on the Golden HH dataset under a challenging 30% noise condition (c = 0.3). These variants use our meta-learning framework but are restricted only a single diagnostic input: Ours (PPLDiff only), which uses preference consistency; Ours (Loss only), which uses learning difficulty; and Ours (Uncertainty only), which uses generation confidence.
7

---

Published as a conference paper at ICLR 2026
The results, presented in Figure 4 unequivo- cally demonstrate the necessity of diagnostic fusion. While the Ours (PPLDiff only) vari- ant emerges as the strongest single-diagnostic model-confirmng PPL.ifff's crucial role as a primary noise indicator-out our full Ours (Fu- sion) model surpasses by significant mar- gin. This performance gap highlights key finding: although less powerful isolation, the learning difficulty (loss) and generation con- fidence (uncertainty) diagnostics provide es- sential, complementary information. They act as crucial correctives, addressing the inherent blind spots of PPLDiff-only approach. Fur- thermore, the relatively modest performance of the Ours (Loss only) and Ours (Uncertainty only) variants reveals the potential pitfalls of relying on these more ambiguous signals alone. For instance, training loss can be high for both noisy samples and genuinely difficult (but clean) ones. Without the anchoring context provided by strong signal like PPLDiff, model relying solely on loss may incorrectly down-weight valuable, hard examples. Our fusion mechanism, guided by the meta-learning objective, learns to navigate these ambiguities, leveraging the strengths of each diagnostic while mitigating their individual weaknesses. This synergy is the primary driver of our method's state-of-tt tiiee-ofthhearr art robustness.
Fusion
14 97.1
DPL only
H 95.8
Mothod
Loss unly
88.4
Uncertainty enly
R4.7
85
90
95
100
Reward Accuracy (%)
Figure 4: Ablation study on the Golden HH test set with 30% training noise.
We also verified that our method robust meta-learnr architectural choices; comprehensive ab- lations in Appendix D.2 show that performance remains stable across variations in depth, width, and design paradigm. Additionally, we explored augmenting the diagnostic vector with other candidates (e.g., gradient norms, response length), but found them to be largely redundant or uninformative compared to our core trio; a detailed analysis of alternative diagnostics is presented in Appendix
4.4 ANALYSIS OF INTRINSIC DIAGNOSTICS
To gain deeper insight into how our model learns to fuse the different intrinsic diagnostics, we conduct a final set of analyses on the learned weightiigg function. Our goal understand the relative importance of each diagnostic, their interplay, and how their roles may adapt under different noise conditions.
Quantifying Diagnostic Importance with SHAP. We first seek to understand the overall influ- ence of each diagnostic. We employ SHAP (SHapley Additive exPlanations) (Lundberg & Leep 2017, game-theoretic approach to explain the output of the trained meta-learner V(: W). Fig- ure a) plots the mean absolute SHAP value for each diagnostic, representing its average impact on the weight assignment across the test set under 30% noise. The analysis reveals a clear hier- archy: Preference Consistency (PPLDiff) is the most influential diagnostic, confirming its role as the primary signal for label-model conflict. Crucially, Learning Difficulty (Loss) and Generation Confidence (Uncertainty) also exert substantial influence, validating our hypothesis that multi- perspective assessment is essential. This quantitative ranking is consistent with our ablation study, where the PPLDiff-only model performed best among single-diagnostic variants but was signifi- cantly surpassed by their fusion.
Uncovering Interplay and Non-linear Relationships. Beyond average importance, we investi- gate how these diagnostics interact. The SHAP summary plot in Figure provides more granu- lar view, showing not just the magnitude but also the direction of each diagnostic's impact. Several key patterns emerge:
Dominant Role of PPLDiff: As expected, high (positive) PPLDiff values (red dots on the right) consistently push the assigned weight lower (negative SHAP values), acting as a strong penalty for inconsistency. Conversely, low (negative) PPLDiff values robustly support higher weight.
8

---

Published as a conference paper at ICLR 2026
Big
1.0
0.90
Mean (SLAAP Value
PELDUR
0.8
Fasture Milac
0.6
Tmining Loss
0.4
0.24
Usscrraiutty
0.2
0.0
PPLDxE
Training Loss (a) Fenture Importance
Uncertaizty
Value (b) SILA Value Distribution
Figure 5: In-depth analysis of the learned meta-weighting function on Golden HH. (a) PPL.Dff emerges as the most influential diagnostic overall. (b) The beeswarm plot reveals the distinct roles and non-linear interactions of the diagnostics.
Loss as values have minimal impact the weight, but high loss values strongly correlate with significant reduction weight. This suggests the meta-leamae has learned to use high loss powerful flag for problematic samples, be they noisy hard examples.
Flag: The training loss exhibits a clear one-sided effect. Low loss
Uncertainty as a Nuanced Modulator:
The effect of uncertainty is more nuanced. High uncertainty generally corresponds to lower weights, but its impact is most pronounced when interacting with other diagnostics. For instance, sample with a moderately negative PPLDiff (suggesting it is clean) might still be down-weighted if its generation uncertainty is very high. This indicates the meta-learner uses uncertainty temper confidence in samples that, while superficially plausible, are generated with low conviction by the model.
This analysis reveals that the meta-learner does not simply learn linear combination of signals. Instead, it discovers complex. non- non-lineat relationships, using each diagnostic to cover the blind spots of the others. detailed qualitative example illustrating this synergy provided in Appendix D.
Adaptive Roles under Varying Noise Levels. Finally, we examine whether the meta-learner's 's strategy adapts as the train- ing environment changes. We analyze the SHAP values of meta-learners trained un- der different noise ratios (e E {0.1, 0.4)).
100%
Diagnostic PPLDaH Training Uncertainty
Relative Importance (%)
80%
78.3%
60%
As shown in Figure 66 the relative impor- tance of the diagnostics shifts. In low- noise regimes
50.0%
40%
0.1), the meta-learner relies heavily on PPLDiff, as is a highly reliable signal when the data is mostly clean. However, in high-noise regimes
28.6%
21.4%
20%
13.0%
0.4), the relative importance of Training Loss and Uncertainty increases. This an important finding: as the pri- mary signal (PPLDiff) itself becomes less reliable due to the model being trained on increasingly corrupted data, the meta- leamner adaptively increases its reliance on secondary, corroborating signals. This demonstrates that our paradigm does not learn static fusion rule. but rather a dynamic, adaptive policy that intelligently adjusts its diagnostic strategy based on the perceived difficulty of the learning environment.
Low Noise 0.1)
High Noise 0.4)
Training Data Noise Ratio
Figure 6: Relative importance of diagnostics (normal- ized mean SHAP values) learned under low (c 0.1)
and high (c 0.4) noise.

---

Published as a conference paper at ICLR 2026
4.5 SCALABILITY AND REAL- WORLD NOISE VALIDATION
While the controlled experiments in Sections 4.2e4..ee fusion paradigm under synthetic noise, real-world preference datasets often exhibit substantially larger scales and more diverse noise characteristics. To validate that our approach maintains its advantages in practical settings, we conducted additional experiments on two large-scale datasets with naturally occurring noise.
demonstrate the effectiveness of our diagnostic
Large-Scale Datasets. We evaluated on HuggingFacc H4 StackExchange Preferences (Lambert et al. 2023). dataset of 10.8M community-voter technical Q&A pairs, and GPT4AIl (Anand et al. 2023), comprising 0.8M LLM-distilled preference pairs. StackExchange contains inherent subjec- tivity from community voting, while GPT4All exhibits generation artifacts such truncation and repetition. For StackExchange, we conducted progressive scaling at 100K, 1M, and 10.8M samples to assess computational scalability. Meta-dataset sizes were set to M 200 for both datasets.
Performance and Scalability. T shows that our method consistently outperforms base- lines, with advantages ranging from 0.8% (GPT4AII) 1.9% (StackExchangee. The per- formance gap widens as dataset size increases (+0.6% at 100K - +1.9% at 10.8M), suggest- ing that diagnostic fusion becomes increasingly valuable with more diverse training data. Train- ing on 10.8M samples required 240 GPU-hours on 8xA40 with 48 GB peak memory, represent- ing 41% % time and 19% memory overhead versus vanilla DPO both metrics scale linearly with dataset size. Detailed analysis of noise patterns and representative case studies are provided in Ap- pendix
Table I: Performance on large-scale datasets.
StackEx.
GPT4AIl
Method
100K
1M
All
100K
All
Vanilla DPO
71.2
73.2
74.5
68.5
69.3
DR-DPO
74.8
76.4
78.2
72.6
73.5
Perp. (Dyn.)
75.3
77.0
78.7
73.1
74.0
Ours
75.9
77.9
80.6
73.8
74.8
Real-World Noise Evaluation. Beyond large-scale validation, we evaluated on datasets with organic human annotation noise: We-
Table 2: Performance on naturally noisy datasets.
bGPT NNkaan et al., 202T) Comparisons contains real human preference annotations with documented inter-annotator disagreement
Method
WebGPT
Chatbot Arena
Vanilla DPO
68.3
84.2
DR-DPO
71.5
86.5
(Cohen's k 0.56), and Chatbot Arena (Chiang ct al. 2024) represents completely organic user preferences from real interactions. As shown in Table 2 our method demonstrates consistent improvements on both naturally noisy datasets, validating effectiveness beyond synthetic noise simulation.
Perp. (Dyn.)
73.2
87.8
Ours
76,9
89.6
These experiments demonstrate that our method scales to 10M+ samples with linear overhead while maintaining robust performance across diverse real-world noise patterns. The consistent improve- ments validate that diagnostic fusion is essential for handling dataset-specific failure modes, with naturally occurring noise exhibiting fundamentally different signatures than synthetic abbel-fliping. Beyond natural noise, our fusion paradigm also maintains robustness against sophisticated adversar- ial attacks; detailed evaluations are provided in Appendix D.4
5 CONCLUSION
We have presented a new paradigm for robust LLM preference aligmment that empowers models to perform self-diagnosis by fusing multiple intrinsic feedback streams. Our meta-learning implemen- tation of this paradigm sets a new state-of-the-aar in handling noisy preference data. By providing the first systematic analysis of how different internal signals be synergistically combined, this work lays a foundation for a new class of diagnostic-driven, adaptive alignment algorithms. We
believe that building models capable of such sophisticated self-assessment is a fundamental step towards creating more reliable, robust, and trustworthy Al systems.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
In accordance with ICLR policy, we disclose that large language models (LLMs) were employed as writing assistants during the preparation of this paper. Their primary function was to support grammar correction and language refinement, with the goal of improving the overall readability of the manuscript. All core ideas and analyses were conceived and developed solely by the human authors, who assume full responsibility for the final content of the paper.
ACKNOWLEDGMENTS
This work was supported in part by the National Natural Science Foundation of China under Grant No. 62171321.
REFERENCES
Yuvanesh Anand, Zach Nussbaum, Adam Treat, Aaron Miller, Richard Guo, Benjamin Schmidt, Brandon Duderstadt, and Andriy Mulyar. Gpt4all: An ecosystem of open source compressed language models. In NLP-OSS 2023, pp. 59-64, 2023.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm understand learning from human preferences. In AISTATS, pp. 4447-4455, 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint wrXiv:2204.05862. 2022.
Tim Baumgärtner, Yang Gao, Dana Alon, and Donald Metzler. Best-of-venom: Attacking rlhf by injecting poisoned preference data. In CoLM, pp. 1-10, 2024.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS, volume 33, pp. 1877-1901, 2020.
Zehong Cao, KaiChiu Wong, and Chin-Teng Lin. Weak human preference supervision for deep reinforcement learning. IEEE TNNLS, 32(12):5399-5378 2021.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, al. Chatbot arena: An open platform for evaluating Iims by human preference. In ICML, 2024.
Sayak Ray Chowdhury, Anush Kini, and Nagarajan Natarajan. Provably robust DPO: Aligning language models with noisy feedback. In ICMI., pp. 42258-42274, 2024.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In NeurlPS, volume 30, pp. 1-10, 2017.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01336, 2024.
Benoît Frénay and Michel Verleysen. Classification in the presence of label noise: survey, IEEE TNNLS, 2555)8445-869 2013.
Yang Gao, Dana Alon, and Donald Metzler. Impact of preference noise on the alignment perfor- mance of generative language models. CoLM, pp. 1-10, 2024.
Bo Han, Quanming Yao. Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NewrIPS, volume 31, pp. 1-10, 2018.
11

---

Published as a conference paper at ICLR 2026
Muhammad Abdullah Jamal, Matthew Brown, Ming-Hsual Yang, Liqiang Wang, and Boqing Gong. Rethinking glass-balancee methods for long-tailed visual recognition from a domain adaptation perspective. In CVPR. pp. 7610-7619, 2020.
Mojan Javaheripi, Sébastien Bubeck, Marah Abdin, Jyoti Aneja, Sebastien Bubeck, Caio César Teodoro Mendes, Weizhu Chen, Allie Del Giorno, Ronen Eldan, Sivakanth Gopi, et al. Phi-2: The surprising power of small language models. MSR Blog, 1(3):3, 2023.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data- driven curriculum for very deep neural networks on corrupted labels. In ICML, pp. 2304-2313, 2018.
Keyi Kong, Xilie Xu, Di Wang, Jingfeng Zhang, and Mohan S Kankanhalli. Perplexity-aware correction for robust alignment with noisy preferences. In NeurlPS, volume 37, pp. 28296-28321, 2024.
Andreas Köpf, Yannic Kilcher, Dimitri Von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Richard Nagyfi, et al. Openassis- tant oinprrrsttiinsdddmmmrraaiiingn large language model alignment. NeurIPS, volume 36, pp. 47669-47681, 2023.
Nathan Lambert, Nazneen Rajani Lewis Tunstall, and Tristan Thrush. Huggingface h4 stack ex- change preference dataset, 2023. URI. exchange-prefferences, 2023.
Mengyang Li, Xiaoling Zhou, and Ou Wu. Delving into the training dynamics for image classifica- tion. JEEE TIP, 34:6783-6798, 2025.
Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE TPAMI, 38(3):447-46,, 2015.
Scott M Lundberg and Su-In Lee, unified approach to interpreting model predictions. NeurIPS, volume 30, pp. 4768-4777 4777, 2017.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332. 2021.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In NeurIPS, volume 35, pp. 27730-27744, 2022.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In NeurlPS, volume 36, pp. 53728-53741, 2023.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In ICML, pp. 4334-4343, 2018.
Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Meta-weight- net: Learning an explicit mapping for sample weighting. In NeurIPS, volume 32, pp. 1-10, 2019.
Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. IEEE TNNLS, 34(1 1 -1135-8153, 2022.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize with human feedback. NewrIPS, volume 33, pp. 3008-3021 2020.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshor, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel
12

---

Published as a conference paper at ICLR 2026
Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuming Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. URL hww.httss//// wtwpvv/aaxivv.ooo www.xiiv.or.... wwt.or/g///oo abs 2000 09288
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforcement learning, 2020.
com/ coom//ugggingface/trtl
Sen Zhao, Mahdi Milani Fard, Harikrishna Narasimhan, and Maya Gupta. Metric-optimized exam- ple weights. In ICML, pp. 7533-7542,
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. In NeurlPS, volume 36, pp. 46595-46660 46623, 2023.
13

---

Published as a conference paper at ICLR 2026
A THEORETICAL ANALYSIS
This section provides a theoretical lens through which to understand our paradigm's 's mechanism. We first interpret the bi-level optimization as learning implicit weighting scheme and then present high-level generalization bound for the learned weighting policy.
A.1 IMPLICIT WEIGHTING SCHEME
The meta-learning process be viewed as learning implicit, adaptive scheme for re-weighting noisy training preferences. The update to the meta-learner's parameters, W, is driven by its ability produce weights that guide the main LLM towards better performance on a clean meta-dataset.
The update rule for W at step / is given by gradient descent on the meta-loss:
W4+1 We aw V Lmmaa (W.
(9)
Using the chain rule, the meta veta-gradient V W (W;) can be expanded as:
d0, (W,)
V metta E.s
V Vor_Drr(Te((w.))
(10)
dW,
The term de((w) atV, represents how the virtual parameters change with respect to the meta-parameters. Substituting the definition of 0g from Eq. we get:
(W,, ag V w V Lweigneddd ( Wi)lo=o.-
(11)
dW,
The Hessian- product in this term connects the meta-leamners parameters W to the main model's update. Specifically, the gradient Vw operates on Lweighted through the generated weights = Viz,; Wt).
This structure implies that the meta-leamnr parameters W are updated in direction that rewards the generation of weights v which, when used to train the virtual LLM on the noisy batch B,, lead improved performance (lower (PO) on the clean meta-batch Bmeta- In essence, training instances (via their diagnostic vectors 2) that are assigned beneficial weights by (-; - judged by their downstream utility for clean alignment- exert stronger and more favorable influence on the meta-learner's 's update.
A.2 GENERALIZATION BOUND
We provide a high-level generalization bound for our method, drawing inspiration from standard analyses in meta-learning and learning with noisy labels Zhao al. (2019). Let Relean (W) be the true expected risk (e.g., expected LDOO on the true clean preference distribution Patenn) of the main LLM policy that is trained using the weights generated by the meta-learner V(;; W)) Let
Rmera (W) mmeu(()) be the empirical risk on the clean meta-dataset Dieu of size M. We aim to bound the generalization gap (W*) (W" where W is the set of parameters learned by method.
Assumptions. We make the following standard assumptions: 1) The meta-leamer's parameter space W is bounded. 2) The DPO loss is bounded, LDPO [0, Bl 3) The meta-dataset D consists of M i.i.d. samples from Palean-
Theorem (Generalization Bound - Informal). Let W = arg minwew Rieet ( (W) be the param- eters learned by minimizing the meta-loss. Then, for any 8 > with probability at least 8 over the random draw of D mesaa-
Comp(Fw) ) - log(1/6)
Retean (W*) Rmeca (W*)
(12)
M
where Comp(Fw) is measure of the complexity of the function class induced by the meta-learner, for instance, its Rademacher complexity. For a parametric model like a neural network for V(: W), this complexity term is related its size and depth.
14

---

Published as a conference paper at ICLR 2026
Implication. This bound indicates that the performance of the trained meta-learner unseen clean data is controlled by its empirical performance on the meta meta-dataset and the complexity of the meta-learner itself. As the size of the clean meta-dataset M increases, the generalization gap shrinks, ensuring that minimizing the meta-loss on Meeta leads to a meta-learner that is effective on the true clean data distribution. This provides theoretical justification for our data-driven approach to learning a robust diagnostic fusion policy.
B IMPLEMENTATION DETAILS
B. DATASET DETAILS AND PREPROCESSING
Golden HH and OASSTI. The public preference datasets, Golden HH (Bai et al. 2022, Etha-
yarajh et al. 2024) and OASSTI (Köpf et al.] 2023), underwent minimal preprocessing beyond standard tokenization provided by the TRL library. We used the versions and splits processed by Rafailov et al. ).023). For each dataset, we constructed the data splits as follows:
Test Set (Diea): We used the original, official test split, which was assumed to be clean and was used exclusively for final evaluation.
Meta-Dataset (Deeta): We randomly sampled M = 100 preference pairs from the original training split to serve the clean meta-dataset.
Validation Set (Dva)): We randomly sampled 300 preference pairs from the remaining training split for hyperparameter tuning.
Noisy Training Set (D): The rest of the original training split was used as the main training set. Noise injected into this set by randomly swapping 'chosen' and 'rejected" labels at rates E {0.1, 0.2, 0.3, 0.4].
There was no overlap between these four data splits.
B.2 HYPERPARAMETERS AND TRAINING CONFIGURATION
All experiments conducted on NVIDIA A40 GPUs. The main LLM parameters fine-tuned using the Adam W optimizer with a weight decay of 0.01. The meta-learner was also optimized with AdamW. Key hyperparameters are listed in Table
Table 3: Key hyperparameters for our method and DPO-based basclines.
Parameter
Llama-2/
Phi-2
Main Model Learning Rate (og)
x 10-6
x 10-5
Meta-Learner Learning Rate (ow)
10-4
10-1
Batch Size (B,)
8
16
Meta-Batch Size (Bmeta)
DPO
0.1
0.1
Training Epochs
Meta-Lcarner Architecture
MLP with layers (hidden dim = 100)
Baseline Configurations. All baselines were trained with the same main model learning rate, batch size, and training duration as our method for a fair comparison. For PerpCorrect, the PPLDiff threshold was tuned on D For DR-DPO, we used the hyperparameters recommended in the original paper.
Computational Cost. Training our full fusion method for one epoch on the Golden HH dataset with Llama-3-8B required approximately 8 hours single A40 GPU. In comparison, standard DPO took approximately 6 hours. The overhead is primarily due to the dynamic computation of diagnostics and the bi-level optimization loop.
15

---

Published as a conference paper at ICLR 2026
B.3 REWARD MODEL FOR EVALUATION
The independent reward model (RM), used for calculating Reward Accuracy, was trained on the entirety of the clean original training split for each dataset. The RM architecture was initialized from the same base SFT checkpoint as the policy models (e.g., -lama---8B-Insruuct) and included a final linear layer to output a scalar reward. was trained for one epoch using standard pairwise preference ranking loss, learning rate of 10-5, and a batch size of 4. This RM remained fixed during the evaluation of all aligned policy models.
C SENSITIVITY TO META-DATASET CHARACTERISTICS
To assess the robustness and practicality of our method, we investigated its sensitivity to the two primary characteristics of the meta-dataset mennn size (M) and its purity (i.e., potential contam- ination with noise). These analyses were conducted on the Golden HH dataset with main training noise of = 0.3 (for the noise sensitivity test) or 6 0.4 (for the size sensitivity test), using the Llama-2-7B model.
Impact of Meta-Dataset Size. Figure 7 il- lustrates the performance of our method as the size of the clean meta-dataset, M, varied from 10 to 300 samples. A clear trend of im- proved performance is observed with increas- ing M, though with diminishing returns. The results show that strong performance is achiev-
Find Reward Accuncy (1)
96
94
able even with modest meta-dataset size of M 100, where our method already sig- nificantly outperforms baselines that lack such meta-gun guide-uuddde.. Performance begins to saturate around M x 100 200. suggesting that a rel- atively small amount of clean data is sufficient for the meta-learner to deduce an effective diag- nostic fusion strategy. This finding underscores the practical applicability of our paradigm, as the effort required to curate a small, high-quality dataset is substantially lower than cleaning the entire training set.
90
88
Methoi
86
Saader Devintier
84
Savertio
16.25
160
200
são
Meta-PParaset Size (M)
Figure 7: Impact meta-dataset size (M) on the final Reward Accuracy.
Impact of Deta-DDtaset Noise. A crucial question is how our method performs if the meta-dataset itself is not perfectly clean. To simulate this, we intentionally introduced label-fipping noise into Dieaa (with a base size of M 100) and evaluated the final model's performance. The results presented in 4 As expected, performance gracefully degrades as the noise level in the meta-set increases. However, the method exhibits remarkable tolerance low levels of contamination. Even when Dmcca contains 5% noise, our method achieves a Reward Accuracy of 92.5% + 0.8%. This still substantially higher than Vanilla DPO trained on the main set with 30% noise (which scored approximately 68.5% in our main experiments). This suggests that while a clean meta-dataset is ideal, our paradigm is not overly brittle to minor imperfections, further enhancing its practical utility in real-world scenarios where perfectly curated data is rare.
Table 4: Impact of noise rate in Dieta on final Reward Accuracy (%). The main training data has 30% noise (Golden HH, Llama-2-7B, M = 100).
Meta-Noise Rate
0 %
1 1%
3%
%
Reward Accuracy (%)
96.0 + 0.4
95.5 + 0.5
94.2 + 0.6
92.5 + 0.8
16

---

Published as a conference paper at ICLR 2026
Table 5: A qualitative case study from the Golden HH dataset 0.3) illustrating diagnostic synergy. Despite misleading PPLDiff signal, the high Training Loss and Uncertainty correctly flag the sample as noisy preference containing a factual error, leading low learned weight.
Component
Content / Value
Prompt
'What year did the Riffel Tower open to the public?'
Chosen Response
'The Eiffel Tower, an iconic symbol Paris, officially opened its doors to the public in 1892. It was marvel of engineering for its time.'
Rejected Response
'It opened in 1889.'
Ground Truth
The preference label is noisy. The rejected response factually correct (the tower opened in 1889 for the Exposition Universelle).
Diagnostics
Analysis of the model's intrinsic feedback on the noisy preference pair:
Preference Consis- tency (PPLDiff)
-0.85 Misleading Signal. The higher fluency and length of the incorrect response cause the model to assign a lower perplexity, suggesting the sample is clean.
Learning Difficulty (Training Loss)
1.23 Informative Signal. The high loss value indicates a strong conflict between the instruction to prefer the incorrect re- sponse and the model's 's internal knowledge about the correct date.
Generation Confi- dence (Uncertainty)
0.95 Informative Signal. The model exhibits high token-level entropy (low confidence) when generating the factually incorrect year "1892." indicating a lack of conviction.
Final Outcome Learned Weight
0.15 Correct Outcome. The meta-learner correctly interprets the combination of conflicting diagnostics and assigns very low weight, effectively mitigating the harm from the noisy label.
D ADDITIONAL EXPERIMENTAL RESULTS
D.1 QUALITATIVE ANALYSIS: SYNERGY IN ACTION
As discussed in Section 4.4 the quantitative results and ablation studies strongly indicate that the fusion of multiple diagnostics is the primary driver of method's robustness. To provide more concrete illustration of this mechanism, we present detailed case study in Table 5
This example, drawn from the Golden HH dataset after injecting 30% noise, showcases challenging scenario where a single-heuristie approach relying solely on PPLDiff would fail. The 'chosen" response, while fluent and well-structured, contains a critical factual error. The 'rejected" response is terse but factually correct. An ideal robust alignment method should identify this noisy preference and reduce its influence during training.
As shown in the "Diagnostics" section of the table, the PPLDiff is negative (-0.85), highly mis- leading signal that suggests the model finds the factually incorrect response more plausible than the correct one, likely due to its greater length and more confident-sounding tone. An approach like PerpCorrect, which relies on a positive PPLDiff threshold, would incorrectly treat this sample as clean.
However, our multi-penn cealpiiperseetivee diagnostic system correctly identifies the anomaly. The Training Loss is high (1.23), indicating that forcing the model to prefer the incorrect response creates significant conflict with its existing internal knowledge representation. Furthermore, the Generation Confidence is low, reflected by high Uncertainty score (0.95). A closer look reveals this uncertainty
17

---

Published as a conference paper at ICLR 2026
is concentrated around the generation of the incorrect date, suggesting the model is hesitant or lacks a strong factual basis for this claim.
The meta-learner, having been trained on the clean meta-dataset, learns to recognize this specific pattern: plausible-looking PPLDiff coupled with high loss and high uncertainty is a strong sig- nature of fluent misinformation. Consequently, it assigns very low weight (0.15) to the sample, effectively nullifying its harmful impact on the alignment process. This case study vividly demon- strates that by fusing complementary feedback streams, our paradigm can overcome the limitations of any single diagnostic, leading to a more truly robust and discerning alignment.
D.2 ABLATION STUDY ON
ARCHITECTURE
To assess the stability and generality of our method, we conducted comprehensive ablation studies on the meta-learner (-: W). While our main experiments utilize 2-layer MLP with a hidden dimension of 100, we investigate the sensitivity of the performance the network's depth, width, and overall architectural design. All ablations below are performed on the Golden HH dataset with e 0.3 noise.
Impact of Network Depth. We evaluated meta-learner architectures ranging from to 4 layers while keeping the hidden dimension fixed at 100. As shown in Table performance saturates at layers. The I-layer architecture exhibits substantial performance drop of 1.8%, indicating that the linear mapping is insufficient and the non-linear feature interactions captured by the 2-layer network essential for effective diagnostic fusion. However, increasing the depth to 3 or 4 layers offers no meaningful performance difference, suggesting that the complexity of the diagnostic fusion task is well-bounded.
Table 6: Ablation meta-leamer depth (Golden HH, 0.3).
Depth
Reward Accuracy
1 I-layer
94.2% + 0.6%
2-layer
96.0% + 0.4%
3-layer
96.2% + 0.5%
4-layer
96.1% + 0.6%
Impact of Network Width. We varied the hidden dimension from 50 to 200 while maintaining the 2-layer architecture to test the method's sensitivity to capacity. The results in Table 7 demonstrate high stability. The performance fluctuates by less than I 1% across the 50-200 range. We selected hidden dimension of 100 for our main experiments as it offers robust trade-off between capacity and computational efficiency.
Table 7: Ablation meta-learner width (Golden HH, 0.3).
Hidden Dimension
Reward Accuracy
50
95.5% + 0.5%
100
96.0% + 0.4%
150
96.1 1% + 0.5%
200
96.1 0.6%
Alternative Fusion Architectures. Finally, we compared our 2-layer MLP against other distinct meta-learner designs to verify if more sophisticated mechanisms could yield better fusion. We tested:
Linear Mapping: A simple linear layer without activation functions.
Attention-based Fusion: A mechanism where diagnostics attend to each other via self-
attention.
Gated Fusion Network: A Mfxture-of-Eai -xpertsss style gating mechanism for the diagnostic
signals.
18

---

Published as a conference paper at ICLR 2026
The results in Table reinforce the findings from the depth ablation. The linear baseline suffers a significant 4.3% drop, confirming that the relationship between diagnostics (e.g., how Uncer- tainty modulates PPLDiff) is inherently non-linear, Notably, more complex models like Attention or Gated networks did not outperform the simple 2-layer MLP. This suggests that the diagnostic signals themselves are highly informative and orthogonal (as discussed in Appendix (D.3). requiring only moderate non-lincarity to fuse effectively. The 2-layer MLP is thus justified as the optimal design choice.
Table 8: Comparison of different fusion architectures (Golden HH, 0.3).
Architecture
Reward Accuracy
Lincar mapping
91.5%
0.8%
2-layer MLP
96.0% 1 0.4%
Attention-based fusion
95.8% 1 0.5%
Gated fusion network
95.6% 1 0.6%
D.3 EXPLORATION ALTERNATIVE DIAGNOSTICS
We systematically explored eight candidate diagnostic signals beyond the three used in our main approach.
Candidate Diagnostics. Summaiiss the candidates, their motivation, and correlation with our three core diagnostics.
Table 9: Alternative diagnostic candidates and their properties.
Diagnostic
Motivation
Corr: (PPL)
Corr. (Loss)
Corr. (Unc.)
Gradient Norm
Large gradicnts may indicate
0.42
0.91
0.38
noisy samples
Token PPL Variance
High variance suggests incon-
0.61
0.47
0.73
sistent quality
Attention Entropy
Low entropy may indicate
0.33
0.52
0.68
memorization
Reward Model Score
External signal from pre-
0.58
0.44
0.31
trained RM
Response Length Ratio Embedding Distance
Length bias in preferences
0.27
0.19
0.22
Semantic similarity sponses
re-
0.34
0.29
0.41
Margin (logit diff) Perplexity Rank
Confidence in DPO preference
0.82 0.95
0.67
0.45
Ordinal ranking vs. raw value
0.39
0.28
Redundancy Analysis. our existing signals:
Diagnostics with correlation > 0.7 (bolded) are largely redundant with
Gradient Norm: Nearly perfectly correlated with Training Loss (p = 0.91). This is ex- pected as || R k L for preference losses.
Token PPL Variance: High correlation with Uncertainty ation quality degradation, but variance is less interpretable.
0.73). Both capture gener-
Margin (logit difference): Strongly correlated with PPLDiff (p = 0.82). Margin measures the same signal in logit space.
Perplexity Rank: Nearly redundant with PPL.Dff (p 0.95). Ordinal ranking loses fine- grained information.
Augmentation Experiments.
We tested adding the least correlated diagnostics (Response Length Ratio, Embedding Distance) to fusion:
19

---

Published as a conference paper at ICLR 2026
PPL + Loss Unc. + Length: 96.1% (=0.5% Golden HH 0.3 0.3-only +0.1% improvement with 33% more computation.
PPL Loss Unc. Embedding: 96.0% (+0.5%) significant improvement.
These results confirm our design principle: the three diagnostics are maximally informative and minimally redundant. Each captures distinct failure mode (label inconsistency, learning difficulty, generation quality), and their low pairwise correlations (p < 0.35) ensure complementary coverage. Adding more diagnostics yields diminishing returns while increasing computational overhead and the risk of overfitting the meta-leamer to the small D meta-
D.4 ROBUSTNESS TO ADVERSARIAL NOISE
While our main experiments focus on random and natural noise, we also evaluated robustness against three adversarial noise patterns designed to exploit weaknesses in diagnostic- methods.
Ad Adversarial Noise Scenarios.
1. Targeted High-PPL Flipping: An adversary with knowledge of our PPLDiff heuristic se- lectively flips labels where PPLDiff is already high (top 30% of samples), making detection harder.
2. Strategic Annotation Targeting: Noise is injected into high-impact samples- those with low initial loss (casy learn) that would maximally mislead the model if corrupted.
3. Adaptive Attack: white-box adversary with access to a surrogate meta-learner crafts noise to minimize the diagnostic signature (low PPLDiff, low Loss, low Uncertainty simul- tancously).
Table 10: Robustness under adversarial noise (Golden HH, 30% noise rate).
Method
Targeted PPL
Strategic Target
Adaptive Attack
Vanilla DPO PerpCorrect (Dyn.)
68.9%
67.2%
69.5%
83.4%
85..1% 1%
78.6%
DR-DPO
85.1 %
85.2%
84.8%
Ours (PPL only)
82.7%
88.3%
74.2%
Ours (Fusion)
92.3%
90.7
89.4 %
Key Findings. (1) Single-heuristic vulnerability: PerpCorrect and our PPI.Diffonnl variant are highly vulnerable targeted attacks (78.6% and 74.2% under adaptive attack), confirming that lying on a single diagnostic creates an exploitable weakness.
(2) Fusion provides robustness: Our full fusion method maintains strong performance even under white-box adaptive attacks (89.4%), degrading only 6.6% from the random-noise baseline (96.0%). This is because simultancously fool- ing three orthogonal diagnostics is combinatorially difficult difficult-crfttnn sample with low PPLDiff, low Loss, and low Uncertainty while being genuinely noisy requires adversarial control over model internals beyond label manipulation alone. (3) Complementary coverage: The Targeted PPL at- tack harms PerpCorrect (-11,4%) but our fusion degrades only -3.7% demonstrating that Loss and Uncertainty provide critical backup when PPLDiff is compromised.
This analysis validates that diagnostic fusion is not merely beneficial for benign noise but also pro- vides inherent robustness against sophisticated adversarial corruption strategies.
D.5 PRACTICAL. STRATEGIES FOR DATAS-DTASETT CONSTRUCTION
A practical concern is how to obtain the clean meta-dataset Dmetta in real-world scenarios. We eval- uated three strategies.
Strategy 1: High-Agreement Subset (Recommended). When multi- annottorn data is available, select samples with strong inter-annotator agreement. We tested this on a subset Anthropic- HH with 3+ annotations per sample.
20

---

Published as a conference paper at ICLR 2026
Table 11: Meta-dataset construction strategies (Golden HI,, e 0.3).
Strategy
Meta-Set Size
Est. Purity
Final Reward Acc.
Random sampling
100
70% (by assumption)
96.0% + 0.4%
High agreement (Cohen's 0.8)
50
94% (measured)
96.3% + 0.3%
Expert curation
100
97% (measured)
96.8% + 0.3%
Model-assisted filtering (ensemble RM)
150
88% (measured)
95.7% = 0.5%
Key Findings. (1) Quality over quantity: High-agreement subsets achieve equivalent perfor- mance with only 50 samples versus 100 random samples, confirming our theoretical analysis that meta-set purity is critical.
(2) Expert curation is effective but costly: Manual verification by do- main experts yields the highest purity (97%) and best performance (+0.8%), but requires 2-3 hours of expert time-acceptalle for safety-critical applications. (3) Model-assisted filtering is scalable: Using an ensemble of reward models to identify high-confidence samples achieves 88% purity and only 0.3% performance drop. providing practical middle ground.
Recommendation. For practitioners: (1) If multi-annotator data exists, use high-agreement sam- ples (zero additional cost). (2) For safety-critical deployments, invest expert curation of 100-150 samples. (3) For large-scale production, use model model-assisted filtering with ensemble reward models.
D.6 SUMMARY OF ADDITIONAL ANALYSES
The analyses in this appendix complement our main contributions three ways:
Our extended analysis provides problem-specifie bounds for prefer- ence corruption and formalizes why diagnostic fusion outperforms single heuristics under weakly- correlated noise-a property we empirically observe across all real-world datasets.
Theoretical Grounding:
Methodological Validation: Extensive ablations confirm that (1) our 2-layer MLP fusion archi- tecture strikes optimal balance between capacity and efficiency; (2) our three diagnostics are maximally informative with minimal redundancy (p < 0.35 pairwise); and (3) more complex archi- tectures additional diagnostics yield diminishing returns.
Our method demonstrates resilience against adversarial noise (maintaining 89.4% accuracy under white-box attacks) and provides actionable guidance for constructing clean meta-datasets in real-world settings nigh-agrreemet sampling, expert curation, or model-assisted filtering).
Practical Robustness:
Together with our main experiments, these analyses provide comprehensive evidence that diagnostic fusion offers principled, robust, and practical approach to preference alignment under noise.
E LARGE-SCALE DATASET ANALYSIS AND CASE STUDIES
To understand how our diagnostic fusion adapts real-world noise patterns in large-scale datasets, we provide detailed analysis of noise charaeteristics and representative qualitative examples from StackExchange (10.8M samples) and GPT4AIl (0.8M samples).
E. NOISE PROPORTION ANALYSIS FROM THREE DIAGNOSTIC DIMENSIONS
To systematically identify potentially noisy samples, we computed the 90th percentile threshold for each diagnostic dimension and flagged samples exceeding these values. StackExchange, this cor- responds to PPLDiff > 0.62 (preference inconsistency), Loss 1.18 (high learning difficulty), and Uncertainty > 0.83 (low generation confidence). For GPT4AIl, the thresholds are PPLDiff > 0.71, Loss 0.89, and Uncertainty 0.79. Applying these criteria reveals distinct noise distributions:
The noise distributions differ significantly between anmottaed (StackExchange) and LLM- distilled (GPT4All) data. StackExchange exhibits more label ambiguity (16.8% high loss) due to subjective human voting on similar-quality answers, while GPT4All shows more generation artifacts (11.5 (11.5% high uncertainty) from truncation and degeneration issues common in distillation pipelines.
21

---

Published as a conference paper at ICLR 2026
Table 12: Noise proportions in large-scale datasets by diagnostic dimension.
Dataset
High Loss
High PPLDiff
High Uncertainty
StackExchange
16.8% 4.1%
8.4%
3.2%
GPT4AIl
9.7%
11.5%
Our SHAP analysis reveals that the meta-learner automatically adapts diagnostic importance based on these dataset-specific noise patterns. On StackExchange, PPLDiff importance decreases to 0.712 (from 0.956 on synthetic noise) while Loss rises to 0.458, reflecting the prevalence of label ambiguity. On GPT4AII, PPLDiff maintains 0.823 importance with Uncertainty at 0.382, reflecting the model's ability to detect generation quality issues through token-level entropy.
Notably, 1.3% (StackExchange) and 0.9% (GPT4AII) of samples exhibit simultaneous high values across all three diagnostics- tiagnosties""triplee-niiss cases that represent the most challenging instances. Our fusion achieves 68.2% accuracy these cases versus 52.7% for PPLDiff-only, demonstrating the critical value of multi-perspectiive assessment when all diagnostic signals indicate potential issues.
This analysis validates that method adapts to dataset-specific noise patterns without manual intervention, with naturally occurring noise exhibiting fundamentally different diagnostic signatures compared synthetic label-flipping
E.2 QUALITATIVE CASE STUDIES
We now examine naturally occurring noise from StackExchange (10.8M samples) and GPT4AII (0.8M samples). As reported in Section 4.5 these datasets exhibit distinct noise profiles: Stack- Exchange shows predominantly high-Loss samples (16.8%, reflecting subjective voting), while GPT4AIl shows high-Uncertainty samples (11.5%, reflecting generation artifacts).
Case A: Length Bias in Community Voting (StackExchange)
StackExchange platforms exhibit systematic bias toward longer, more elaborate answers regardless technical correctness. This creates noise detectable primarily through PPLDiff.
Context.
Question
"How can make first layer after raft print at first-layer speed in Slic3r?"
Chosen
(score
"Slic3r doesn't provide this option directly. but you can use post-processing scripts to modify the G-code output..."" [400+ words with code examples and detailed technical explanation]
Tormod:
+3)
Rejected (score 1)
kareem: "You shouldn't 't need to. The raft already provides bed adhesion, so the first layer of the actual print can at regular speed." [40 words, directly addresses the underlying misconception]
Diagnostics
PPLDiff: 0.74 (high), Loss: 1.02 (moderate). Uncertainty: 0.38 (low)
Analysis
The rejected answer has lower perplexity despite negative votes, indicating it's more coherent. The high PPL. reveals StackFxchange's length bias. Loss is moderate (model somewhat confused by the preference), Uncertainty is low (both responses are fluent).
Case B: Label Ambiguity Between Similar Answers (StackExchange)
Context. When multiple answers provide essentially identical advice with only stylistic differ- ences, the preference label becomes arbitrary, creating high-Loss noise.
Case C: Format Contamination (StackExchange)
Context. Raw HTML. and broken URLs from data scraping create high token-level entropy, de- tectable primarily through Uncertainty.
22

---

Published as a conference paper at ICLR 2026
Question
"As 3D printing newbie, how can help the site during private beta?"
Chosen
(score
Zizouz212:
"Private betas love votes! If you have an easy question that's 's specific and high-quality, go ahead and ask it. You can also suggest edits to posts and tag wikis..."
+4)
[120 words, encouraging tone]
Rejected +2)
(score
kenorb: "That's the goal of the site-learn, research, ask. You improve posts via edits, be active in meta, review queues, propose tag descriptions, vote on questions..." [50 words, nearly identical advice]
Diagnostics
PPLDiff: 0.28 (low), Loss: 1.42 (very high), Uncertainty: 0.64 (moderate)
Analysis
Both answers have similar perplexity (PPLDiff zero). The very high Loss indi- the model struggles justify strong preference when content is semantically equivalent. vote differential likely reflects posting time rather than quality.
Question
"Can we change our site's default Stack Exchange logo?"
Chosen
(score
Oscar: "Yes! From this Meta SE answer 'What's the process..." we can read [URLs: https:/nmeta.stackeechaee httpaaan..co////95.... com/a/2983411. multiple broken href; tags and HTML markup)... Feel free add your thoughts." [180 words with heavy markup contamina- tion]
+2)
Rejected +1)
(score
darth pixel: "I'd like to remind my old suggestion :) |image link] here is the original post [link]" [20 words, casual but clean]
Diagnosties
PPLDiff: -0.22 (slightly negative), Loss: 0.68 (moderate-low), Uncertainty: 0.91 (very high)
Analysis
The chosen answer has acceptable perplexity when HTML is tokenized, and Loss elevated. However, Uncertainty spikes dramatically when generating URLs and malformed markup, indicating the model's confusion about whether output text or code.
23


Reviews:
[{"review_id": "Gd5ZCDh5bI", "reviewer": "Reviewer_hQMu", "summary": "This paper proposes a meta-learning paradigm to address the noisy annotations in preference data for RLHF. Specifically, it proposes to diagnose the model itself by using intrinsic feedback, and a diagnostic vector composed of preference consistency, learning difficulty, and generation confidence is employed to adaptively reweight training samples, where noisy or unreliable preference data is down-weighted.\nThe paper provides quantitative and qualitative experimental analysis to demonstrate its effectiveness.", "questions": "My main concern is twofold: one is the generalization ability, which could the approach compare on some general benchmarks with some representative methods, such as DPO series, and another is the analysis of the meta-learner, more ablation analysis is needed.\n\nIf you can provide convincing evidence or clarification, I would be open to increasing the score.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper is well-motivated. Instead of employing a pre-defined criteria, the paper introduces a multi-dimensional diagnostic vector and demonstrates that a meta-learned fusion strategy substantially enhances the robustness and reliability of preference alignment. Moveover, detailed ablations and fine-grained analysis clearly demonstrate that the proposed fusion strategy consistently outperforms any single aspect, with adaptive weighting under varying noise conditions.\n2. This paper is well-organized, the coherent structure and figures clearly demonstrate the core idea.", "weaknesses": "1. Lack of analysis of meta-learner: The meta-learner is consistently a 2-layer MLP. Ablation studies varying depth, width, or overall capacity would help assess the method's stability and generality. Without this, it is unclear to what extent the results depend on a hand-tuned meta-learner architecture.\n2. The method builds heavily on well-established meta-learning frameworks; the provided generalization bound is essentially a direct adaptation of standard meta-learning. Deeper theoretical insights or problem-specific analysis could further strengthen the contribution. \n3. Lack of discussion for generalization ability. The experimental noise is synthetically generated via label flips. Although this enables controlled benchmarking, real-world noise (e.g., conflicting annotators, domain shifts) may exhibit different characteristics. While qualitative examples are provided, additional experiments on general benchmarks, naturally noisy or adversarial datasets would better demonstrate generalizability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xsmRBpeNH5", "reviewer": "Reviewer_rJ2H", "summary": "This paper introduces a diagnostic-driven meta-learning paradigm for robustly aligning LLMs with human preferences in the presence of noisy preference data. The authors propose to empower models to \"diagnose themselves\" during training by fusing three types of intrinsic feedback—preference consistency (perplexity difference), learning difficulty (training loss), and generation confidence (token-level entropy)—to form a diagnostic vector per sample. A meta-learning approach adaptively reweights training samples based on this vector, optimizing for performance on a small clean meta-dataset. Extensive experiments on 2 benchmarks show superiority over state-of-the-art robust baselines.", "questions": "1.\tThe diagnostic vector is a concatenation of three normalized signals. Did the author experiment with more complex fusion architectures for the meta-learner's input beyond a simple MLP on the concatenated vector?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.\tThe paper formalizes the idea that preference reliability for LLM alignment is inherently multi-perspective, motivating the use of preference consistency (perplexity difference), learning difficulty (DPO loss), and generation confidence (entropy) as three synergistic diagnostics. Rather than relying on a single heuristic, the method operationalizes a holistic, adaptive approach.\n2.\tThe use of SHAP to quantify feature importance and uncover non-linear interactions provides deep, novel insights into how the meta-learner works. \n3.\tThe method consistently achieves SOTA performance across multiple models, benchmarks, and noise levels.", "weaknesses": "1.\tAll noise scenarios are simulated via label-flipping at random rates. There is no direct evaluation or demonstration of real-world occurring noisy preferences.\n2.\tAlthough the chosen baselines are quite relevant, it’s better to take more recent methods into consideration.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1rBluV9sre", "reviewer": "Reviewer_wsUf", "summary": "This paper introduces a method to deal with the issue of label noise in preference tuning datasets for aligning LMs. This new method incorporates separate signals that suggest/correlate with noise (consistency, difficulty, and confidence). Specifically, these are combined into a learnable vector representation that is tuned alongside the model parameters themselves during RLHF training. They find that in a variety of noise settings, their method outperforms baselines methods.", "questions": "1. What are some other diagnostic methods you think could be used?\n2. Why was a 2-layer MLP chosen? Was this in comparison with other simple models?\n3. Do you have a sense of what would be a practical way of constructing a \"clean\" set for meta-learning?\n4. What assumptions are being made regarding characteristics of the noise (i.e. is it randomly distributed)? Would it work if the noise is adversarial in some way? I suppose in that case noise is not the best descriptor, but I am curious about this.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. Overall, the paper is very clear and well-written. I found Section 3, the most technically dense part of the paper, to be very easy to follow compared to other papers where similar concepts are used. \n2. To my knowledge, the meta-learning approach introduced by this paper is novel and is an interesting way of combining heuristic signals that have been used in the past as a proxy for label noise. \n3. The authors mention that only about 100 clean meta-samples are required for their approach. This makes the method highly scalable and efficient. Therefore, there is meaningful practical value for using this particular method. \n4. The experiments are sound and the analysis regarding meta-dataset sensitivity will be useful to practitioners using the method.", "weaknesses": "The results seem to suggest that PPLDiff is by far the most important and influential diagnostic criterion. While the ablations show that there is a statistically significant difference between using the fusion of all of the criteria and just using PPLDiff, the difference is very small, and that also seems to be the case in the results presented in Figure 2. That being said, I don't feel as if this is necessarily a demerit against the method itself, but rather I think it would be useful to see results with other metrics as well to see if any other proxy diagnostics could be very helpful. \n\nAnother reason I don't think this fact in and of itself is a demerit is the analysis provided in Section 4.4, which was very interesting. However, I do think the paper would be strengthened by varying some of the important design choices in the meta-learning set-up. In addition to the diagnostics themselves, I think it would be interesting to vary the meta-learner itself to see how sensitive the method is to that choice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NCtUiL5ESq", "reviewer": "Reviewer_EPXP", "summary": "This work presents an approach for robust LLM preference alignment. The approach empowers models to perform self-diagnosis by fusing three key factors, including preference consistency, learning difficulty and generation confidence. Experiments on GoldenHH and OASSIT1 preference datasets  are conducted with simulated noises.", "questions": "Please find the questions in the weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Strengths:\n\n(1)Two more factors are considered to diagnose noise in preference data, which bring more robustness.\n\n(2)A meta-learning based approach is proposed to learn adaptive weights on the three factors.\n\nSystematic analysis of the interplay and relative importance of the three factors are performed.", "weaknesses": "Cons:\n\nIn experiments, the performance gains of the proposed method have been validated by the simulated noise data. However, as shown in Figure 2, the performance difference between various robust DPO variants is not significant, when \\epsilon = 0. Does it mean that the qualities of the two datasets (GoldenHH and OASSIT1) have been very high? Can you validate the effectiveness of the proposed method without noise simulation?\n\nMoreover, the simulation operation for injecting noise follows the protocols in (Kong et al 2024) which propose PPLDiff to measure the preference inconsistency. Consequently, the finding of the predominant role of PPLDiff is inherently expected. Can you give more various noise to simulate the uncertainties and the learning difficulties in preference data?  \n\nMoreover, to investigate the efficacy of the proposed method in the presence of annotation errors commonly found in real-world applications, it is advisable to conduct experiments on more extensive datasets, such as the HuggingFace H4 StackExchange Preference Dataset (with a size of 10 million entries) and the GPT4All dataset (comprising 1 million entries).\n\nSome typos: page 8, the text under Figure 4 is abruptly interrupted by a line break.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)