Robustness of Probabilistic Models to Low-Quality Data: A Multi-Perspective Analysis
Authors: Liu Peng
Yaochu Jin
Venue: ICLR 2026 Poster (2026)
Keywords: data quality; probabilistic model; multi-perspective analysis;
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: This paper investigates a critical challenge in modern machine learning: 
how different probabilistic models withstand low-quality training data. 
Through a systematic, comparative investigation, 
we reveal a stark spectrum of robustness. 
Empirically, we find that autoregressive language models 
exhibit remarkable resilience against both token-level noise 
and structural corruption (for GPT-2, test NLL increases modestly from 
2.87 to 3.59 despite 50\% corruption). 
By sharp contrast, class-conditional diffusion models degrade catastrophically 
under identical noise levels (image-label consistency plummets by 56.81\%), 
while image classifiers show a moderate vulnerability that diminishes with dataset scale.
To explain these discrepancies, 
we analyze the results through a multi-perspective lens 
integrating information theory, PAC learning, and gradient dynamics. 
This framework identifies what informational properties drive robustness, 
why they are required for generalization, 
and how the optimization process achieves this resilience.
These analyses suggest that robustness is heavily influenced by two key principles: 
the richness of conditioning information, which constrains the learning problem, 
and the absolute information content of the training data, which allows the signal from correct information to dominate statistical noise.

Published as a conference paper at ICLR 2026
ROBUSTNESS OF PROBABILISTIC MODELS TO Low- QUALITY DATA: A MULTI-PERSPEECE PERSPECTIVE ANALYSIS
Liu Peng"
Yaochu Jin
Trustworthy and General AI Lab School of Engineering, Westlake University Hangzhou, China LiuPeng NGP Sout look.com
Trustworthy and General Lab
School of Engineering, Westlake University Hangzhou, China sinyaochu@west wawe lake. edu.
ABSTRACT
This paper investigates a critical challenge in modern machine learning: how differ- ent probabilistic models withstand low-quality training data. Through systematic, comparative investigation, we reveal a stark spectrum of robustness. Empirically, we find that autoregressive language models exhibit remarkable resilience against both token-level noise and structural corruption (for GPT-2, test NLL increases modestly from 2.87 to 3.59 despite 50% corruption). By sharp contrast, class- conditional diffusion models degrade catastrophically under identical noise levels (image-label consistency plummets by 56.81% %), while image classifiers show a moderate vulnerability that diminishes with dataset scale. To explain these dis- crepancies, we analyze the results through a multi-perspective lens integrating information theory, PAC learning, and gradient dynamics. This framework iden- tifies what informational properties drive robustness, why they are required for generalization, and how the optimization process achieves this resilience. These analyses suggest that robustness is heavily influenced by two key principles: the richness of conditioning information, which constrains the learning problem, and the absolute information content of the training data, which allows the signal from correct information to dominate statistical noise.
INTRODUCTION
Contemporary deep learning models are trained increasingly vast datasets where the presence of low-quality data is inevitable (Radford et al.) 2018 2019 Brown et al. 2020 Podell et al. 2023bb al 2024). How models contend with such data, however, is far from uniform. Our systematic investigation reveals a stark divergence in robustness across modern probabilistic models: while autoregressive language models and large-scale classifiers are remarkably resilient to high levels of data corruption, class conditiona diffusion models exhibit catastrophic degradation under the same conditions.
This dramatic disparity, which synthesizes observations from privous work on discriminative model robustness (Rolnick et al., 2018) and generative model fragility (Na et al. (023), motivates the central goal of this paper: to move beyond model-specific observations and uncover the fundamental principles governing this behavior. Why do some of the most powerful models Al occupy opposite ends of the robustness spectrum?
To systematically probe this disparity, we conduct a suite of controlled experiments across these representative model families. Our methodology involves dynamically introducing quantifiable, random errors into the training data, allowing us to precisely control the level of corruption. This paradigm allows study the effects of what term low-quality data, which we define functionally as samples where the relationship between inputs, conditions, and target outputs has been corrupted in way that is detrimental the specific learning task.
"Liu is the family name. 1 "Coresponning Author

---

Published as a conference paper at ICLR 2026
To answer this question, adopt multi pellipipeseeeeivee analytical approach, integrating insights from information theory, PAC learning, and gradient dynamics. We hypothesize that the observed disparities can be explained by coherent set of underlying factors. By integrating empirical findings with these theoretical viewpoints, we aim to provide foundational insights for understanding and predicting model robustness in real-world noisy environments.
The key contributions of this work are as follows:
We conduct systematic empirical investigation that validates and quantifies a stark di- vergence in robustness across autoregressive language models, class-conditional diffusion models, and image classifiers, providing controlled evidence for this critical phenomenon.
We propose and apply multi-perspective analytical framework that uses information theory, PAC learning, and gradient dynamics to explain what informational properties drive robustness, why they are formally required for generalization, and how the optimization process mechanistically achieves this resilience.
Through this integrated approach, identify two fundamental factors that govern model robustness: (1) the richness of conditioning information available to the model, and (2) the absolute information content of the training data.
RELATED WORK
The challenge of training on imperfect data is central theme in machine learning, giving rise to rich literature noise robustness. For discriminative models, this is well-established field; the surprising resilience of deep classifiers to label noise is well-documented (Rolnick et al. 2018 ZhangChiyuan al. 202T). leading an ecosystem of solutions, from noise-robust loss functions (Menon et al. [2019] Chen et al., 2020) to techniques for noise correction (Yi & Wu 2019). More recently, attention has turned the fragility of modern generative models. This has spurred new wave of targeted, architectural fixes for issues like noisy labels in dass-conditional diffusion models (Na et al. 2023) and corrupted contexts in language models (Gio et al.) 2024).. In parallel, empirical work has validated the principle that massive data volume can overwhelm supervision noise (Tia et al.) 202I). While these approaches are vital, they focus on fixing individual vulnerabilities rather than explaining their origins.
To analyze such phenomena, our work draws upon several foundational theoretical frameworks. The information-theoree perspective builds the seminal work of Shannon (Shannon 1948) and its application to neural networks, which frames learning as process of preserving a useful signal from noisy inputs (Tishby & Zaslavsky 2015). The PAC learning framework provides formal link between a hypothesis class's complexity (e.g., its typically matched the task's complexity), the required volume of clean data, and the feasibility of generalization (Valiant 1984). Finally, the gradient-based perspective offers a mechanistic view rooted in the extensive literature on stochastic gradient descent (SGD) dynamics, where factors such as the batch size and the nature of gradient noise are known to be crucial for optimization and stable learning (Keskar et 2017).
dimension, which is
Our work departs from the prevailing focus on model-specific engineering to conduct a fundamental comparative investigation. Rather than chasing state-of-the-art performance on individual benchmarks, we aim to isolate the intrinsic properties that govern robustness across diverse model families. We are the first to systematically synthesize these distinct theoretical viewpoints to explain why a stark divergence in robustness exists between autoregressive, diffusion, and discriminative models. By integrating controlled experiments with this multe-perseective framework, we identify two core principles, richness of conditioning information and absolute information content, that provide unified explanation for these disparate behaviors. A more comprehensive review of the literature is provided in Appendix F
2

---

Published as a conference paper at ICLR 2026
3
EXPERIMENTS
3.1
EXPERIMENTAL SETUP
Our experimental methodology is designed to precisely measure the impact of low-quality data under controlled conditions. We introduce noise at ratios (r) from 0.1 to 1.0 relative to the clean data volume, creating effective error rates (c r/(1 r)) up 50.0%. We analyze the results using two complementary paradigms.
Noise Generation Protocols. To establish foundational baseline for intrinsic robustness, primary experiments employ unstructured, random noise. For text-based tasks, we corrupt target tokens by replacing them with tokens chosen uniformly at random from the entire vocabulary. For classification and class-conditional generation, labels are corrupted by replacement with a class chosen uniformly from the C incorrect alternatives. Full pscudocode provided in Appendix [ for reproducibility. While this stochastic corruption isolates the model's ability to extract signal from noise, we also investigate the impact of realistic, systematic errors through structured noise experiments in Section 3.3 and Appendix
Isolating Intrinsic Robustness. For most of our experiments (autoregressive, diffusion, and classification models), our goal is to isolate the model's intrinsic tolerance to noise. To do this, we hold the amount of correct supervision constant by scaling the total training compute by (1+ This design ensures that any performance degradation is direct consequence of the added noise, not a lack of clean data. For stability in high-noise regimes, batch sizes were increased and iterations proportionally reduced to preserve this principle (see Appendix H)).
Primary Paradigm:
Secondary Paradigms: Fixed-Budget and Structured Noise. While our primary experiments evaluate unstructured random noise under scaled training compute (increased total iterations or epochs), we introduce two crucial variations to broaden our analysis. First, to isolate the effect of noise under strict computational constraint, we employ fixed-budget paradigm (constant iterations) where clean tokens are directly swapped with unstructured, random noise (see Table 7 and Appendix D. Second, move beyond random corruption, we introduce structured noise paradigm for sequence-to-sequeme strle, partially trained version of the model. This provides a crucial test of our rich-context hypothesis under more realistic, non-random error distribution that mimics machine- machine-generated artifacts.
experiments (Sec. 3.3). Here, the flawed target data is generated by an
3.2 AUTOREGRESSIVE MODELS FOR TEXT GENERATION ARE ROBUST LOW-QUALITY DATA
To investigate the impact of incor- rect data the training of decoder- only transformer-based autoregressive models, trained GPT-2 models (Radford et al. 2019) on the Open- Web'Tex dataset (Gokaslan 2019). OpenWeb'les
Nogative Likelhood (NLL)
test NLL during training
7.6200
-...
6.1402
al.)
is an open- source replication of the private Web- Text dataset originally used to train GPT-2 and comprises approximately 38 GB of text from 8,013,769 docu- ments. The training set contains ap- proximately 9 billion tokens, and the validation set contains approximately 4 million tokens.
5.2031
3.844g
2.8639
333873
3.3/05
2.7369
100:0100:10
100:30
100:50
100:100
Correct data: Incorrect data
We trained 124M parameter GPT-
Figure 1: Impact of Increased Low-Quality Data on NLL. Results for 100:50 and 100:100 used increased batch sizes and proportionally reduced iterations to maintain training stability and equivalent correct sample exposure.
2 models using the AdamW (Loshchilov & Hutter, (017)) op- timizer. The baseline model was trained for 600,000 iterations. For experiments with added noise, the

---

Published as a conference paper at ICLR 2026
total iterations were proportionally scaled to maintain constant exposure to the original clean data. The baseline batch size was maintained for most settings, but was selectively increased in high-noise regimes, necessary intervention to overcome severe training instability. Full architectural and training configuration details are provided in Appendix H
Figure T shows the negative log-likelihood (NLL) resulting from training language models with different ratios of additional incorrect data. The NLL on the test set represents the final evaluation after training, while the NLL on the (noisy) training set is reported from the end of the training process. Even when trained on data with a high error rate, language models can still achieve good performance on the test set. Notably, high ratios of additional incorrect data (r 0.5, = 1.0) introduced significant instability; training with baseline batch sizes failed to converge due to what we identify as overwhelming gradient noise. To counteract this, was necessary to increase the batch size (doubling it for 0.5 and using twelve-fold increase for 1.0) while proportionally reducing iterations maintain equivalent exposure to correct samples. This necessary intervention provides direct empirical support for the gradient-averagin mechanism discussed in our analysis (Section 4.3). As the ratio of incorrect data increases, the NLL on the clean test set increases only slightly compared to the baseline (trained correct data only), while the NLL on the noisy training set itself increases significantly with higher error rates. This phenomenon demonstrates that decoder-only transformer-basee autoregressive models can learn effectively even in the presence of substantial proportion of incorrect data.
To provide complementary view under fixed computational budget, also analyzed perfor- mance where adding noisy data displaces clean data within constant number of training steps. This analysis, detailed in Appendix reinforces our finding: even as the model attempts fit the corrupted samples (leading to high training NLL), its generalization to the clean data distribution remains largely intact (validation NLL increases only modestly). This further highlights the model's resilience.
3.3 THE PROTECTIVE EFFECT OF RICH CONDITIONING SRQURNCE-TO-SSQURRE MODELS
We hypothesize that model's robustness is pro- foundly influenced by the richness of its con- ditioning information, a principle we formally ground in information theory and PAC learning later in Section 4 Intuitively, rich context severely constrains the possible outputs, reduc- ing the complexity of the function the model must learn (as reflected by lower effective VC dimension), thereby making it less susceptible to noise in the target. To test this prediction di- rectly, we compare two sequeenc-to-ssequence tasks with vast informational disparity: WMT 2014 translation (Bojar et al. 2014) (sparse con- text, 99. 99.9th percentile source length of 153 to- kens) and CNN/DailyMaii (Chen et al.] (016) summarization (rich context, 99.9th percentile source length of 2343 tokens).
30
WMT 2014
CNN/DailyMail
Relative NLL Increase
25
20
15
5
50.0
Effective Error Rate in Target (%)
Figure 2: Relative NLL increase versus target noise. The model trained on CNN/DailyMail, with its information-rich conditioning input, is signif- icantly more robust target corruption than the WMT model, demonstrating how rich context con- strains the learning problem.
To ensure stringent test, we trained models from scratch and introduced structured, non- random errors into the target data using "noisy teacher" paradigm. The results, shown in Figure [ offer a clear empirical confirmation of theoretical framework. At 50% effective enror rate, the NLL for the sparsely-condittoned WMT model degraded by 31.5%. By contrast, the richly-conditioned CNN/DailyMai model was far more resilient, with its NLL increasing by only 17.9%.
This finding provides strong evidence that robustness not an inherent property of an architecture alone. Instead, it is heavily modulated by the information asymmetry between input and output.
4

---

Published as a conference paper at ICLR 2026
When model can draw upon strong, constraining signal from rich context, it can effectively average out and overcome substantial noise in a comparatively now-information target. The full results and experimental details are available in Appendix K
3.4 CLASS-CONDITIONS DIFFUSION MODELS ARE NoT ROBUST TO LOw-QUALIN DATA
To investigate the impact of substantial low-quality data on image generation, we trained class- conditional diffusion models and classifier network separately on CIFAR-I0 and CIFAR-100 (Krizhevsky 2009). After training the diffusion model, we generated images by randomly selecting class labels as conditions. The pre-trained classifier then predicted labels for these generated images. We calculated an accuracy score by comparing these predicted labels with the conditioning labels used for generation.
We employ the EDM framework (Karras et al. 2022) for the diffusion model, using a U-Net architec- ture for the denoiser and ResNet- 18 model as the external classifier. Detailed hyperparameters for the diffusion process, network architectures, and training are available in Appendix
Algorithm B in Appendix used to generate incorrect labels. For specific image, with a probability equal to the effective error rate e, its correct label was replaced by new label randomly selected from the alternative class labels.
Table I:
Ratio of Additional Incorrect Data and Corresponding Classification Accuracy of Generated Images (Consistency with Conditioning Labels) for Image Generation Tasks.
Correct: Incorrect CIFAR-10 Generation CIFAR-100 Generation
100:
94.082%
65.236%
100: 10
84.160%
54.262%
100: 30
69.864%
38.882%
57.876%
30. 30.404%
100: 100
40.630%
16.996%
The results in Table show substantial decrease in the accuracy of the generated images (consistency with the conditioning labels) as the proportion of incorrect training labels increases. For example, on the CIFAR-10 dataset, when 100% additional incorrect data is used (effective error rate e 0.5, corresponding to the 1100:100" condition), the accuracy drops from a baseline of 94.082% to 40.630% 16.996%
For CIFAR-100. the impact is even more pronounced, with accuracy falling from 65. 65.236% under the same conditions. Notably, the Fréchet Inception Distance (FID) scores for these generated images remained relatively stable across different levels of label incorrectness (see Appendix O for details). This suggests that the degradation in performance is primarily due to weakened association between images and their conditioning labels, rather than a general decline perceptual image quality.
3.5
ABSOLUTE INFORMATION CONTENT: CLASSIFIER ROBUSTNESS EMERGES AT SCALE
Table 2: Ratio of Increased Incorrect Data and Corresponding Accuracy for CIFAR Classification Tasks
Correct:Incorrett CIFAR-10 Classification
CIFAR-100 Classification
100:
95.30%
78.96%
100: 10
95.11% %
77.33%
100: 30
90.18%
67. 67.68
100: 50
89.19%
63.71 %
100: 100
85.35%
61.65%
While autoregressive models demonstrated inherent robustness, the behavior of classifiers presents more nuanced picture that powerfully highlights the role of dataset scale. On smaller datasets like CIFAR- 10 and CIFAR-100. ResNet- 18 model trained from scratch exhibits moderate sensitivity to label noise, with performance degrading as corruption increases (Table This establishes a baseline for moderately complex tasks with limited data.
5

---

Published as a conference paper at ICLR 2026
Table 3: Ratio of Increased Incorrect Data and Corresponding Accuracy for ImageNet Classification Tasks
Correct:Incorrect
ImageNet-10
ImageNet-100
ImageNet-1000
100:
62.302%
64.520%
73.784%
100: 10
62.500%
63.360%
73.530%
100: 30
58.929%
57.560%
73.646%
100: 50
54.563%
57.220%
73.684%
100: 100
50.794%
45.920%
74.778%
To investigate whether vast volume of correct signal can provide inherent resilience (a principle we formalize as absolute information content Section (41.2). we scaled our experiments to ImageNet (Deng et al. (209) using a VīT-Base model trained from scratch. The results in Table Jre striking. While the ImageNet- and -100 subsets degrade similarly to CIFAR, the model trained on the full 1.28M sample ImageNet- 1000 dataset becomes almost impervious to label noise. founter-intuitivee performance did not degrade but slightly improved, even when the training data contained 50% incorrect labels under the same setting, effect attribute to the additional training compute in our experimental design.
In high-noise regimes the subsets, it necessary increase batch sizes to stabilize training- an empirical confirmation of the gradient-averaging mechanism we analyze in Section This interven- tion, detailed in Appendix H ensures fair comparison. The extreme robustness on ImageNet-1000 thus provides compelling evidence that sufficiently large volume of correct signal can dominate statistical noise. This robustness is further confirmed by our complementary fixed-budget analysis (see Appendix J. which isolates the effect from increased compute.
4 ANALYSIS
We analyze why autoregressive models and classification models can learn effectively despite sub- stantial low-quality training data, while cass-conditional diffusion models struggle under similar conditions. Our analysis is conducted from three complementary perspectives: information information-thhorctiic. probably approximately correct (PAC), and gradient-basedd This convergence analysis explains what informational properties drive robustness (information theory), why these properties are a formal requirement for generalization (PAC learning), and fow the model mechanistically achieves this resilience (gradient dynamics). The analysis is built upon two fundamental principles. The first is the richness of conditioning information, which fundamentally governs task's complexity. The second is the absolute information content of the data, which provides the learnable signal that can be mechanically extracted from noise via gradient aggregation.
4.1 INFORMATION-THEORTTI PERSPECTIVE
Information theory (Shannon 1948) introduced to quantify information in communication, also offers valuable lens for understanding machine learning as a process of information transfer to model.
4.1.1 RESIDUAL INFORMATION IN Low- NUW-QUALITY DATA
To understand how models learn from corrupted data, we first quantify the amount of instructive signal that survives the introduction of noise. We measure this using relative information loss: the fraction of label uncertainty attributable to data corruption, normalized by the total entropy of the true labels. Let y be the true label and be the observed (potentially corrupted) label from of classes. Assuming a uniform error model where an incorrect label is chosen randomly from the n 1 alternatives with probability Pe, the relative information loss is:
information loss
(1 Pe) log, (1 Pe) Pe loga Pe Pe log, - 1)
H(y)
log,
This formulation (derived in Appendix Q isolates the information-theoreee penalty of label noise itself. Analyzing this equation shows that for large number of classes n, the loss increases

---

Published as a conference paper at ICLR 2026
approximately linearly with the error rate Pe- Additionally, for fixed error rate, the relative information loss decreases as n grows.
These behaviors help explain the general performance degradation trends in our ex- periments (Section 3)) However, the prac- tical impact of n is often coupled with other factors, such as the absolute data vol- ume. The crucial insight from this analysis that instructive information persists as long as the observed labels are not statisti- cally independent of the true labels. This independence occurs at single, precise
1.0
Pe Values:
Relative Information Loss
0.8
P. 0.2
P. = 0.3
0.6
Pe =
P.
0.6
R. - 0.7
Pe
0.2
point: when Pe = (n 1)/n. For er- rates greater than this, the corrupted labels can paradoxically become informa- tive again (e.g.. Pe 1 simply represents a perfectly inverted signal when n = 2). Our analysis and experiments operate in the realistic, information-degrading regime of Pe S (n- 1)/n. Within this scope. a residual signal always exists, allowing a model with sufficient capacity and data to extract meaningful patterns even from substantially noisy datasets. Figure SaarreJilllunn Illlustrates this behavior.
0.0 10a
10
10a
10a
Figure 3: Behavior of Relative Information Loss with
Varying Ps and
4.1.2 ABSOLUTE INFORMATION CONTENT THE PRIMARY DRIVER OF ROBUSTNESS
While the existence of residual information suggests that even corrupted data contains some useful signal, our analysis identifies the absolute information content of the training data principal driver of robustness. We define this as the total, aggregate quantity of information provided by the training data, comprising both the correct information for the input distribution p(z) and the instructive information for the conditional distribution p(y z), where z denotes the input.
It is crucial distinguish what information a corrupted sample provides. An image with an incorrect label, for instance, still contributes to the model's 's understanding of the input distribution, p(z), aiding the learning of robust visual features in manner similar to unsupervised learning. Thus, while provides zero instructive information for the supervised task itself, it still contributes to the absolute information content by enriching the model's 's representation of the input domain. Consequently, un- comupted samples are essential as they provide the instructive component of the absolute information content that correctly guides the model to learn the input-output relationship.
Our experimental design directly investigates this principle. By scaling the total training duration by a factor of (1 + +r. we ensure that across all experiments, the model exposed to a constant quantity of this correct instructive information, holding the absolute information content relatively steady.
The remarkable robustness of the classifier trained on the full ImageNet- dataset a powerful illustration of this concept, especially when compared smaller subsets like ImageNet-10 10 and ImageNet-1000 which suffer much more severe performance degradation under identical noise rates. While the feature extractors learn from all processed images, the final classification is shaped by the immense absolute information content provided by the 1.28 million samples. This aggregate signal is overwhelmingly strong that it provides a clear directive for the learning task, effectively allowing the model to average out and disregard the conflicting gradients from the noisy labels. This establishes that a sufficiently large absolute quantity of information, both in terms of raw input diversity and correct instructive signal, is a dominant factor in ensuring model robustness, explaining why massive datasets can often tolerate significant levels of noise.
4.1.3 THE ROLE OF RICHER CONDITIONING INFORMATION
We hypothesize that robustness is deeply influenced by the information asymmetry between the conditioning variables (inputs) and the target variables (outputs). Richer conditioning variables provide a more constrained and informative context, which can empower a model to overcome noise, particularly when that noise is in comparatively information-sparse target.
7

---

Published as a conference paper at ICLR 2026
This principle is demonstrated across our experiments. In autoregressive language models, the conditioning context of previous tokens. p(next_token||p previous _tokess), is information- compared to the single target token. Similarly, for image classifiers, the input image, p(label] image), contains vastly more information than the simple class label. Both of these model types proved robust when their information-sparse targets (the next token the class label) were corrupted.
Conversely, our cladsicondii diffusion models, p(image class class_label label), represent the opposite scenario. The conditioning variable (a single class label) is extremely information-sparse relative to the high-information target (a complete image). As predicted by our hypothesis, these models were highly fragile when this www-ifoormation conditioning signal was corrupted.
The seuuuence-tossqqueeee experiments in Section 3.3 provide an even more direct and compelling validation of this principle. We compared two tasks where the targets were corrupted: one with a short, less informative conditioning input (WMT 2014, with 99.9th percentile source length of 153 tokens) and one with a long, information-rio conditioning input (CNN/DailyMaill, 2343 tokens). The results were unambiguous: the model with the richer conditioning information (CNN/DailyMaiil was significantly more robust, exhibiting only a 17.9% performance degradation compared to 31.5% for the model with the spanser input.
This demonstrates clear pattern: models are vulnerable when low-information conditions are used guide high-infooe -ithiifformation outputs, but they can be remarkably robust when rich conditioning information provides a strong signal to overcome noise in simpler targets. This establishes the relative richness of the conditioning information key determinant of model's s resilience to low-quality data.
4.2 PROBABLY APPROXIMATELY CORRECT PERSPECTIVE
The Probably Approximately Correct (PAC) learning framework (Valiant 1984) offers theoretical lens through which we can understand the principles of richer conditioning information and absolute information content. PAC theory defines the sample complexity, m, as the minimum number of examples required to learn a concept with low generalization error. For any concept class with Aopnik-Chervonenkis (VC) dimension of d, this sample complexity m is lower-bounded:
m 2 co ( 1og
d log e
where e and 5 are the error and confidence parameters, and co is constant. (Kearns & Vazirani 1994) This inequality reveals how both of our core robustness principles are grounded in learning theory.
First, the theoretical requirement for a minimum total number of samples, m. provides a foundation for our concept of absolute information content. As defined previously, this content comprises both structural knowledge and instructive signal. In deep neural networks, the global sample complexity m largely driven by the high capacity of the hypothesis class (e-g.. the massive parameters of the shared feature extractor). Our ImageNet experiment clearly illustrates this: although subsets like ImageNet-10 and the full ImageNet- 1000 share a similar number of samples per class, the full ImageNet-1000 dataset provides vastly larger aggregate volume of clean examples across all classes. This massive total quantity ensures that, even under severe corruption, the absolute number of remaining clean examples across the entire dataset far exceeds the global sample complexity m required to stabilize the shared learned knowledge. By contrast, the total number of uncorrupted examples in smaller subsets like ImageNet-10 100 is dangerously close to (or below) this critical threshold m, leaving no buffer to absorb the impact of noise.
Second, the VC dimension, d, which reflects the capacity of the hypothesis class the model belongs to, is tightly coupled with the principle of richer conditioning information. Strictly speaking, while d is property of the model architecture, the required value of achieve low approximation error is dictated by the complexity of the conditional distribution being modeled.
Richer Conditioning (e.g., Classification): tasks like pmaage) the conditioning variable (image) is information-richh while the target (label) is simple. The rich input severely constrains the possible outputs, simplifying the learning problem. This allows the task to be effectively solved by hypothesis class with lower effective VC dimension d.
8

---

Published as a conference paper at ICLR 2026
Sparse Conditioning (e.g., Conditional Diffusion): In tasks like p(imagel label), the conditioning variable (label) is information-spate while the target (image) is extremely complex. The sparse input provides very little constraint, meaning the model must learn a far more complex mapping (e.g., high-dimensional score functions). This necessitates a hypothesis class with much higher effective VC dimension d.
According to Inequality 2 higher VC dimension d demands significantly larger number of samples m. condiionniii diffusion models, with their sparse conditioning and consequently higher have an enormous requirement for absolute information content. This makes them exceptionally vulnerable to low-quality data, as noise rapidly depletes the effective number of clean samples below the critical threshold needed for successful learning.
Thus, the PAC framework converges with the information-theoeee perspective, identifying richer conditioning information (which lowers d) and sufficient absolute information content (which satisfies m) as the intertwined, principal drivers of a model's robustness to noisy data.
4.3 GRADIENT- BASED PERSPECTIVE
The training of modern neural networks via backpropagation (Rumelhart et al. 1986) provides mechanistic explanation for how models achieve robustness to noisy data. This perspective highlights how aggregating samples amplifies the coherent signal from correct information while averaging out divergent noise from corrupted data, thereby leveraging the dataset's ' absolute information content.
Within any given training batch, the total gradient, gtotal, can be decomposed into coherent signal from correct samples and divergent noise from incorrect ones:
Σ gnoise__ommee component__
grotal Georrect_sing signall
Here, Icorrect_ séggal represents the consistent directional update from clean samples, guiding the model toward the true data manifold. By contrast, each Groise_compontt coteeeccomooen___ arises from corrupted sample and points in less predictable, often orthogonal, direction. To quantitatively validate this decomposition and the effect of sample aggregation, we analyzed per-example gradients at initialization across different data corruption ratios and batch sizes. The results are summarized in T
Table 4: Quantitative Analysis of Gradient Coherence. Clean gradients exhibit strong, coherent positive alignment (+0.52), while corrupted gradients are directionally random and orthogonal (similarity 0). This disparity allows larger batch sizes amplify the coherent signal relative the noise, systematically improving the Signal-to-Non Ratio.
25% Corruption
50% Corruption
Metric
Batch Size =
Batch Size
Batch Size = 4
Batch Size 8
Directional Coherence (Mean Cosine Similarity)
Clean Clean
+0.52
+0.52
+0.52
+0.52
(Min, Max)
(+0.00, +0.76)
(-0.00, +0.76)
(+0.00, +0.73)
(+0.00, +0.76)
Corrupt vs. Corrupt
+0.001
+0.001
+0,001
+0.001
(Min, Max)
(-0.02, +0.02)
(-0.02, +0.02)
(-0.01, +0.02)
(-0.01, +0.02)
Clean Corrupt
+0.001
+0.001
+0.001
+0.001
(Min, Max)
(-0.03, +0.03)
(-0.03, +0.03)
(-0.03, +0.03)
(-0.03, +0.05)
Aggregated Signal Magnitude (Mean L2 Norm)
Aggregated Clean Signal
6.13
11.37
4.19
7.88
Aggregated Noise Signal
0.84
1.36
1.42
2.06
Signal-to-Noise Ratio
7.31x
8.34x
2.96x
3.83x
Table|4 direct empirical validation of our hypothesis. First, the coherence analysis confirms fundamental disparity: gradients from clean data are consistently and strongly aligned (mean similarity +0.52), while gradients from corrupted data are directionally random, centered symmetrically around

---

Published as a conference paper at ICLR 2026
zero and orthogonal to the clean signal. This holds true regardless of noise ratio batch size. Second, and most critically, the table demonstrates the power of aggregation. For both 25% and 50% corruption levels, doubling the batch size causes the magnitude of the aggregated clean signal to nearly double, consistent with constructive accumulation. By contrast, the aggregated noise magnitude grows a much slower rate, reflecting partial cancellation. Consequently, the signal-to-noiss ratio systematically improves with a larger batch size in all scenarios. The full methodology and setup for this targeted gradient analysis are provided in Appendix M
Beyond these per-step gradient dynamics, this fundamental mechanism of signal amplification has a direct, macroscopic consequence: it stabilizes the overall learning trajectory. To explicitly quantify this, we conducted a complementary experiment measuring inter-batch loss variance, As detailed in Appendix N this dedicated supplementary analysis confirms that gradient aggregation drastically reduces loss variance. Together, these findings provide a complete mechanistic explanation for the empirical interventions required in our main experiments, where increasing the batch size was a necessary step to overcome severe data corruption.
Our specific intervention during the autoregres- sive model training (Section 3.2) vividly illus- trates this principle in action. When baseline batch sizes led to instability in high-noise toregressive model training, We increased the batch size up to twelve-fold to achieve conver- gence. As Figure (b)) illustrates, this directly strengthens the cumulative Georrect_sing siggaal suf- ficiently to dominate the increased, but largely canceling, noise.
Incerrect gradient direction
Correct gradient direction
Figure 4: Interpretation of the gradient-based per- spective, (a) Gradients from correct data are coher- ent, while those from incorrect data are divergent. (b) Aggregating samples in larger batches ampli- fies the correct signal relative the noise.
Therefore, the gradient perspective confirms that aggregating samples the crucial mechanism through which the statistical power of absolute information content is realized, enabling robust learning even with substantial low-quality data. This statistical averaging effect may be a funda-
mental reason why training large models often requires very large batch sizes (Yang et al. 2024
Touvron et al. 2023 Dubey et al.) 2024 DeepSeek-Al 2024).
5 CONCLUSION
This paper confronts a critical challenge in modern machine learning: the impact of low-quality data probabilistic models. Our systematic investigation reveals stark divergence in robustness across model families. We find that autoregressive models are remarkably resilient to significant data corruption, large-scale image classifiers. By sharp contrast, class-conditional diffusion models exhibit catastrophic degradation within our comparative analysis, pinpointing a critical vulnerability,
To explain these disparities, we analyze these results through a multi-perspective lens, integrating principles from information theory, PAC learning, and gradient dynamics to show what informa- tional properties drive robustness. why they are formally required for generalization, and how this is mechanistically achieved. Our convergence analysis suggests that robustness in this context is heavily influenced by two key factors: the richness of conditioning information, which constrains the learning problem, and the absolute information content of the training data, which allows the aggregate signal from correct supervision to dominate the statistical noise from flawed examples. These principles move beyond model-specific observations to provide a more fundamental under- standing of learning dynamics, offering crucial guidance for designing the next generation of reliable models intended for imperfect real-worth data environments.
ACKNOWLEDGMENTS
Many thanks to Hengjie Yu, Yizhi Wang. and Peng Yue for their helpful discussions. This work is funded in part by an International Collaboration Fund for Creative Research of National Natural Science Foundation of China (NSFC ICFCRT) under the Grant no. W2441019.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rombach. Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets, November 2023a.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rombach. Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets, November 2023b.
Ondiej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Level- ing, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Aleš Tamchyna. Findings of the 2014 Workshop Statistical Machine Translation. In Ondfej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Christof Monz, Matt Post, and Lucia Specia (eds.), Proceedings of the Niuth Workshop on Statistical Machine Transla- tion, pp. 12-58, Baltimore, Maryland, USA, June 2014. Association for Computational Linguistics. doi: 10.3115/v 1/W11///1111433000 14-3 /cl.attos///llaaaooooooooo org. 3302/
Zalán Borsos, Raphaël Marinier. Damien Vincent, Eugene Kharitonov, Olivier Pictquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghi- dour. AudioLM: language modeling approach audio generation, 2023. URL http:
/ aaaxivv www.iv.orr//b///2200 2209 03143
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alee Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learn- ers. In Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. wttpppcppppp.p////ccepcccpppp haahh/ html
Curran Associates, Inc., 2020. URL https: 11C.0/0cccc41111111111114341 111111101000011112-0000000001000
Danqi Chen, Jason Bolton, and Christopher D. Manning. A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task. In Katrin Erk and Noah A. Smith (eds.), Proceedings of the 54th Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 2358-2367, Berlin, Germany, August 2016. Association for Computational I. Linguistics. doi: 10.18653/v1/P16-1222 URL.https: /tacsp::///claeee anthooiiin ogy. www.hhhw............ooo.o.
Jian Chen, Ruiyi Zhang, Tong Yu, Rohan Sharma, Zhiqiang Xu, Tong Sun, and Changyou Chen. Label-Rerrieval-Auggmeeedd Diffusion Models for Learning from Noisy Labels. November 2023a.
URL http:://...oooaeeeeeeeee////eeaa
Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao. Enze Xie, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Stapph--S Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis. In The Twelfth International Conference on Learning Representations, October 2023b.
Pengfei Chen, Guangyong Chen, Junjie Ye, Jingwei Zhao and Pheng-Ann Heng. Noise against noise: stochastic label noise helps combat inherent label noise. October 2020. URL https
//openreview.o
Christopher Bishop.
Springer, 2006. httpp::/www.micer wttp::wwwwiireeoo www.mmootcotocmomommmmoooooco..t com/ hes///n-uuue http:///wr../eeeeeerrraooo......
Pattern Recognition and Machine Learning.
URL
pattern- recheerereeeottiiiirrracnnieneee Lew.mmmaahhhieeeeeinnnneneeeeng learning/
DeepSeek-AL. DeepSeck-v2: A strong, economical, and efficient mixture-of-experts language model,
2024. URL htt.::t:://axxx..rrooooo/
11

---

Published as a conference paper at ICLR 2026
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large- scale hierarchical image database. In 2009 JEEE Conference on Computer Vision and Pattern Recognition, pp. 248 248-255, June 2009. doi: 1LL109//VVR.2200522668888
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. In Advances in Neural Information Processing Systems, volume 34, pp. 8780-8794. Curran Asso- ciates, Inc., 2021. UKL.htppss/pppoceiisss...ooopppp
pwwp://app...... files/ httppppprrrerepppppprerarppppp h18p:///000444111111444400000660
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas
Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations, October 2020.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, and et al. The llama herd of models, 2024. URL org/ org/abs/ 2407 217833
Hongfu Gao, Feipeng Zhang, Wenyu Jiang, Jun Shu, Feng Zheng, and Hongxin Wei. On the Noise Robustness of In-Context Learning for Text Generation. November 2024. URL http:////w.eeeeeeee/////////c/ee (0gpcc/eeeeosecloooooooaaaaaaaa2
Gao2)
Aaron Gokaslan, Vanya Cohen, Ellie Pavlick, and Stefanie Tellex. Openwehtext corpus. http:
//Skylion007. http://an00..gtthaa..//////eeeee wttpp//eeeeeeeeeetttoorpoopp
2019.
Youdi Gong, Guangzhen Liu, Yunzhi Xue, Rui Li, and Lingzhong Meng. A survey on dataset
quality in machine learning. Information and Software Technology, 162:107268, October 2023. ISSN 0950-5849, doi: 10. 10.10166f.intsof22233..7777
URL //wtpp:://ww.... htR.itttee//ww..cciennecccc.e.
COm/ /tom:///ccnooeeeerrreeee/////333
Alex Graves. Generating sequences with recurrent neural networks, 2014. URL http:://rxxi
orq/abs/1308.00 0850
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Ad- vances in Neural Information Processing Systems, volume 33, pp. 6840-6851. Curran Asso- ciates, Inc., 2020. URL htllitttaa//ppoccediiiie wwwphinngg..uuipppe.eee nwwripuurp.... CC/ http://ss..cppppe////////eeee/ee (0
html
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruigi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David Fleet, and Tim Salimans. Imagen Video: High Definition Video Generation with Diffusion Models, October 2022a.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, and Tim Salimans, Imagen video: High definition video generation with diffusion models, 2022b. URL htttt::://axiv..oot/// 2210 02303
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, (8):1735-1780, 1997. ISSN 0699-7667, 1530-888X. doi: 10.1162/neco. 1997.9.8 9.8.. 1735.. URL https://direct mit www. /tt.t///aa..aoooaat.a.a////////3
Stuart Hunter. The Exponentially Weighted Moving Average. Journal of Quality Technology, October 1986. ISSN 0022-4065.
Aapo Hyvärinen. Estimation of Non-Noomaliize Statistical Models by Score Matching. Journal of Machine Learning Research, 6(24))695-709, 2005. ISSN 1533-7928.
12

---

Published as a conference paper at ICLR 2026
Jaaming Ji, Kaile Wang. Tianyi Alex Qiu, Boyuan Chen, Jiayi Zhou, Changye Li, Hantao Lou, Josef Dai, Yunhuai Liu, and Yaodong Yang. Language Models Resist Alignment: Evidence From Data Compression. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 1-11-22343, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 97988917625 doi:
URL
2025 acl acw.aong---.o-oo.ooo 11417]
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision. In Proceedings of the 38th International Conference on Machine Learning, pp. 4904-4916. PMLR, July 2021. URL httpt: htttps::/proceed.ingg///o// mww. mwt.: ISSN: 2640-3498.
v1 1/39bb html
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. In Advances in Neural Information Processing Systems, 2022. URL
https:://ppenret
net forum?
Michael J. Kearns and Umesh Vazirani. An Introduction to Computational Learning The- ory. MIT Press, 1994. URL httpp:://dii rectt mit. edw/books /tt///mmoogrg//00/0/ An Int roduct ion to- Computational -aaanna----earaann Theory
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima.
February 2017. URL httpthttt////eereeeeeeeeeeeeeeee
Zhifeng Kong. Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. DiffWave: A Versatile Diffusion Model for Audio Synthesis. In International Conference on Learning Representations, October 2020a.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. DiffWave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations, 2020b. URL httt:ttt///peerree/// net. foorumiidaa--
http://peaaaeieooo.oeeoeoeaaaaaa XB Ymz5J
Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. AudioGen: Textually guided audio generation. In The Eleventh International Conference on Learning Representations, 2022. URL https
//openreview. net./ wwwprrvvview.ne/eororeeeoe
Alex Krizhevsky, Learning multiple layers of features from tiny images. Technical Report TR-2009,
University of Toronto, 2009. Accessed: 2024-07-10.
Kuaishou. KLING AI: Next-generation ai creative studio. https: http://wwkkln..g com/ 2024. Accessed: 2024-09-22.
Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, Yingfang Zhang, Xingchao Liu, and et al. Hunyuan-Dij -DT:: A powerful multi-resolutionn diffusion transformer with fine-grained chinese understanding, 2024. URL http:p:://aar.v.oorg/aaaaaa///aa wtt.://///abs///s/// 08..005..88444 08748,
Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D. Plumbley. AudioLDM: Texi-to-Audio Generation with Latent Diffusion Models, September 2023.
I. Loshchilov and F. Hutter. Decoupled Weight Decay Regularization. In International Conference on Learning Representations, November 2017.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Er- mon. SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations. In International Conference on Learning Representations, October 2021.
Aditya Krishna Menon, Ankit Singh Rawat, Sashank J. Reddi, and Sanjiv Kumar. Can gradient clipping mitigate label noise? September 2019. URL https: http://:///pp/eeee...o wet.:/eww.eeeetteee
13

---

Published as a conference paper at ICLR 2026
Byeonghu Na, Yeongmin Kim, HeeSun Bae, Jung Hyun Lee, Se Jung Kwon, Wanmo Kang, and ll-chul Moon. Label-Noise Robust Diffusion Models. October 2023. URL https
//openreview net/ Eor htt.://www..../..ooooo//o//.....
Aäron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Koray Kavukcuoglu, Conditional image generation with PixeICNN decoders. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, pp. 4797 Curran Associates Inc., 2016. ISBN 978-1-5108-3881-9
OpenAl. Video generation models as world simulators. https: Sttps://openai. opooapeii......... com/ index 2024. Accessed: 2024-09-22.
vi
Published by OpenAl.
OpenAl, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, and et al. GPT-4 technical report, 2024. URL http: sttt://aaxiv.. wwtpaxiiiw.rggaa... 2200 08774
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis. In The Twelfth International Conference on Learning Representations, October 2023a.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis, 2023b. URL http:///waxxv.../abbbbbbbe wttp.//r/r/abb/.....o.oo.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, 2019.
David Rolnick, Andreas Veit, Serge Belongie, and Nir Shavit, Deep Learning is Robust to Massive Label Noise, February 2018.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional Networks for Biomedical Image Segmentation. In Nassir Navab, Joachim Hornegger, William M. Wells, and Alejandro F. Frangi (eds.), Medical Image Computing and Computer-Assisted Intervention MICCAI 2015, Lecture Notes in Computer Science, pp. 234-241, Cham, 2015. Springer International Publishing.
ISBN 978-3-319-24574-4 doi: 10.
David E. Rumelhart, Geoffrey Hinton, and Ronald Williams. Learning representations by back-propagating errors. Nature, 323(6083):533-536, October 1986. ISSN 1476-4687. doi:
10.1038/323533a0, URL
Maximilian Seitzer. pytorch-fid: FID Score for PyTorch. httpps://github..cmmmeeet/// pytorch fid
August 2020. Version 0.3.0.
C. E. Shannon. A mathematical theory of communication. The Bell System Technical Journal, 27(3): 379-423, July 1948. ISSN 0005-8580. doi: 10. 1002/. 1538-7305. 194....19......88..
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang. Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-A-Vides Text-to--iide Generation without Text-Vis Video Data. In The Eleventh International Conference Learning Representations, September 2022a.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang. Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. a-keea- Text-to- video generation without text-video data. In The Eleventh International Conference on Learning Representations, 2022b. URL TRLLhttss: : /tttps://openne /oppn revi ew net www.:/ee//ffuutteeem forum? wtw.rrummmiinnfyyygogo
14

---

Published as a conference paper at ICLR 2026
Jascha
Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Proceedings of the 32nd International Confer- on International Conference on Machine Learning Volume 37, ICML' 15, pp. 2265-2265, Lille, France, July 2015. JMLR.org.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising Diffusion Implicit Models. In International Conference Learning Representations, October 2020a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data dis- tribution. Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL wtwp://wweeedennn..nee.erur.pp.e
http://ceeeeaaaaaaa.eee....ppppa fires/fiie///eeeeeeee lese/pperr
.html
Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-Based Generative Modeling through Stochastic Differential Equations. In International Conference on Learning Representations, October 2020b.
Gemini Team, Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry, Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, Ioannis Antonoglou, Rohan Anil, Sebastian Borgeaud, and et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024. URL httpp:/arxiv..rr//aae 34C3.05533
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), pp. 1-5, April 2015. doi: 10. 10.1109/TT.220157133116 URL
leee. org/ httgg//ooomee////333333
Hugo Touvron, Louis Martin, and Kevin Stone. Llama models, 2023.
Open foundation and fine-tuned chat
L. Valiant. A theory of the learnable. Symposium on the Theory of Computing, 1984.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https::/ neurinpg..neuursse hash/ Bash/3ffee24335 3f5ee243347de Saaaaeeeeee3331177003000aaaaaaaa Abstaaaaaaaaaaaaatt..aaaaa.aa... .htmllltttmmmmm
neurinu httprrpsssepppppeffeeeeseeeeeeee saapperrfffespppeeee htt.lll/pepppe/2217
Pascal Vincent. A Connection Between Score Matching and Denoising Autoencoders. Neural
Computation, 23(7):1661-1674 July 2011. ISSN 0899-7667. doi: 10.1162/NECO_a_ 000122.
Song Wang, Zhen Tan, Ruocheng Guo, and Jundong Li. Noise-Robust Fine-Tuning of Pre- trained Language Models via External Guidance. In Houda Bouamor, Juan Pino, and Ka- lika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 12528-12540, Singapore, December 2023a. Association for Computational Linguistics. doi: 10. 10.18653/v1/2023 findinng--emllp.844 URL httpss//aclaantooooy..oooo
Findings-emnip.e 834/
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolific- Dreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation. In Seirth-Seventh Conference on Neural Information Processing Systems, November 2023b.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, and al. Qwen2 technical report, 2024. URL http:tt//aa/xw..rg///////// abs/
2407. 10671
Dongehao Yang, Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu. Diffsound: Discrete Diffusion Model for Text-to-Sound Generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:1720-173 1733, 2023. ISSN 2329-9304. doi:
15

---

Published as a conference paper at ICLR 2026
Kun Yi and Jianxin Wu. Probabilistic End-To-Enu Noise Correction for Learning With Noisy Labels. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7010- 7018, Long Beach, CA. USA, June 2019. IEEE. ISBN 978 1-72813-29---8.. doi: 10.1 109//P./ 2019.00718. URL
/ttpee/een////uuue////m///////// ese. org dww.mentuumeecooooco/occ / 8953202/
Hong-Xing Yu, Haoyi Duan, Junhwa Hur, Kyle Sargent, Michael Rubinstein, William T. Freeman, Forrester Cole. Deqing Sun, Noah Snavely, Jiajun Wu, and Charles Herrmann. WonderJoumey: Going from Anywhere to Everywhere. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6658-6667, June 2024. doi: D0.1109/CVPP527333224...00
ZhangChiyuan, BengioSamy, HardtMoritz, RechtBenjamin, and VinyalsOriol. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, February 2021. doi: 10.1 1453444674.
Zhanxing Zhu, Jingfeng Wu, Bing Yu, Lei Wu, and Jinwen Ma. The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects. In Pro- ceedings of the 36th International Conference on Machine Learning, pp. 7654-7663. PMLR, May 2019. URL https: httpass/pppcceedings.. mlr wwwpp//weseeeeepeeeee 2057//hu19e 9e html ISSN: 2640-
3498.
Alon Ziv, Itai Gat, Gael Le Lan, Tal Remez, Felix Kreuk, Jade Copet, Alexandre Défossez, Gabriel Synnaeve, and Yossi Adi. Masked audio generation using a single non-autoregressive transformer. In The Twelfth International Conference Learning Representations, 2023. URL https:
7/ //openeeieww net/ f
Difan Zou, Jingfeng Wu, Vladimir Braverman, Quanquan Gu, Dean Foster, and Sham M. Kakade. The Benefits Implicit Regularization from SGD Least Squares Problems. November 2021.
URL https: /opeervve/epeeeiieww net ftr.mm -0-00r0n Y-dgp.
16

---

Published as a conference paper at ICLR 2026
A DISCUSSION
Decoder-only transformer-based autoregressive models for text generation are largely insensitive to low-quality data, which may partially explain their success (Radford et al. [018 2019 Brown et al., 2020, OpenAl al. 2024). By contrast, class-conditional diffusion models exhibit greater sensitivity to data quality, suggesting that training such models requires a larger volume of high-quality data. This dichotomy is explained by our core finding: when rich conditioning information (e.g., a long text prefix) is available, models can overcome noise in oow-information target (the next token). When the conditioning is sparse (a single class label), the model is far more vulnerable. However, in the text-to-image task (Podell et al.) 2023a), text conditions provide more information than categorical conditions, thereby reducing the amount of high-quality data needed for training.
A primary goal of our study was to establish this foundational understanding using a controlled, unstructured noise model, a necessary first step analogous to using a standardized test to measure system's 's baseline capabilities. Our framework, however, also provides crucial foresight into the effects of more complex, structured noise, which presents a vital avenue for future work. Unlike the random errors studied here, which create diffuse gradient noise, structured noise introduces coherent, competing learning signal. For example, a dataset where images of wolves are consistently mislabeled "husky" would create a strong, incorrect gradient direction. Our gradient- -aaee perspective predicts that this systematic error would be significantly more difficult to overcome, as the -averaging effect would be less effective against persistent, biased signal, prediction we where systematic mislabeling led to a catastrophic drop in
confirm experimentally in Appendix L classifier accuracy that was not observed with unstructured noise. In scoping our work, we distinguish between two types of structured noise that fall outside our primary research question. The first is correctable noise, such as the systematic wolf/husky error mentioned above, or the lexical misuse of "Complement" vs. "Compliment," which can often be solved contextually by other models. The second is noise from inherent ambiguity, such as the Trolley Problem, which lacks a single ground truth even for humans. By focusing on unstructured noise, work addresses the more fundamental challenge of a model's intrinsic ability to find signal amidst stochastic corruption, a prerequisite for tackling these more complex scenarios.
The comparative validity of our theoretical framework is grounded in the clear disentanglement of model learning objectives. A class-conditional diffusion model, for instance, must solve two difficult problems simultaneouslly learning the distribution for high-quality images and learning the correlation between those images and their labels. A classifier, by contrast, only needs to learn the correlation. We argue that our analysis is valid because our findings reveal clear disentanglement of these two problems. For the class-conditional diffusion model, we show that label noise does not impact its ability to model image quality. Its capacity to learn the image manifold remains unimpaired, as evidenced by the stable FID scores detailed in Appendix The failure is catastrophic but also highly specific: it is isolated entirely to the label correlation, leading a massive drop in image-label consistency as shown in Table I In stark contrast, the image classifier is largely robust. However, the sensitivity it does exhibit is clearly isolated to its correlation mechanism. This isolation is evident when comparing two factors: a moderate degradation in per-sample accuracy (Tables 2 and against near-perfect preservation of the marginal label distribution, which is quantitatively confirmed in Appendix (KI. Divergence < 0.0003). By isolating the correlation as the symmetric point of vulnerability, we can make direct and insightful comparison, validating our analysis.
Our analysis intentionally focuses on models trained from scratch, rather than the dominant pre- training and fine-tuning paradigm, to establish controlled, foundational understanding of robustness. This methodological choice is crucial for a clear interpretation of our results. Pre-trained models already possess deep understanding of the world from their initial training, which acts as a powerful but confounding factor. By training from scratch, we remove this variable, allowing us to better isolate and understand the principles governing model's robustness. Our findings then offer strong evidence that this robustness is shaped by the richness of conditioning information and the absolute information content of the data. Furthermore, the impact of fine-tuning can be transient: models exhibit strong tendency to revert to their pre-trained behaviors, phenomenon known as "elasticity" (Ji et al. 2025). Understanding how robustness is established in the initial training phase is therefore paramount, this phase instills the core properties of the model. Our work provides this essential baseline, upon which future investigations into the more complex dynamics of fine-tuning with noisy data can be built.
17

---

Published as a conference paper at ICLR 2026
The findings of this research contribute to deeper understanding of how different probabilistic models handle imperfections in training data. This enhanced understanding can positively impact the machine learning community by enabling potentially estimate the data quality requirements for a given model by considering the information asymmetry between its inputs and outputs. If the input is information-rich, data quality constraints can be relaxed; otherwise, larger volume of high-quality data is necessary. As Al capabilities improve, driven in part by such foundational research, there potential for accelerated productivity across various sectors. However, it is also crucial to recognize that more powerful AI, stemming from better grasp of its learning mechanisms, could also be misused for malicious purposes not developed and deployed responsibly. Therefore, continued research on Al safety, ethics, and governance is paramount along with advancements in model capabilities.
principled approach to data curation. One could
18

---

Published as a conference paper at ICLR 2026
B LIMITATIONS
The central goal of this paper is to provide a systematic and foundational analysis of how core model properties affect robustness. To achieve this, our experimental design primarily employs simplified noise model: the dynamic introduction of unstructured, random errors. This approach ensures that the error rate is precisely quantifiable and reproducible, allowing us to isolate the effects of our core principles.
While our main experiments use unstructured noise isolate core principles, we also validate our framework's predictive power on structured noise. In our sequence-to-sequenee experiments (Section 3.3p. we apply a form of targeted, structured noise the output. Furthermore, our analysis of systematic label corruption in classifiers (Appendix [L] confirms that our framework correctly predicts increased model fragility under such conditions. The primary limitation, therefore, is not the absence of structured noise analysis, but that study does not systematically compare various forms of more complex, correlated noise (e.g., where errors depend on the input data). Exploring these scenarios is crucial next step.
Additionally, our work has some other scope limitations. First, computational constraints precluded training diffusion models very large-scale datasets. Second, we did not perform direct analysis of the number of classes (n) as an independent variable, since its effects are inherently entangled with dataset size and model capacity. Third, we acknowledge that direct comparison across tasks is challenging, as no unified metric exists for objectively scoring text, image, and classífication models against one another. This is particularly relevant our structured noise experiments comparing translation (WMT 2014) and summarization (CNN/DailyMaai)) different tasks. this was deliberate choice test our hypothesis under clear disparity in context richness while controlling for model architecture and training configuration. We contend this is more insightful proxy than intra-task comparisons, where a model trained on long-context summarization task, for instance, might learn trivial copying heuristic, or a long-contex translation task could introduce output length as a new confounder. Our comparative insights are therefore derived from the starkly different relative degradation patterns each model exhibits against its own clean-data baseline. Finally, our experiments intentionally employ well-established and representative model architectures rather than the latest ofate-of-the-ant systems. This choice is crucial for ensuring findings are attributable to fundamental model properties, rather than confounding effects from specific, highly-tuned components of particular SOTA model. The contribution of this work lies in analyzing principles of robustness, for which these architectures serve as clear and effective testbeds.
While these are fundamentally
C REPRODUCIBILITY
Our work is designed to be fully reproducible. The complete source code, configuration files, and analysis scripts are publicly available prctps://githie For full and transparent account of our methodology, we refer the reader the detailed appendices, which document the precise training configurations (Appendix (H). noise generation protocols (Appendix G). and other experimental specifics that form the basis of our findings.
cww. Limm//Luuenggggg NGP
D LLM USAGE STATEMENT
Large Language Models (LLMMs) were utilized as an assistive tool in the preparation of this manuscript and its associated code. The L.LM's role included: (1) improving the grammar and clarity of the text; (2) generating boilerplate code snippets; and (3) assisting in the articulation of authors arguments.
The fundamental scientific contributions, including the formulation of the key ideas, the experimental design, and the final interpretation of the results, are the original work of the human authors. The authors have critically reviewed, validated, and take full responsibility for all text and code presented.
19

---

Published as a conference paper at ICLR 2026
E PROBABILISTIC MODELS
Probabilistic models are widely used in machine learning to learn distributions from data. After training, the learned probabilistic model approximates the underlying data distribution. Probabilistic models can be broadly categorized into two types: generative models and discriminative models.
Generative models aim learn the joint probability distribution Pdata (x. y) the data distribution Pdota (x). By learning this underlying distribution, generative models, such as those that model Pmodel (x), can generate new data samples x that resemble those drawn from Paato (x). Conditional generative models, which model Pmodel y), generate data x based on specific imputs y.
In contrast, discriminative models directly learn a decision boundary or the conditional probability Pmodel (y x) of a label y given an input x. They focus on predicting the label for a given input rather than modeling how the data itself is generated. Classification models are prominent example of discriminative models, where the goal is to assign an input to one of several predefined classes Generative models generally require more sophisticated mechanisms to model complex distributions compared to discriminative models (Christopher Bishop 2006).
Recently, generative models have achieved remarkable success across various domains, including text generation (OpenAl et al. 2024) Team et al. 20244 Dubey et al. 2024 Yang et al. 20244 DeepSeek
AI 2024). image generation (Ho et al. [2020] Song et al. 2020a, Song & Ermon |019]] Dhariwal
2019|Dharival &
Nichol| 2021 Karras al. [022] Podell et al. (0023b). video generation (Kuaishou) 2024 OpenAL
2024 Blattmann et | [Sineer et al. [2022b]]H Ho et al. 2022b). and audio generation (Borsos)
et al. 2023 Kreuk al.
2002
Ziv et al. [023]] Kong et al. (020b). Transformer-basse autoregressive
models Vaswani et al. 2017) and diffusion models (Ho et al. (020) have demonstrated exceptional
capabilities in these areas.
E.1 AUTOREGRESSIVE MODELS
Consider a sequence of random variables x = "is....,p)), where each x; belongs to a defined domain. An autoregressive model decomposes the joint probability p(x) as:
D p(xa xed).
p(x) = p(x1) II d=2
Specifically, for text generation, autoregressive models generate the next token conditioned on previous tokens (Brown et al. 2020). while for image generation, they can generate the next pixel conditioned on previous pixels (Oord et al. 2016). Recurrent neural networks (Graves) 2014) (such as long short-term memory networks (Hochreiter & Schmidhuber) 1997)) and transformers (Vaswani cct al. 2017) can be used as autoregressive models to generate data. Decoder-only transformer-based autoregressive models currently prevalent for text generation (Radford et al. 20188] 2019 Brown
et al. 2020 OpenAl et al. 2024) and audio generation (Borsos et al. 2023 Kreuk et al. 2022
E.2
DIFFUSION MODELS
Diffusion models Sohl-Dickstten et al. 2015: Ho al. 2020 Song & Ermon 2019; Song ct al.
2020b) have achieved remarkable success across various domains, including image generation (Chen
ct al. 2023b Meng et al. 2021 Podell al. 2023a), video generation (Ho et al. 2022a Singer et
2022a Blattmann et al., 2023a). audio generation (Liu et al., 2023 Yang et al.
2023 Kong et al.
2020a and more (Wang et al. 2023b Yu al.. 2024).
Diffusion models generate data by gradually denoising pure noise into meaningful data samples. The EDM formulation for diffusion models (Karras et al. 2022), proposed to elucidate the design space of diffusion models, is employed in this work to examine the influence of incorrect training data.
Assume Pdota (x) is the data distribution with standard deviation ddata- Let og = F > 0 > ON min 2 0 be sequence of decreasing noise levels. We denote p(x; or) as the marginal distribution of clean data samples from Pdota after being corrupted by i.idd. Gaussian noise with standard deviation Thus, p(x; σ,)) represents the distribution of data with noise level o; In practice, the distribution at the maximum noise level, p(x; max)) (where o mmax = oo), is indistinguishable from standard Gaussian noise. Diffusion models first sample random noise xo NO0,, o max I) and
20

---

Published as a conference paper at ICLR 2026
then sequentially denoise it according to the noise levels. The result x thus aims to sample from the data distribution Pdata (x)..
The probability flow ordinary differential equation (ODE) (Song et al. 2020b) is the deterministic counterpart of the stochastic differential equation (SDE), whose solutions describe a diffusion process.
The probability flow ODE can continuously increase or reduce the noise level of the data depending on the direction of time:
dx = - (t) log (I))dt,
where the dot denotes the derivative with respect to time, and Vx p(x; σr) is the score function (Hyvärinen] 2005), which points in the direction of steepest ascent of the log- bog-probability density O)
The denoiser D(x; d), which predicts clean data y from a noisy imput x = y + n (where y ~ Pdate and n (O, o² I)), is trained by minimizing the following denoising score matching objective (Vincent 2011
Ey.n D(y + n; - yyl
From the trained denoiser, the score function Vx log p(; o)) can estimated as:
D(x, σ) -
Vx Vx
During inference, sampling is performed by substituting this estimated score function back into the probability flow ODE. Starting from initial Gaussian noise sampled at the maximum noise level, Xg ~ (0, o a I), numerical ODE solver (such as Euler's or Heun's method) is employed to integrate the trajectory backwards along the decreasing noise schedule from max down to min- This deterministic integration process ultimately yields the final clean data sample x from the learned distribution Pdato:
21

---

Published as a conference paper at ICLR 2026
F RELATED WORK
The challenge of training models on imperfect data is foundational issue in machine learning. Our work contributes by systematically analyzing how these imperfections affect modern probabilistic models, moving beyond model-specific fixes to uncover the general principles that govern robustness. This section situates our contribution by reviewing the literature on the data quality problem and the theoretical principles that inform our perseepperpecctvve analysis.
F.1 DATA QUALITY AND ROBUSTNESS IN DISCRIMINATIVE MODELS
The problem of data quality extends beyond simple label errors to include a range of imperfections like missing values and feature inaccuracies (Gong al. 2023), all of which constitute form of information corruption. Historically, the study of robustness to such corruption has centered on discriminative models. well-documented that deep classifiers can be surprisingly resilient to massive label noise (Rolnick et al., 2018), finding that stands in tension with their ability to memorize random data Chinnun et al. 2021). This observation has spurred the development of rich ecosystem of methodological solutions, including techniques for noise correction YY & Wu 2019) and the design noise-robust loss functions (Menon et al. 2019 Chen et al. 2020).
F.2 EMERGING FRAGILITY IN GENERATIVE MODELS
While discriminative models have proven robust, the implications of information corruption for modern generative models present distinct and more recent research frontier. These models often tasked with learning highly complex, high-dimensional distributions, making them potentially more sensitive to noise.
Recent work has begun to document these vulnerabilities and propose targeted fixes. For instance, the sensitivity of class-conditional diffusion models has led specialized solutions, such transition- aware score matching (Na et al. 2023) and retrieval-augmenner training (Chen et al. (202aa). Similarly, for language models, methods have been developed mitigate noisy contexts during in-context learning (Gao et al., 2024) or fine-tuning (Wang et al. 2023a).
more fundamental question. Instead of asking how to fix single model's sensitivity, we provide the first systematic, comparative analysis to explain why these starkly different robustness behaviors emerge. The fragility of diffusion models, in our view, is not a bug to be fixed but a key piece of evidence in this analysis.
Our work shifts the focus from these model-specific solutions
Complementing this picture the principle that massive data scale can often compensate for low data quality. The success of models trained on a billion noisy image-text pairs is powerful demonstration of this effect (Jia et al. 2021). Our findings on large-scale classifiers align with this. We unify these seemingly disparate empirical observations under proposed principles, suggesting that the richness of conditioning information and the absolute information content are the primary factors heavily influencing the observed spectrum of robustness.
F.3 THEORETICAL FOUNDATIONS FOR ANALYZING ROBUSTNESS
Our analysis seeks to explain the observed disparities in robustness by synthesizing insights from three distinct but complementary theoretical viewpoints. While our convergent application of these perspectives is novel, each grounded established literature.
The cetemmattionThhoretie Perspective frames learning as process of extracting useful signal from noisy data. Our analysis of robustness through the lenses of "richness of conditioning informa- tion" and "relative information loss" is a direct application of foundational concepts like entropy and mutual information (Shannon 1948). This perspective allows us to quantify how much usable signal remains in corrupted data and explain why models with information-rich conditions (e.g., ... an image for a classifier, a long token history for an LM) are better equipped to handle noise in their targets than models with sparse conditions (e.g., single class label for diffusion model). This follows tradition of analyzing neural networks through an enformmtion-thhorcee lens (Tishyy & Zaslavsky 2015).
22

---

Published as a conference paper at ICLR 2026
The PAC Learning Perspective provides a formal link between a task's complexity, the amount of data required, and the feasibility of generalization (Valiant) 1984). The theory establishes that more complex concept classes (i.e.. those with higher Kapnik-Chervonent clean samples to learn effectively. This principle helps to formally explain why inherently complex generative tasks, such modeling the high-dimensional distribution of natural images, have higher demand for information and are thus more sensitive to data corruption than comparatively simpler classification tasks.
dimension) require more
The Gradient- -aaeed Perspective offers a mechanistic explanation for how learning occurs amidst noise at the optimization level. The dynamics of stochastic gradient descent (SGD) are central deep learning, and the inherent noise in the gradient estimation is known to have a regularizing effect (Zou) et al. 202Th. Our analysis builds on modern studies of these dynamics, which have highlighted the crucial roles of batch size in navigating the loss landscape (Keskar et al. (017) and the anisotropic nature of gradient noise in escaping sharp minima (Zhu et al. 2019). This literature provides firm
basis for our argument that aggregating samples (e.g.. through larger batches) strengthens the coherent
"signal" from correct data against the chaotic "noise" from corrupted data, enabling effective and stable learning.
23

---

Published as a conference paper at ICLR 2026
G ALGORITHMS
This section provides the algorithms used to calculate the error rate and generate low-quality data.
Algorithm Algorithm Calculate Scaled Training Duration (assuming constant batch size) and Effective Error Rate e.
Require: r: The ratio of additional incorrect data relative to original correct data (c.g., r=1.0 means
100% additional incorrect data)
Norig: The number of original epochs or iterations
Ensure: 2 O, Norig > 0
1. Nnew + Norig x (1 + r)
2. Xo
Nagw
3. refurn Nnew;
Algorithm 2 Algorithm Generate Incorrect Text Data
Require: e: The effective eπror rate (calculated as r/(1 + r), see Algorithm I
V: The size of the vocabulary
data: The correct text data
B: The batch size
L: The sequence length
Ensure:
1. idr + random int(0, len(data) - L, B) [Random starting indices)
2. X + dota[B, idz, idæ L] (Extract input sequences)
3. Y < dala B, idr + 1, idæ + + 1] (Extract target sequence (shifted by 1))
4. mask <- rand_ land_like(Y)) < e (Create mask for introducing errors)
5. rand vals randint lYke((, low 0, high V) Gennraaee random values for errors)
6. Y [mask]] f- rand paandvvllsppp [ad[ssassk)] (Replace tokens where mask is true with random tokens)
7. return X.Y
Algorithm 3 Algorithm to Generate Incorrect Image Labels
Require: e: The effective error rate (calculated as r/(1 + r), see Algorithm y: The true class label
C: The number of classes
Ensure: O,C 0
1. if rand < e then
3. possible_labels labees...moeee)))
2. possible
labels list(range(C))
4. incorrect label < randoaa ccailpo__chcceppssibbeeee labels)
5. return incorrect_label
6. else
7. return y
8. end if
24

---

Published as a conference paper at ICLR 2026
H TRAINING CONFIGURATION DETAILS
This section provides a summary of the batch sizes and training durations (iterations or epochs) used for the autoregressive language model experiments (Section 3.2) and the ImageNet classification experiments (Section 3.5p. The configurations were designed to ensure that the total number of number of samples processed was scaled by factor of (1 +r) relative to the baseline (where r the ratio of added incorrect data), while the number of correct samples processed remained equivalent to the baseline.
H.1 AUTOREGRESSIVE MODEL (GPT-2) TRAINING CONFIGURATION
The GPT-2 model architecture used (the 124M parameter version) consists of 12 transformer blocks. Each block sequentially applies Layer Normalization, Causal Attention, a second Layer Normaliza- tion, and Multi-layer Perceptron (MLP). Each Causal Attention layer utilizes 12 heads. The model employs an embedding dimension of 768, a vocabulary size of 50,257, and has approximately 124 million parameters. Models were trained using the Adam W (Loshchilov & Hutter 2017) optimizer with a weight decay of 0.1, 31 0.9, 82 = 0.95, and a maximum learning rate of 6 x 10 The baseline model (0% added incorrect data) was trained for 600,000 iterations.
The baseline GPT-2 model (r 0) was trained for Noni 600,000 iterations with baseline batch size of Bbase 491,520 tokens (12 samples/GPU 1,024 sequence length gradient accumulation steps x 8 GPUs). For experiments with incorrect data, batch sizes and iterations were adjusted as detailed 5
Table GPT-2 Training Configuration on Open WebTTex. Norig 600, 000 iterations. Bhase the baseline batch size. Iterations are adjusted to maintain (1 + r) scaling of total number of samples processed relative to baseline, keeping correct sample exposure constant.
Correct:Incorredd (r)
Batch Size
Iterations
100:0 (r 0)
X Bhase
Norig (600,000)
100:10 (r (l1))
1 X Bbuse
1.1 x N Vrig (660,000)
100:30 (r = 0.3)
X Brase
1.3 x Norig (780,000)
100:50 (r 0.5)
2 X Bbase Norig (1 + 0.5) /2 (450.000)
100:100 (r = 110)
12 X Bbose
Norig x (1 + 1.0) 12 (100,000)
H.2 DIFFUSION MODEL AND CLASSIFIER CONFIGURATION
For the class-conditional diffusion models, we employ the EDM (Krrras et al. 2022) frame- work with training settings: data 0.5, Pmean
-1.2, Patd 1.2. For sampling. we use Tmin = 0.002, maxx 80, p = 7, and sleps 18. The denoise network is a U-Net architecture (Ronneberger et al., 2015 Song et al. (220b) with 15.7 million parameters. For training, we used batch size of 128, a learning rate of 0.0001 with 200 warm-up epochs, and an exponential moving average decay rate of 0.9993 (Hunter 1986). The classifier model is ResNet-18 (He et al. 2016). trained for 200 epochs on CIFAR- 10 and CIFAR-100, respectively.
H.3 IMAGENET CLASSIFICATION (VIT-BASE) TRAINING CONFIGURATION
For the ImageNet experiments, we used the ViT-Base architecture (Dosovitskiy al. 2020), which has 86M parameters. The baseline ViT-Base models (r 0) for ImageNet classification tasks were trained for N, = 300 epochs with a baseline batch size of Bg 128 per GPU. For experiments with incorrect data, batch sizes and epochs were adjusted as detailed in Table
25

---

Published as a conference paper at ICLR 2026
Table 6: ViT-Base Training Configuration on ImageNet Subsets and Full ImageNet. No 300
epochs. Bo = 128 (per GPU). Epochs are adjusted to maintain (1 + r) scaling of total number of samples processed relative baseline, keeping correct sample exposure constant.
Dataset
Innrect:Incorrec (r) Batch Size (per GPU)
Epochs
ImageNet-10
100:0 (r 0)
Bo (128)
No (300)
100:10 (r 0.1)
Bu (128)
1.1 x Ng (330)
100:30 (r 0.3)
Bu (128)
1.3 X No (390)
100:50 (r 0.5)
Bp (128)
1.5 X No (450)
100:100 1.0)
2 x Bo (256)
(1 + 1.0)/2 x No (300)
ImageNet-100
100:0 (r = 0)
Bp (128)
No (300) 1.1 x No (330) (1+ 0.3) x No (195)
100:10 0.1)
Bo (128)
100:30 (r 0.3)
2 x Bo (256)
100:50 (r 0.5)
2 x Bu (256)
(1+ 0.5) x No (225)
100:100 (r = 1.0)
4 x Bu (512)
(1 1.0)/4 / x No (150)
ImageNet-1000
100:0 (r = 0)
Bu (128)
Nu (300)
100:10 (r 0.1)
Bn (128)
1.1 x No (330)
100:30 (r = 0.3)
Bp (128)
1.3 x No (390)
100:50 (r 0.5)
Bo (128)
1,5 No (450)
100:100 (r = 1.0)
Bo (128)
2.0 X No (600)
I FIXED TRAINING COMPUTE FOR GPT-2
Table 7: Language Model NLL with Fixed Total Training Compute.
Ratio of Clean Noisy Data
Training NLL Validation NLL
100: 0
2,7369
2.8650
100:10
3.8744
2.9758
100:30
5.3622
3.1646
100:50
6.2423
3.3455
100:100
7.6048
3.6525
To further isolate the effect of noise from computational budget, we fixed the total training compute (i.e., total number of training steps) across all noise ratios. The results, presented Table 7 provide further quantitative detail on this divergence. As the proportion of noisy data increases, the training NLL on the noisy data rises substantially, showing the model is attempting to the corrupted samples. By contrast, the validation NLL on clean data increases only modestly. This demonstrates the model's s resilience; while its performance on the training distribution degrades, its generalization the true, clean data distribution remains largely intact.
additional analysis where
J FIXED TRAINING COMPUTE FOR IMAGENET CLASSIFIER
To provide a complementary view, we also conducted an analysis on ImageNet-1000 with fixed computational budget, where adding noisy data means reducing the proportion of clean data seen per epoch. The results (Table show that while training accuracy degrades significantly as the model attempts to fit the noisy labels, test accuracy remains remarkably stable, dropping by less than 3% even at a 50% error rate. This reinforces the finding that the model effectively learns from the true signal while averaging out the random noise.
26

---

Published as a conference paper at ICLR 2026
Table 8: ImageNet- 1000 Classification with Fixed Total Training Budget.
Ratio of Clean to Noisy Data
Training Accuracy
Test Accuracy
100:0
94.244%
73.784%
100:10
92.728%
72.992%
100:30
87.162%
72.302%
100:50
80.533%
71.854%
100:100
66.870%
71.093%
K DETAILED EXPERIMENTAL SETUP FOR SEQUEENEE-TO-SQQUENN ROBUSTNESS
This section provides comprehensive overview of the experimental setup for the sequence-to- sequence robustness investigation presented in Section 3.3 Our objective was to rigorously compare the robustness of Transformer models in short-context versus long-context generation tasks when trained with structured target noise, while carefully controlling for confounding variables.
K.1 DATASETS AND PREPROCESSING
Short-Context Task (Short-to-Short Generation): We utilized the WMT 2014 English- German machine translation dataset. To ensure comparable data volume with the long- context task, the full WMT' 14 training set was subsampled to 287,113 examples. The validation and test sets remained the original WMT' 14 splits.
Long-Context Task (Long-to-Short Generation): We used the CNN/DailyMaii sum- marization dataset. Its training set naturally comprises 287,113 examples, providing an identical training data volume to the subsampled WMT' 14.
Tokenization: For fair comparison, both tasks employed separate, task task-specific Byte-Level BPE tokenizers, each trained on its respective dataset's full text (source and target). A crucial control was setting the vocabulary size identically to 32,000 tokens for both WMT" 14 and CNN/DailyMai tokenizers. This ensures equivalent embedding layer capacity across models.
Sequence Lengths: We used maximum sequence length of 256 for WMT 2014 (99.9th percentile) and 2048 for CNa/ This 2048-token limit for CNN//ailyMail covers the vast majority of articles (95th pet: 1711; 99th pet: 2102; 99.9th pet: 2343), with target summaries capped at 256 tokens.
K.2 MODEL. ARCHITECTURE AND TRAINING
Model: A standard Encoder-Decoder Transformer architecture employed for both tasks. Models were trained entirely from seratch.
Identical architectural hyperparameters were used across both tasks: 6 encoder layers, 6 decoder layers, 512 embedding dimension (dmodet), 8 attention heads, 2048 feed-forward hidden dimension, and a dropout probability of 0.1.
Hyperparameters:
Optimizer: Adam optimizer with a learning rate of 1 x 10 and gradient clipping at 1.0.
Training Duration: All models were trained for 50k training steps, ensuring that models were exposed to consistent number of total samples (clean + noisy) for each noise ratio, adhering to the "fixed-budget" paradigm for noise analysis.
K.3 STRUCTURED NOISE GENERATION PROTOCOL
To introduce realistic, structured low-quality data into the target sequences, we employed "noisy teacher" approach:
Noisy Teacher Training: For each task (WMT') 14 and CNN/DailyMaill, a clean Trans- former model (the "Noisy Teacher") was trained its respective clean dataset for early,
27

---

Published as a conference paper at ICLR 2026
fixed number of steps (e.g.. 5,000 steps). This early-stage model is capable of generating text but produces outputs that are less coherent and accurate than a fully converged model, mimicking common forms of machine-generiin
emror..
2. Noisy Target Generation: The "clean_souue souunsssureee""ee inputs from the training sets were fed into their respective "Noisy Teacher" models to generate "noisy_target sequences, For machine translation, this produced poorly translated German sentences given English source. For summarization, this produced incomplete or inaccurate summaries given an article source.
3. Mixed Training Datasets: New training datasets were constructed where a specified per- centage of the "clean_tare" satrnett""" sequences were randomly replaced with these "noisy_target" sequences. Noise ratios of 0.1, 0.3, 0.5, and 1.0 were applied as other experiments, corresponding to effective error rates of 0.0909%, 0.2307%, 0.3333%, and 0.5%. The "clean source" inputs always remained uncorrupted.
Table 9: Negative Log-Likelihood (NLL) on WMT 2014 and CNN/Daily aii with varying levels of structured target noise. These are the full results supporting the analysis in Section 3.3 Lower NLL is better.
Ratio of Clean to Noisy Data
NLL (WMT 2014)
NLL (CNN/DallyyMa))))
100:0
2.5488
3.3859
100:10
2.7591
3,5246
100:30
2.9625
3,6942
100:50
3,0777
3,8015
100:100
3,3525
3,9931
28

---

Published as a conference paper at ICLR 2026
L ANALYSIS OF ROBUSTNESS TO STRUCTURED NOISE
primary goal of our study was to establish a foundational understanding of robustness using a controlled, unstructured noise model. However, acknowledge that real-world data imperfections are often structured. To test the predictions of our analytical framework under this more challenging condition, we conducted an additional set of experiments on CIFAR-10 and CIFAR- 100 using structured noise protocol.
The experimental setup, including the ResNet- model architecture and all training hyperparameters. was kept identical to the classification experiments in Section 3.5 to ensure direct comparison. The sole modification was the noise generation mechanism. Instead of replacing a label with a randomly chosen incorrect class, we applied a systematic and consistent error: with probability corresponding to the effective error rate, a true label y was deterministically replaced with (y + 1) (mod C), where C' is the total number of classes. This creates coherent, competing signal, all instances of a given class, when corrupted, are mislabeled as the same incorrect class.
The results, presented in Table 10 reveal dramatically different picture of robustness compared to the unstructured noise scenario.
Table 10: Impact of Structured Label Noise on CIFAR Classification Accuracy. Unlike the diffuse gradients from random noise, the coherent incorrect signal from systematic mislabeling leads to catastrophic performance decline, especially at high corruption rates.
Correct:Incorret CIFAR-10 Accuracy
CIFAR- Accuracy
100:0
94.17 94%
75.54%
100:10
94.15%
75.83%
100:30
91.28%
74.05%
100:50
87.99%
62.44%
100:100
40.72%
33.39%
While the model shows resilience low levels of structured noise, its performance collapses higher ratios. The contrast with unstructured noise is stark. For example, at 50% effective error rate (the 100:100 condition) on CIFAR-10, accuracy plummeted 40.72%, whereas the model maintained an accuracy of 85.35% under the same level of unstructured noise (Table 2). A similar catastrophic drop observed for CIFAR 100, from 61.65% (unstructured) to 33.39% (structured).
This outcome provides strong validation for the gradient-based perspective detailed in Section 4.3 Unstructured, random noise generates divergent gradients (E, Inoise_commen coniise_commooent__ that are direc- tionally varied and can effectively averaged out, allowing the coherent signal from correct data @correct eiigna))) to dominate. Structured noise, however, creates coherent but incorrect gradient signal that systematically pulls the model parameters toward wrong data manifold. This introduces a persistent, biased signal that cannot be canceled out through aggregation. The model is thus forced to learn a competing, incorrect hypothesis, leading to severe performance degradation. This experiment therefore confirms that our analytical framework not only explains robustness to random noise but also correctly predicts the increased fragility of models when faced with systematic errors.
29

---

Published as a conference paper at ICLR 2026
M EXPERIMENTAL DETAILS FOR GRADIENT COHERENCE ANALYSIS
To quantitatively validate the claims made in our gradient-based perspective (Section 4.3 we conducted a dedicated experiment to analyze the directional properties and aggregate magnitudes of per-example gradients. This analysis the source of the data presented in Table
Our methodology mirrored the experimental context of primary autoregressive model experi- ments (Section 3.2). We used a randomly initialized 124M parameter GPT-2 model, with the same architecture detailed in Appendix H and sampled data from the Open WebText dataset. We applied the same on-the-fly, unstructured noise protocol detailed in Section The analysis focused on the word token embedding layer (trrans formrr wte. we weiight ght.).
Per-example gradients were computed using the torch. func vmap transform. We calculated two sets of metries across 200 batches for each experimental condition: (1) Directional Coherence, measured by the pairwise cosine similarity between gradients from clean, corrupt, and mixed pairs and (2) Aggregated Signal Magnitude, the L2 norm of the sum of all clean gradients and, separately, all corrupt gradients within each batch. The Signal-to-Noiss Ratio (SNR) was defined as the ratio of the mean 1.2 norm of the aggregated clean signal to the mean 1.2 norm of the aggregated noise signal.
The results, as shown in Table 4 provide strong empirical support for our theoretical claims. The analysis revealed clear disparity in the directional coherence of the gradients, and we observed how the signal-to to- to-noise ratio consistently improves with larger batch sizes.
N EXPERIMENTAL DETAILS FOR LOSS STABILITY ANALYSIS
To provide quantitative evidence for the gradient-averaging mechanism discussed in Section 4.3 conducted dedicated experiment to measure the stability of the training process under noisy conditions.
Objective The goal of this experiment not measure generalization, but to quantify the consistency of the training signal itself. We hypothesized that while individual batches containing noisy data would produce chaotic gradients, aggregating samples into larger "global batch" would yield a much more stable and consistent update direction. We use the inter-batch variance of the training loss direct proxy for the stability of the aggregated gradient.
The experiment was conducted using the final model checkpoints from two of our GPT-2 training runs: the baseline model trained 100% clean data, and the noisy model trained with 50% effective error rate (the "100:100" condition).
Methodology
The measurement process was follows:
Global Macro-Batch Definition: A "global macro-batch" represents a single, large-scale gradient update step. Its size is defined (micro-batel size per GPU x gradient accumula- tion steps X number of GPUs).
2. Loss Calculation: drawn from the OpenWebTex training set. The appropriate noise ratio (0% for the clean model, 50% for the noisy model) was applied on-the-fly. We recorded the training loss for each micro-batch on each GPU.
For each macro-batch. we processed multiple micro-batches of data
3. Averaging:
The losses from all micro-batches within a single global macro-batch were averaged to produce single, scalar loss value for that macro-batch.
4. Statistical Analysis:
We repeated this process for 200 independent global macro-batches. The final reported metries in Table IT are the mean and standard deviation calculated over these 200 macro-batch loss values.
Analysis These results provide direct quantitative evidence of how sample aggregation stabilizes learning. For a noisy model (50% corruption), the mean loss is significantly higher (+e 7.58) compared to the clean baseline (=s 2.84). It is vital to distinguish this high mean loss from the net direction of the parameter update. The elevated loss is an expected consequence of the objective accommodating the 50% corrupted labels, reflecting necessary compromise in fitting the noisy data.
30

---

Published as a conference paper at ICLR 2026
Table 11: Impact of Global Effective Batch Size Gradient Signal Stability. The high mean loss for the noisy model is an expected consequence of fitting noise, whiles the sharp reduction in loss standard deviation with larger batches demonstrates increased training stability through gradient cancellation.
Noisy Model (50% Corruption)
Clean Model (Baseline)
Global Batch Size
Mean Loss
Std. Dev. (x10-))
Mean Loss
Std. Dev. (x10
480
7.5806
9.45
2,8366
17.50
960
7.5811
7.08
2.8377
13.17
1920
7.5809
4.58
2,8373
8,85
3840
7.5805
3.61
2.8369
6,43
7680
7.5807
2.40
2.8375
4.44
However, the stability of the learning process is revealed by the loss variance. At first glance, the noisy model in Table [T] appears more stable, exhibiting lower loss standard deviation than the clean baseline. This is a statistical artifact: the consistently high, low-variance loss from corrupted random targets statistically dampens the natural, higher variance from the clean data. The crucial insight, therefore, comes not from the absolute variance but from its trend. As the global effective batch size scales from 480 to 7680, the inter- batch standard deviation of the loss, direct proxy for gradient stability, is reduced by approximately 75% in both noisy and clean scenarios. This sharp reduction signifies that the aggregated gradient provides a stable and consistent update direction. Although the noisy samples dampen the overall gradient magnitude, the coherent signal from the correct samples remains dominant after the divergent noisy gradients partially cancel each other out. This enables reliable optimization trajectory that, over many steps, allows the model to learn the true data distribution, explaining its strong generalization despite the high training loss.
31

---

Published as a conference paper at ICLR 2026
O FID FOR IMAGE GENERATION
Table 12 shows the FID calculated for the diffusion model in Section FID was calculated using 50,000 generated images and the original dataset images, employing the "pytorch-fid" package (Seitzer 2020). Even with an increased proportion of incorrect conditioning labels in training, the FID scores remained largely unchanged. The relatively stable FID scores across different levels of incorrect data suggest that the observed drop in classification accuracy for classcconditioal diffusion models is primarily due to mismatch between generated images and their conditioning class labels, rather than degradation in the perceptual quality of the generated images themselves.
Table 12: Ratio of Increased Incorrect Data and Corresponding FII) for Image Generation.
Correct: Incorrect CIFAR-10 Generation CIFAR-100 Generation
100: 0
3.49
5.38
100: 10
3.68
5.70
100: 30
3.66
6.09
100: 50
3.66
6.12
100: 100
3.62
6.28
P ANALYSIS OF LEARNED LABEL DISTRIBUTIONS IN CLASSIFIERS
This section details supplementary experiment conducted to quantitatively validate the claim made in our Discussion (Section A.. The goal is to demonstrate that the image classifier successfully learns the marginal label distribution, even when its per-sample conditional accuracy degraded by label noise. This provides the empirical basis for our argument that the classifier's 's sensitivity noise is isolated to its conditional guidance (correlation), not its understanding the output space's 's structure.
For this dedicated analysis, replicated the training process for the CIFAR-10 classification experiments presented in Section 3.5 This involved retraining the ResNet- 18 models under identical architectural and hyperparameter configurations for each noise level. While minor variations exist due to training stochasticity, the final test accuracies of these replicated models are consistent with those reported in Table 2 confirming that they exhibit the same fundamental robustness characteristics.
The evaluation process was as follows:
Each newly trained model (corresponding to effective error rates 0%, 9.1%% 23.1 %, 33.3%, and 50%) was run the full, clean CIFAR-10 10 test set (10,000 images).
We collected the complete of 10,000 predicted labels generated by each model.
We then compared the statistical distribution of these predicted labels against the true, uniform distribution of the test set labels (1,000 samples per each of the 10 classes).
To quantify the similarity between the predicted and true label distributions, employed two standard metrics:
Kullback-Leibler (KL) Divergence: Measures how one probability distribution diverges from second. A KI. Divergence value close to zero indicates that the two distributions are nearly identical.
Total Variation Distance (TVD): Measures the total difference between two probability distributions. A TVD value close also signifies high similarity.
The results, summarized in Table 13 reveal stark contrast between the model's conditional perfor- mance and its grasp of the marginal label distribution.
As shown in the table, while the model's test accuracy, which is measure of its per-sample conditional mapping ability, (/labellimage) degrades under high noise ratios, the KL Divergence and TVD remain extremely low and stable across all conditions, A KI. Divergence of ss 0.0002
32

---

Published as a conference paper at ICLR 2026
Table 13: Impact of Label Noise Conditional Accuracy Learned Marginal Distribution (Repli- cated CIFAR-10 Runs). While per-sample accuracy degrades, the KL Divergence and TVD remain exceptionally low, indicating the model consistently learns the true underlying label distribution.
Correct:Incorret
Effective Error
Test Accuracy (Conditional)
KL Divergence Total Variation Dist. (Marginal)
Ratio
Rate
(Marginal)
100:0
0.0%
93.85%
0.000228
0,0088
100:10
9.1%
94.08%
0,000241
0.0089
100:30
23. 13
92.11%
0,000068
0.0045
100:50
33.3%
87.96%
0.000152
0.0076
100:100
50.0%
86.28%
0.000155
0,0077
signifies that the distribution of the model's 10,000 predictions on the test is statistically almost indistinguishable from the true uniform distribution.
This provides powerful empirical support for the argument presented in Discussion. This demonstrates that the classifier successfully learns the correct marginal distribution of the output space. Even when its conditional, per-sample predictions reproduce the true statistical frequencies of the test set. The performance degradation is therefore isolated to the conditional guidance mechanism (the correlation). This finding crucial, as validates our comparison with the class-conditional diffusion model, which exhibits an analogous failure mode: its knowledge of the output structure (image quality) is preserved, while its conditional guidance (label correlation) collapses.
less accurate, aggregate predictions
33

---

Published as a conference paper at ICLR 2026
Q RELATIVE INFORMATION Loss
Let represent the true label and X the observed label provided to the model during training (which may be corrupted from y with probability Pe). Let n. be the number of label classes, and Pe be the error rate, which is the probability that any given label is incorrect. Additionally, assume the classes follow a uniform distribution, such that p(i) = #/-
The entropy of the true labels under a uniform distribution is:
((y) = Σ ((i) logg =
Σ
! 72
log2
1 = log, n
(8)
the labels are mislabeled with an error rate Pe, the observed labels are correct with probability of assume the incorrect classes follow uniform distribution, meaning
1 Pe- Furthermore, each piece of data can be mislabeled as any of the - incorrect labels with probability n-1- The conditional entropy is then:
Hly x)
Σ p(t)
(1-pe) logz Pe) + Σ
Pe
log2
Pe 71
(9)
jf1
Since p(i) = 1 for all i:
H(y x) = Σ p(i)
(1 - po) 1og2(1 (1 - -po) +
Σ jzz
Pe
log2
Pe
(10)
71
1
po) 1ogg(1 (1-82(11100 Pp (n
Pe
log2
Pe
(11)
11
n1
Pe) logg (1 Pe) + (n Pe
Pe
log2
(12)
n
-
P.) loga (1 p.) pe log, P.. P.. loga (n
1)
(13)
If we use the difference between the entropy of the true labels and the mutual information to represent information loss, then:
information loss HHy) - 1(x;y)
(14)
Hy)) - (H H(y x))
(15)
-y x)
(16)
The ratio of the information loss to the total entropy, which we define as the relative information loss, becomes:
information loss
-(1 Pe) logz(1 po) Pe logg Pe Pe log 1)
(17)
Hly)
log2
For be independent of y, the observed label must provide no information about y. This occurs when the probability of "observing the correct class" is equal to the probability of "observing any specific incorrect class." " Under uniform label noise, this reduces to:
P(x=
P(x -) Vj
(18)
Substituting the noise model probabilities:
Pe
Pe
(19)
11
34

---

Published as a conference paper at ICLR 2026
Solving for Pe:
(n - 1)(1 Po) = = n 1 (n2 1)
Pe = 1 npe
(20)
1
Pe T
(21)
Thus, when Pe = العة the observed labels x contain information about the true labels y, and the relative information loss reaches its maximum value of 1.
35


Reviews:
[{"review_id": "QISwkENRWb", "reviewer": "Reviewer_rdHq", "summary": "[Please note that, unlike for the other papers in my batch, I am expert only for a subset of the material in this paper, so my review and score is overall pretty reserved/ weak confidence. I will of course participate in the discussion period, and am amenable to changing my score. I apologize if some of my questions seem basic; answering them will nevertheless help me engage more effectively in the discussion period. I will do my best to give a fair assessment, or recommend that the AC down-weight my review if by the end of the discussion period if I have not met my own (I think high) bar for substantive participation.]\n\nThe paper studies how different classes of probabilistic models (autogressrive LLMs, sequence-to-sequence transformers, class-conditional diffusion models, image classifiers) behave when trained with noisy (low quality) data. The work systematically injects controlled random corruption into inputs or labels and compares how performance changes across these architectures/ training dynamics. \n\nHere is my summary of the setup and contributions (please let me know if you think I missed something important):\n\n**Core setup**\n- Noise model: random uniform corruption of tokens or labels; corruption ratios from 10–100% (effective error rate $\\tfrac{r}{1+r}$\n- 4 types of models, as discussed above \n- 2 types of experiments: (a)  constant clean-signal exposure, scales total compute by $1+r$ to keep the number of correct samples constant; (b) fixed budget, noisy data displacing clean data\n- Overall, finds that AR models are pretty resilient; seq-to-seq models with richer conditioning can be made more robust; diffusion models can degrade a ton; classifiers are more sensitive on smaller datasets; some nice results showing that gradient averaging can help", "questions": "I've integrated by questions in the \"weaknesses\" section so that they are directly next to observations I've made. I'm hoping this is clearer than separating them out.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper has many strengths:\n\n- I don't think I've ever seen a paper put this extent of models into one controlled framework. I think the design here is very careful, and highlights meaningful differences between these model types and training dynamics\n\n- The theoretical framing is clean. It ties together a variety of ideas (information theory, PAC, gradient perpsective) to try to explain what's going on here, to give more explanation beyond the high-level takeaways (e.g., that the richness of the context matters).\n\n- The experiments are well constructed to support the theoretical analysis; they seem very careful and contained. \n\n- Overall, the writing does not over-lcaim causal mechanisms and is very clear about limitations (e.g., scale). I appreciate this a lot, given the ambitious nature of this paper. I think this is a clear/refreshing strength of the work.", "weaknesses": "As stated in my summary, please note I am not an expert on all material in this paper. I've combined observations that I think might be weaknesses with related questions.\n\n**Noise model**\n\nAll injected noise is random and unstructured (uniform token or label replacement). In practical settings, low-quality data are often structured (e.g., correlated, systematically mislabeled). Is it fair to say that the results therefore demonstrate robustness to _stochastic corruption_, not necessarily to _realistic dataset noise_?\n\n**Experimental design**\n\nIn the constant/clean-signal experiments, increasing compute by $1+r$ changes the effective LR, regularization. I think the isolation of robustness, as a result, is not entirely clean. Can you please discuss this furhter? \n\n**Theory vs. experiments**\n\nGradient-averaging explanation is intuitive but the experiments don't go into this, unless I missed something. There's no ablation separating variance reduction from bias.\n\n**A couple of instances of slight over-claims**\n\nE.g., I think it is over-generalized to say things like decoder-only LMs are largely insensitive to low-quality data; the writing in general is very careful, but things like this should be hedged. For that experiment, the study uses one model and random noise, not realistic web contamination or fine-tuning noise.\n\n**Novelty re: theory**\n\nThis isn't a big deal, but wanted to ask about it. My take take is that the theory isn't super novel; the principles elicited restate known relationships between sample complexity, conditioning, SNR. What's nice here is the unification, but I don't think it's fundamentally new.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IG3MS5unWd", "reviewer": "Reviewer_XFTw", "summary": "The authors of this paper provide a review of generative models' evaluation in the context of corrupted (noisy) data, taking into account several levels of noise injection, and explore their corresponding robustness. They observe that autoregressive language models, are  more resilient to low quality data, compared to diffusion models.\nIn order to explain these discrepancies, they analyze the results through employing tools from information theory, PAC learning, and gradient dynamics. Through their analyses they conclude that robustness is affected by two factors, namely the richness of conditioning and the absolute information content of the training data, where the former is driven by the learning problem and the latter is related to the data per se", "questions": "I would like to ask the authors if they could please :\n\n-- compare coth generative models under same task and same metrics;\n-- employ the SOTA of both generative models whilst analyzing their performance", "limitations": "", "rating": 2, "confidence": 3, "soundness": 1, "presentation": 1, "contribution": 2, "strengths": "In this paper the authors provide a comprehensive review of two classes of generative models: autoregressive models for text generation and class-conditional diffusion models, in the presence of noisy/low quality data, for several signal to noise ratios.\nIn order to perform their analysis they employ metrics from a spectrum of perspectives, namely information theory, PAC learning, and gradient dynamics. They conclude that the robustness of the method is dependent on the task (richness of conditioning) and the training data (absolute information content)", "weaknesses": "The authors test two different types of generative models under different tasks, for different types of data (categorical and continuous) using different metrics.\nThey do not provide a single task that test these models against, using the same criteria, so in a sense, the comparison in not objective/very informative.\n\nAlso, the results that they present under no corrupted data do not match the ones in the corresponding literature \nFor example the test accuracy of CIFAR-100 was found to be significantly higher than the one reported here, at [1] and [2], namely 84.3% and 75.22% respectively\n\n\n[1] \"Text-to-Image Diffusion Models are Zero-Shot Classifiers\", Clark et al., NeurIPS 2023, and \n[2] \"Better Diffusion Models Further Improve Adversarial Training\", Wang et al., ICML 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lqDL3cAO1d", "reviewer": "Reviewer_gCmf", "summary": "This paper studies how different models behave when trained with low-quality (corrupted) data and proposes a three-part explanation with information-theoretic, PAC, and gradient-based perspectives. Empirically, the authors report: \n1. Autoregressive LMs remain strong even with heavy token/target corruption\n2. Class-conditional diffusion models collapse as label corruption grows (while FID stays roughly flat, the accuracy of conditional generation drops)\n3. Image classifiers are moderately sensitive on small datasets but surprisingly robust at ImageNet scale. \n\nThen they offer interpretations of these empirical observations based on distinct theoretical frameworks.", "questions": "See Weakness section.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "The empirical observation that GPT-2 and ImageNet-scale classification can tolerate, or even thrive under heavy label/target corruption is interesting and perhaps worth documenting.", "weaknesses": "Beyond that, the theoretical analysis/explanation they provide seems weak and superficial. Each of the three perspectives allows for straightforward counter-arguments, and the paper stops short of unifying them into a single, predictive theory. The writing is polished and persuasive on the surface, but the theoretical substance is thin; as a result, the paper takes rather longer to read than it should.\nBelow, I provide my view on each perspective the paper provides.\n\n1) Information-theoretic perspective\n\n- Residual information cannot explain “no degradation” at 50% noise. The relative information loss calculation under uniform noise decays with the number of classes but cannot go below $p_e$, which is its asymptotic limit. Hence, this argument cannot by itself explain why the full-ImageNet classifier exhibits no degradation (and even slight improvement) with ~50% incorrect labels.\n\n- The paper argues that the “absolute information content” drives robustness, but in their experiments (to my understanding) the total exposure to correct samples, i.e., absolute information, is held constant across noise conditions. If so, the framework again does not clarify why the full-ImageNet setting is uniquely robust while smaller-scale settings are not, given that the protocol equalizes clean-signal exposure.\n\n2) PAC perspective\n\n- They merely discuss a classical bound, not a targeted explanation. The appeal to $\\Omega(\\epsilon^{-1}\\log\\delta^{-1} + d\\epsilon^{-1} \\log \\epsilon^{-1})$ lower bounds is standard and doesn’t address the core issue in these experiments. Even when the number of correct samples seen is held fixed, additional corrupted samples still inject competing supervision that makes the signal harder to extract. This is simply ignored and the message of Subsection 4.2 reads as if “making the number of clean samples $m$ very large” is the operative solution. However, it does not clarify why $m$ being large should make the training robust to noisy samples.\n\n- The link from “conditioning richness” to effective VC dimension $d$ is hand-wavy. The diffusion example is particularly strained: conceptually, conditional diffusion model jointly learns (i) the data distribution and (ii) a class-conditional guidance. Even with label corruption, part (i) is achieved (as the authors report, the FID remains stable), and what fails is mainly the alignment between the conditional label and the type of the images generated. This is like an unconditional diffusion model equipped with inaccurately trained classifier guidance. Simply attributing this to the larger VC dimension compared to sparse conditions blurs the comprehensive picture. To properly support such a claim, one would need experiments that vary the number of classes in class-conditional generation or also consider text-to-image diffusion setting (richer conditioning) in parallel, and then compare those outcomes. Direct comparison of conditional diffusion model with image classifier does not seem appropriate to me.\n\n3) Gradient-based perspective\n\n- The authors previously mentioned that they use larger batch sizes to stabilize the optimization process under high-noise conditions. However, the reported standard deviations in Table 4 rather show the noisy model exhibiting lower loss variance than the clean baseline at comparable batch sizes. If the loss variance is intended as a proxy for gradient stability, this observation seems inconsistent with their comment on training instability in high-noise regimes. It is genuinely difficult to see what the authors intend to claim with this analysis.\n\n- Line 457 mentions that incorrect gradients are diverse/orthogonal and thus cancel, which might be true considering the statistics of Table 4. However, the paper does not directly test this (e.g., cosine similarity distributions between per-example gradients from clean vs. corrupted samples, magnitude of gradient sum from clean vs. corrupted samples).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O3vxX2o6BO", "reviewer": "Reviewer_rhZG", "summary": "This paper presents a systematic experimental analysis of the sensitivity of mainstream probabilistic models—specifically autoregressive, diffusion, and classifier models—to low-quality data. The authors identify two primary factors governing robustness: the \"richness of conditioning information\" and the \"absolute information content\" of the training data. The paper further provides theoretical grounding for these findings from the perspectives of information theory, PAC learning, and gradient dynamics.", "questions": "1. Following on from weakness #2, the paper's conclusions on robustness, especially the explanation rooted in gradient averaging (Sec 4.3), are derived from experiments using unstructured random noise. How well would these principles generalize to real-world, structured noise patterns where the \"noise\" gradients are coherent, not random, and thus would not average out?\n2. The experimental design in Sec. 3.1 attempts to isolate noise effects by fixing the “equivalent correct sample exposure,”but this approach changes both batch size and iteration count, introducing confounding factors. A control experiment with fixed batch size and varying noise ratio would better isolate the true impact of gradient averaging from training configuration effects.\n3. Section 3.3 argues for the effect of context richness by comparing the WMT 2014 translation task (sparse context) with the CNN/DailyMail summarization task (rich context). As these are fundamentally different tasks, it is difficult to attribute the observed difference in robustness solely to the richness of the conditioning information. A more convincing demonstration would involve comparing robustness within the same task.\n4. I am not an expert in this specific area. Based on the paper's clear writing and well-structured experiments, I am open to adjusting my score pending the feedback from reviewers with deeper expertise.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper provides a systematic and logically self-consistent experimental investigation that effectively demonstrates the varying tolerances of autoregressive models, diffusion models, and image classifiers to low-quality data.\n2. The two key factors proposed as governing robustness—the \"richness of conditioning information\" and the \"absolute information content\"—are insightful and provide a strong conceptual framework for understanding the observed phenomena.\n3. The explanatory analysis accompanying the experimental results is thorough and convincing, particularly the multi-perspective theoretical support that unifies the empirical findings.\n4. The paper's technical details are clearly presented, and the experimental setup appears to be highly reproducible.", "weaknesses": "1. The study's focus on training models \"from scratch\" (as noted in Sec 3.5) overlooks the dominant \"pre-training and fine-tuning\" paradigm used in modern practice. The robustness of a large model pre-trained on massive datasets may differ significantly when fine-tuned on low-quality data. This omission limits the direct applicability of the paper's findings to many real-world scenarios.\n2. The analysis relies almost exclusively on unstructured, random noise (as acknowledged in Appendix B). This is a significant limitation, as real-world data noise is often structured or correlated (e.g., systematic mislabeling). It would substantially strengthen the claims to include even a simple experiment with structured noise. For instance, in the classification task, could the authors investigate a scenario where noise is not uniformly random but fixed to specific incorrect labels (e.g., all instances are mislabeled as class 'B')?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)