Consistency Aware Robust Learning under Noisy Labels
Authors: Fahad Sarfraz
Bahram Zonooz
Elahe Arani
Venue: Accepted by TMLR ()

Abstract: Deep neural networks (DNNs) often struggle with noisy supervision, a common challenge in real-world datasets where high-quality annotations are scarce. While DNNs tend to memorize noisy labels, the human brain excels at learning in noisy environments by modulating sensitivity to errors based on their magnitude and consistency. Inspired by this, we propose Consistency-Aware Robust Learning (CARoL), which maintains a memory of past predictions and errors to quantify consistency and guide the learning process. CARoL employs a principled mechanism to distinguish clean from noisy samples and modulates rate of adaptation based on prediction consistency. Furthermore, it integrates multiple learning pathways to fully utilize the dataset, adapting to sample characteristics as training progresses. Our empirical evaluation shows that CARoL achieves high precision in noisy label detection, enhances robustness, and performs reliably under severe noise, highlighting the potential of biologically inspired approaches for robust learning.

Published in Transactions on Machine Learning Research (11/2025)
Consistency Aware Robust Learning under Noisy Labels
Fahad Sarfraz
JssarfraShhue.ul
Tom Tom, Netherlands
Eindhoven University of Technology (TU/e), Netherlands
Bahram Zonooz
b.zonooz@tue.nl
Findhoven University of Technology (TU/e), Netherlands
Elahe Arani"
c.aroni@tue.dt
Wayve Technologies Ltd, London, United Kingdom
Findhoven University of Technology (TU/e), Netherlands
Reviewed on OpenReview: https: // openreview. net/ forum? id- f2lp2ul
Abstract
Deep neural networks (DNNs) often struggle with noisy supervision, common challenge in real-world dalasets where high-quality annotations are scarce. While DNNs tend to memo- rize noisy labels, the human brain excels at learning in noisy environments by modulating sensitivity to errors based on their magnitude and consistency. Inspired by this, we propose Consistency-Awarr Robust Learning (CARoL), which maintains a memory of past predic- tions and errors to quantify consistency and guide the learning process. CARo. employs a principled mechanism to distinguish clean from noisy samples and modulates rate of adapta- tion based prediction consistency. Furthermore, it integrates multiple learning pathways to fully utilize the dataset, adapting to sample characteristics as training progresses. Our empirical evaluation shows that CARoL achieves high precision in noisy label detection, enhances robustness, and performs reliably under severe noise, highlighting the potential of biologically inspired approaches for robust learning.
1 Introduction
The performance of standard deep neural networks (DNNs) highly dependent, on the availability of large amounts of high-quality annotated data. However, maintaining such levels of quality the scale required for real-world applications is a time-intensive, resource-demanding, and costly activity. A more feasible approach is to leverage the abundant open-source web data and automate annotation using search queries and user descriptions (Makadia et al. 2008 Tsai & Hung 2008 to rely crowd-sourced datasets (Mozafari et al. 2014). However, these approaches inevitably lead to label noise, which significantly degrades the performance of DNNs (Xiao et al. 2015 Zhang et al.) 2016). Furthermore, DNNs are capable of memorizing even random labels (Arpit et al., 2017), adding further complexity. Given the ubiquity of noise real-world datasets, it is critical to develop methods that can learn efficiently with noisy supervision.
As the human brain constitutes efficient learning system that has evolved to learn under noise, we take closer look at the dynamics of its error-based learning to draw insights for the design of DNNs. The brain modulates the rate of adaptation to error based on several characteristics, including the magnitude of the error and the uncertainly about its predictions relative Lo the uncertainly of its observations (Castro el al. 2014
et al. 2010 Marko et al., 2012). Essentially, the brain learns more from small, consistent errors compared to large, abrupt errors, enabled by a principled mechanism that utilizes the history of past errors Herzfeld et al. 2014). Moreover, it is believed that small errors influence the
Criscimagna-Hemen
'Equal advisory role.
"odde available at

---

Published in Transactions on Machine Learning Research (11/2025)
process of learning in fundamentally different manner than large errors (Crieimaggaa-Hemminerr et al, 20T0). We believe that these salient features and learning mechanisms enable the brain to learn effectively under high levels of supervision noise and uncertainty in the environment. Hence, distilling these properties into DNNs can improve their robustness to label noise.
To this end, we first analyze how learning progresses under high degrees of label noise with standard train- ing. Specifically, we examine how predictions change for samples with clean versus noisy labels and define consistency as the fraction of predictions across training epochs that agree with the provided training label. Our analysis reveals that clean samples maintain high prediction consistency and alignment with the train- ing label throughout training, while noisy samples initially exhibit low consistency values, which gradually increase as the model begins memorizing the noisy labels. Interestingly, this memorization occurs earlier for noisy labels that are semantically similar to the true label. Furthermore, early predictions for noisy samples often align with their true labels. This observation suggests that if memorization can be avoided, the model's 's prodictions during the carly phase of training can serve as good surrogate for true labels, providing a strong incentive to bias learning lowards this phase. These findings provide systematic mechanism to quantily the phenomenon that DNNs learn simple patterns before memorizing noisy labels (Arpit et al. 2017). These insights underscore the importance of using instance-level statistics rather than population- thresholds to better differentiate between clean and noisy labels, guide the development of more effective noise-detection criteria, estimate noise transition matrices, and design robust training mechanisms.
Drawing insights from our empirical analysis and inspired by the learning dynamics of the brain, we propose. Consistency-Awary Robust Learning (CARoL), a framework designed to enhance the robustness of deep neural net- works under noisy supervision. CARoL main- tains a memory of past prodictions and er- rors during training, leveraging both consis- tency between predictions and labels, along with error magnitude, to guide the learn- ing process. It quantifies prediction consis- tency across training epochs for each sample and uses this information, along with error magnitude, to dynamically adjust the learn- ing process, distinguishing between clean and noisy samples. Based on this, we introduce a novel consisteney-aware error modulation loss, which adjusts the contribution of each sample to the training objective by weighting samples according to their consistency and their deviation from expected loss patterns. This ensures that reliable samples contribute more significantly to the model's optimization while noisy samples are downweighted. To further enhance robustness, CARoL integrates multiple learning pathways, leveraging a combination of supervised and auxiliary learning strategics to handle diverse samples effectively. For reliable samples, CARoL prioritizes supervised learning to focus on extracting clean patterns. For uncertain or noisy sam- ples, it employs auxiliary mechanisms that guide the model to learn useful representations without relying heavily noisy labels. Additionally, CARoL incorporates strategies to encourage diversity in training and maintain consistency in predictions over time, ensuring that learning remains robust across different phases of training. These hwayw are dynamically adapted based on the training phase, enabling the model to focus on generalizable patterns during early training and progressively refine its robustness to noise.
Standard Training
CARoL
100
Ceer
Nolay
Accuracy (%)
60
40
50
100
150 200
250 Epoch
300
350
0
50
100
150
200
250
300
350
Epoch
Figure 1: Training accuracy on clean and noisy samples on CIFAR-10 with 80% symmetric noise. standard train- ing memorizes the noisy labels, even at such severe noise levels, our method, (CARoL), effectively avoids memoriza- tion.
We empirically demonstrate the effectiveness of our approach across range of challenging noisy supervision scenarios. Our proposed selection criterion achieves high precision in detecting noisy labels and ensures balanced selection of samples across classes. Combined with the dynamic rate of adaptation to errors and the integration of multiple learning pathways, CARoL exhibits exceptional robustness to supervision noise
2

---

Published in Transactions on Machine Learning Research (11/2025)
and avoids memorization, even under severe label noise conditions (Figure 1 way for further exploration of biologically inspired mechanisms in robust learning under noisy supervision.
We believe our work paves the
Our key contributions are as follows:
We introduce Consistency-Award Robust Learning (CARoL), which leverages a memory of past predictions to define per-sample consistency and uses this as the guiding criterion for learning under noisy labels.
We proposc a class-adaptive thresholding mechanism that adjusts consistency thresholds per class relative to the global average, accounting for varying class difficulty and noise prevalence.
We design a biologically inspired Consistency-Aware Error Adaptation loss (Cca), which modulates sample weights based on error magnitude and consistency to suppress memorization while preserving informative signals.
We integrate multiple learning pathways supervised learning for consistent samples, SSL via FixMatch for inconsistent samples, and a novel consistency-aware MixUp-ta to effectively leverage all training data.
Through comprehensive experiments on CIFAR-10/100, Tiny-ImageNett and real-world noisy datasots (Web-Aircraft, Web-Bird, Web-Car), we demonstrate that CARoL effectively mitigates memorization and effectively learns under severe noise and in complex real-world scenarios.
2
Related Work
The different approaches to learning under noisy labels can be broadly categorized into three categories:
Estimate the noise transition matrix to relabel training data (Tanaka et al. 2018 Patrini et al. 2017
Hendrycks et al.
2018), improve the robustness of the loss function to label noise (Zhang & Sabuncu, 2018:
Ma et al. 2020 and separate noisy and clean samples to inform learning (Arazo el al. 2019 Ding el al.
2018 Han et al.
2018). Among these, methods that attempt to identify noisy samples and employ a semi-
supervised (Zhu 2005) learning approach have proven to be effective, particularly for high degrees of label
noise and a larger number of classes.
The vast majority of these methods rely on the low-loss criterion, whereby samples with low training loss are considered clean. To reduce confirmation bias, CoTeaching (Han et al.] 2018) trains two models alternately, each model selecting a predefined percentage of samples with the smallest loss as clean samples for the other model. M-Correction (Arazo et al.) 2019 and DMix (Li et al. (0200) models the loss distribution with a mixture model to split the training data into clean and noisy samples and train the two models a semi- supervised manner. MOIT (Ortego el al. 2021) combines semiisuperrised learning with contrastive learning. JPL (Kim et al., 2021) proposes negative learning for noisy labels. ELR (Liu al.] 2020) regularizes the update of the model that utilizes the carly phase of learning. UNICON (Karim et al. 2022) highlights the issue of class imbalance in the low loss selection criterion and proposes uniform selection mechanism. Importantly, the small-loss trick usually requires accurate estimate of noise level or biases learning toward easy samples, as harder samples are likely categorized as noisy. Instead of separating samples, NCT (Sarfraz et al. 2021 improves the robustness the learning framework by leveraging collaborative learning and larget variability. SOP (Liu al., 2022 models label noise as a sparse component and uses implicit regularization
divergence-basse
scparate it from clean data, improving robustness over- parameterized networks. Co-LDL (Sun et al.] 2021a) incorporates low loss sample selection strategy with label distribution learning. RML-Semi (Li et al. 2024 mitigates label noise by combining stable mean and robust median losses via regrouping strategy, improving sample selection and enabling a semi-supervised learning approach.
Closest to our approach, SELFIE (Song et al., 2019). also employs prediction consistency. However, they solely use the model's past predictions within time window to correct the noisy labels and continue employ the small-loss trick differentiate between clean and noisy samples, assuming the availability of the noise rate, which is not known priori (detailed comparison provided in Appendix). Similarly RoCL. (Zhou)
3

---

Published in Transactions on Machine Learning Research (11/2025)
et al. 2020 and CARoL share the intuition that accumulated signals are more reliable than instantaneous ones, their approaches differ fundamentally. RoCL, relies primarily on small-loss detection (applied to EMA of per-sample loss) and uses consistency only for pseudo-label generation (detailed comparison in Appendix). In comparison to these approaches, our study introduces prodiction consistency as a more sturdy and reliable indicator of label quality and dynamically adapts the contribution each sample based on their consistency value and error magnitude. We draw inspiration from the error-based learning dynamics of the brain guide our learning process.
3 Evolution of Predictions during Training
To gain a deeper understanding of the learning dynamics and prediction patterns of DNNs under high levels of label noise, we examine how model predictions evolve during training and differ for samples with clean and noisy labels. Specifically, we track the predictions of each sample at each epoch during training and measure consistency by evaluating the fraction of predictions that align with the provided training label. Figure 2 plots the consistency in predictions at different stages of learning for the PreAct-ResNette model He et al. (2016) trained with the standard cross-entropy loss CIFAR-10 with 50% symmetric label noise. During the carly stages of training, samples with clean labels (clean samples) exhibit significantly higher prediction consistency (skewed toward 1), whereas samples with noisy labels (noisy samples) show very low consistency (skewed toward 0), indicating poor aligument with the training labels. Although clean samples maintain high consistency throughout training, noisy samples progressively memorize the incorrect label during the middle phase, ultimately reaching a median consistency value of 0.5 by the end of training. These observations help characterize how predictions evolve during training and offer systematic way to quantify the phenomenon where DNNs learn simple patterns before memorizing noisy labels (Arpit et al. 2017).
For noisy samples, the model predictions are predominantly split. between the true and in- correct training labels. Figure 3 (a) illustrates the degree of alignment between the most pro- dicted class and both the training and true la-
50 Epochs
150 Epochs
300 Epochs
Samples
15000
#Samples
bels. Interestingly, during the initial epochs, the model often predicts the true label with high consistency, even for noisy samples. As training progresses, however, the model begins memorizing noisy labels, and the most pro- dicted class becomes increasingly aligned with
10000
5000
Noley Samples
15000-
them. This highlights the shift in prediction dynamics the model transitions from learn- ing general patterns to memorizing noisy data.
#Samples ls
10000
5000
We also observe that the onset of memoriza- tion for noisy samples varies for each instance depending on the semantic similarity between the true label and the noisy label. Figure 3 (b) shows that in cases where the noisy label is semantically similar to the true label (e.g., cat and dog, automobile and truck), memorization occurs carlier, as reflected by higher average consistency value. Early memorization results in a larger fraction of predictions aligning with the noisy label. In contrast, when the noisy label dis- similar to the true label (c.g. auto and cat), memorization occurs much later. These findings suggest that instanco-level statistics aggregated over training and adaptive classwise thresholds can be leveraged to design diguinggg criteria, noise transition matrix estimation, and robust training mechanisms.
0.0
0.5
1.00.0
0.5
1.00.0
0.5 Consistency
Consistency
Consistency
Figure 2: Distribution of the percentage of predictions that align with the training label for clean samples and noisy samples.
4

---

Published in Transactions on Machine Learning Research (11/2025)
(a)
(b)
Training Label
True Label
946
924
0114
Match Ratio (%)
80
Tuiniing Label
9.43
9.*
3.2
60
40
3.58
954
303
364
943
966
954
955
Lucke
20
LP0
1:42
612
Clcan Notsy
10/
200
300
100
plant
Epoch
Epoch
Truc Labi
Figure 3: (a) Percentage of samples in which the most frequently prodicted class aligns with the training label (left) the true label (right) during the training process. Aberageee consistency for each noisy training label and the true label pair at the end of training. Diagonal values (clean samples) are omitted for better visualization.
4
Methodology
We motivate our method using insights from human brain learning dynamics and our analysis of DNN prediction patterns, before detailing its components.
4.1 Overview
The design of CARoL motivated by the observation that the brain prioritizes learning from small, consistent errors over large, abrupt ones. Inspired by this principle, CARoL. maintains explicit memories of both model's past predictions and its classwise error statistics. Consistency is defined as the percentage of predictions that match the training label across the trajectory, while the error memory captures class- dependent loss dynamics. Together, these memories provide the basis for robust and informed sample weighting.
Our analysis reveals that clean samples maintain high consistency across training, while noisy samples exhibit low consistency early on but gradually increase as the model memorizes the noisy label. Importantly, early predictions for noisy samples often align with their true labels, and the speed of memorization depends on semantic similarity (e-g., "dog" mislabeled as "cat" memorizes earlier than "dog" mislabeled "car"). These insights suggest that early-phase predictions avoided.
valuable surrogate for ground truth if memorization is
CARoL operationalizes these findings through three mechanisms: (i) an EMA model biased Loward early- phase predictions to preserve reliable signals, (ii) class-adaptive thresholds to account for class-dependent noise patterns, and (iii) principled crror sonsitivity modulation scheme that uses prediction and error memories down-weight inconsistent samples based on their deviation from classwise error statistics. Fur- thermore, CARoL leverages all training data by combining supervised learning for consistent samples, SSL with EMA pseudo-labels for inconsistent samples, and a consistency-aware MixUp objective. This unified and principled design enables CARoL to effectively mitigate memorization while extracting useful information from both clean and noisy samples.
4.2 Biological Basis of Consistency as Learning Cue
Consistency plays vital role error-based learning the brain, enabling to reliably predict future events and understand cause-and-offect relationships to build an internal model of the world. The brain adjusts its sensitivity to errors based factors such error magnitude and the gap between prediction
5

---

Published in Transactions on Machine Learning Research (11/2025)
Eea
LCAM
Dog
Refined Labela(
LRD
Training Labels
EMA-Modd Predictions
SSs
Cat
Amm
1.0
Waighte:)
Corelittnnt
InConsistent
MM
Predictions Memory
Cat Home War
Cat
Hore
Dog
Consiateut(s > Tn)
Wolf
Figure 4: CARoL: consistency-awert robust learning method that incorporates a principled mechanism learn from consistent small errors and adapt learning based sample charactoristics. It uses exponentially moving averaged (EMA) model to accumulate knowledge from different stages of training and maintains a memory of past predictions. CARoL utilizes consistency score to form consistent set (blue colored samples) and inconsistent set (orange samples) as proxy for clean and noisy samples. Subsequently, the weight of inconsistent samples is reduced proportionally to their distance from the classwise error statistics. addition, uses combination of semi-supervised learning and consistency-aware MixUp loss to maximize learning from all samples.
and observation uncortainty (Castro et al. 2014 These factors closcly tied to consistency: small errors suggest that feedback aligus well with the brain's internal model, while lower prediction uncertainty reflects greater alignment of the stimulus with its prototypical representations. Consequently, the brain prioritizes learning from small, consistent errors over large, abrupt ones, adapting its learning process to focus on reliable patterns and filter out noisy or inconsistent information (Criscimagna- 2012
Bescimaagna-Hemmingge. Hemminger et al. 2010 Marko et al.
Additionally, evidence suggests that the brain moderates its learning from present errors via a system- atic mechanism that accounts for the history of previous errors, requiring the retention of an error mem- ory (Herzfeld et al.] 2014). Small errors also appear to influence learning in fundamentally distinct ways compared to large errors, as studies show that the neural basis of motor learning differs between these two cases (Criscimagna- Halscimagnaa-eemmnggor et al., 2010). We hypothesize that these mechanisms enable the brain to learn ellectively in noisy environments, and we aim integrate similar strategies into DNNs.
4.3 Consistency-Aware Robust Learning
Building analysis of prediction evolution and the brain's reliance consistency for learning, we hypothesize that instance-level consistency in predictions serve crucial cue for DNNs. It enables robust identification of label noise while also providing insights into sample's learning difficulty, its coherence with other samples in the same class, and the overall reliability of its label. Inspired by the brain's 's ability to regulate learning through error sensitivity, CARoL maintains a memory of past predictions and errors assess consistency as training progressos. To further refine adaptation, accounts for variations in class difficulty through dynamic classwise thresholds, ensuring systematic calibration both the instance and class levels. By leveraging both prediction history and classwise adjustments, CARoL modulates adaptation

---

Published in Transactions on Machine Learning Research (11/2025)
rates based prediction consistency and relative error magnitude, allowing the model to prioritize reliable patterns while mitigating memorization of noisy labels.
4.3.1 Components
CARoL trains working model, fe, batch of training samples X with labels y, which may include noisy labels. To capture evolving knowledge and mitigate confirmation bias, we maintain an exponentially moving averaged (EMA) model (Tarvainen & Valpola 2017 fg. updated using fa's weights. The EMA model evolves smoothly, preserving early-stage knowledge, and provides a stable representation of the data distribution. We use a stochastic update strategy (B.1p to prioritize carly-stage information, reducing the model's susceptibility memorization. This complementary perspective aids in regulating adaptation and decision-making throughout. training.
4.3.2 Consistency in Predictions
The variance in model predictions during training provides valuable insights into the quality of training labels and sample difficulty. To capture this, we maintain per-sample prediction memory that tracks how often the EMA model assigns each label. Let M E RN*K denote this memory, where N is the number of samples and K is the number of classes. The prediction consistency vector, c E RV, quantifies the proportion of EMA model predictions that match the training label y. For a sample with index i, the consistency score is given by:
Mass
G = ΣØ Ma
(1)
We use the consistency score to distinguish between noisy and clean samples. To account for variations in class difficulty, we adapt the threshold hyperparameter based on the class-wise average consistency, computed as the mean consistency of all samples in class k, and the global average consistency across all samples.
E Is = k)c
(2)
Ei=1
where ng is the number of samples class k. A sample with index i is assigned to the consistent set C if its consistency value is higher than the class-wise threshold:
C= C= Tee ViENJ.
(3)
Furthermore, we integrate prediction consistency and error magnitude to ensure a more balanced selection of samples in the consistent set across classes, dynamically adjusting selection to mitigate class imbalance (see B.2h.
4.3.3 Consistency-Aware Error Adaptation
To emulate the brain's 's ability adjust its sensitivity to errors, we track class- wisc prediction error statistics over training and leverage this information to dynamically weight. each sample in the supervised learning objective. Concretely, we maintain class-wise error memory, e R which is updated using exponential moving average of the EMA model's mean cross-entropy loss per class in each training batch.
We then apply consistency-awaal aware weighting scheme, assigning a high weight to samples in the consistent set C while scaling down others based on their deviation from mean class-wise loss statistics. For each sample with index i, the weight given by:
i otherwise
A
7- 7
(4)
min
1,
Loo(f
too
(xx;0))
7

---

Published in Transactions on Machine Learning Research (11/2025)
Algorithm Consistency-AAware Robust Learning (CARoL) Algorithm
Input:
Training datasel (D), learning rate (1) and hyperparaaneters: (T, 7)
Initialize: 0 0, M (IR = Zeros(R*)
Warm Up Stage: Train on CE loss and update prediction and error memory.
while Training do
Identify Consistent Samples: based on consistency in past predictions (Eq. 3
Consistency Aware Learning:
Sample batch: (X, Xx.X".9
Compute: 2 = [(X; 0), 2 ((x:0)
Evaluate weights À using consistency, memory, and (z, (Eq.
Combine losses: [ (1 CC 3LKD + LSst. LCAM
Update model: 0 < 0 v/aL
Update EMA: 0 + of +1 (1- a)0 rr u ~~ 10, (Eq.
Update memories: Update( M, arg max(2)), Update(e, (z, y)) (Eq. 9
end while Return: 0
where y is scaling factor and tee is the cross-entropy loss. Subsequently, our proposed consistency-aware loss is evaluated as the weighted sum of all sample losses:
CCA= Σ Ao loe (f(x;))))))
By integrating prediction consistency and error adaptation, this approach enables us to incorporate cues about the quality of supervision labels and bias the learning process towards small consistent errors, enhanc- ing the model's robustness to noisy labels and reduce memorization.
4.3.4 Gemi-Supervised Learning
Inspired by the brain's ability to process small, consistent errors differently from large, uncertain ones, we employ distinct learning mechanisms based on sample's consistency relative to past predictions. While samples with small, consistent errors are primarily learned using supervised loss, inconsistent samples (most likely noisy) are leveraged through semi-supervised learning. We adapt the FixMatch loss et al. 2020) by using the EMA modcl's prodictions with semi-supervised loss, LSsI. on the strongly augmented version of the same image. Additionally, to promote consistency across training and incorporate knowledge from the early learning phase, we employ structural knowledge distillation (Hinton et al. 2015). Specifically, we use KL-divergence loss, LKD. between the predictions of the working model and the EMA model. As the EMA model aggregates more knowledge from earlier phases of training, this biases learning toward stable patterns captured before the model starts memorizing noisy labels, acting both form of guidance and regularization.
weakly augmented images pscudo-labels to train the model
By integrating these mechanisms, we exploit the early phase of training, where the model is less susceptible to memorization, while simultaneously utilizing inconsistent samples to enhance representation learning.
4.3.5 Consistency-Aware MixUp
To leverage all samples while mitigating the impact of noisy labels, we employ a consistency-aware MixUp strategy. For samples in the consistent set C, use their given labels, while for the remaining samples, we assign pscudo-labels derived from the EMA model's predictions. Since the EMA model aggregates knowledge over training and is less prone to memorization, it often provides more reliable labels for noisy samples.
We then apply MixUp augmentation, where each sample interpolated with randomly selected sample from the batch, blending both their inputs and refined labels using mixing ratio sampled from a Beta distribution. This increases training diversity and enables supervised learning signal that is more robust to
8

---

Published in Transactions on Machine Learning Research (11/2025)
Table 1: Comparison with prior methods on CIFAR-10 and CIFAR-100 datasets with symmetric noise. Baseline results are obtained from the respective articles.
CIFAR- 10
CIFAR- 100
Method
20%
50%
80%
20%
50%
80%
CE
86.8
79.4
62.9
62.0
46.7
19.9
M-Up
95.6
87.1
71.6
76.8
57.2
30.8
PCI
92.4
89.1
77.5
69.4
57.5
31.1
JPL
93.5
90.2
35.7
70.9
67.7
17.8
MOIT
94.1
91.1
75.8
75.9
70.1
51.4
DMix
96.1
94.6
92.9
77.3
74.6
60.2
ELR+
95.8
94.8
93.3
77.6
73.6
60.8
UniCon
96.0
95.6
93.9
78.9
77.6
63.9
SOP+
96.3
95.5
94.0
78.8
75.9
63.3
RML-Semi
96.5
95.7
93.9
78.9
77.8
64.1
CARoL 96.3
95.9
93.8
79.7
77.9
66.0
Table 2: Comparison with prior methods on Tiny-ImageNet dataset with symmetric and asymmetric (pair flip) label noise. Baseline results obtained from respective articles and (Karmm et al. 2022
Noisc Type
Symmetric
Asymmetric
Noise (%)
20
50
45
Method
Best
Avg.
Best
Avg.
Best
Avg.
CE
35.8
35.6
19.8
19.6
26.3
26.2
Decoupling
37.0
36.3
22.8
22.6
26.6
26.1
MentorNet
45.7
45.5
35.8
35.5
26.6
26.2
Co-tcaching+
48.2
47.7
41.8
41.2
26.9
26.5
M-correction
57.2
56.6
51.6
51.3
24.8
24.1
NCT
58.0
57.2
47.8
47.4
43.0
42.4
UNICON
59.2
58.4
52.7
52.4
CARoL
61.0
60.8
55.2
54.9
47.3
45.2
label noise while reducing the risk of memorization. The resulting samples are trained using the consistency- aware MixUp loss, LCAM, which further improves model robustness under noisy supervision.
4.3.6 Overall Loss
The overall training objective is given by:
== (1 + 3LKD + LSSt. + LCAM
(6)
where 3 controls the relative contribution of the consistency-aware loss and the distillation loss. Following prior work (Sarfraz et al. 2021). we progressively increase 3 from 0 to 0.95 during training using a sigmoid ramp-up Junction, gradually shifting the focus towards distillation. The overall formulation of CARoL is outlined in Algorithm
Additional implementation details provided in the Appendix. For inference, model fer foregoing the commonly used ensemble approach. This ensures computational elficiency while maintaining strong performance, making our method well-suited for real-world deployment.
only the working
5 Experimental Setup
We evaluate our method across diverse simulated and real-world noisy datasets with varying noise lev- els. Following prior works (Karmm et. al.] 2022 Li et al. 2020 introduce symmetric noise on CIFAR-10,

---

Published in Transactions on Machine Learning Research (11/2025)
Table 3: Comparison with SOTA approaches in test accuracy (%) on real-world noisy datasets: Web-Aircraft
Web-Bird, War-Car. Bascline results are from (Sun et al. 2021a
Methods 60.80
Web-Bird
Web-Car
Average
CE
64.40
60.60
61.93
Decoupling
75.91
71.61
79.41
75.64
Co-teaching+
74.80
70.12
76.77
73.90
PCIL
78.82
75.09
81.68
78.53
DMix
82.48
74.40
84.27
80.38
Sell-adaptive
77.92
78.49
78.19
78.20
Co-LDL
81.97
80.11
86.95
83.01
CARoL
88.21
78.10
88.58
84.96
Table 4: Effect of the different components of CARoL performance CIFAR-10 with multiple levels of symmetric label noise.
50%
80%
LKD
LCA
LSSL
LCAM
BEST
LAST
BEST
LAST
X
X
X
82.5
59.1
66.0
28.1 1
V
X
X
X
87.8
60.4
71.2
28.3
X
J
X
X
88.6
77.2
73.9
44.8
J
X
90.7
88.2
77.4
52.9
/
92.1
92.5
84.5
84.1
V
X
95.3
95.1
92.8
92.6
V
V
V
V
95.9
95.5
93.8
93.5
CIFAR-100, and Tinylmagen ImageNet by randomly replacing labels with uniformly sampled incorrect labels. Addi- tionally, we model asymmetric noise on TinylmageNet (Sarfraz et al., 2021 Yu al.) 2019) using the pair-flip strategy, which simulates annotation errors where visually similar classes confused. To assess real-world applicability, evaluate three fine-grained datasets: Web-Aireraft Web-Bird, and Web-Car (Sun et al. 2021b). These datasets contain 13,503, 18,388, and 21,448 noisy training images across 100, 200, and 196 categories, respectively, with clean test sets for evaluation. The training images are curated from web search engines, which contains arbitrary levels noisc
while Table [6 shows that performance remains stable across various parameter choices. Notably, CARoL's 's hyper- parameters exhibit a syncrgistic relationship, allowing most parameters to remain fixed, thereby simplifying hyperparameter optimization.
Training details are provided Appendix C The selected hyperparameters are listed Table
6 Empirical Evaluation
We evaluate CARoL across multiple benchmarks under uniform experimental settings, comparing it with state-of-the-ar (SOTA) methods under varying noise levels and types. Table [ demonstrates that CARoL achieves either superior or competitive performance across different levels of symmetric label noise on CIFAR- and CIFAR-100. Notably, CARoL shows substantial gains the most challenging sctting- CIFAR-100 with 80% noise which represents severe levels of label noise on a complex dataset with a higher number of semantically similar classes. Table 2 further highlights CARoL's effectiveness on TinylmageNet, more complex dataset with 200 classes, where it significantly outperforms prior methods under both symmetric and asymmetric noise conditions.
Crucially, the performance gains become more pronounced as dataset complexity increases, demonstrating CARoL's robustncss label noise scenarios with high inter-class similaritics. These improvements are particularly notable given that CARoL achieves them with single model and single-model inference, unlike many strong baselines that rely concurrent. training of multiple models and ensemble inference. For
10

---

Published in Transactions on Machine Learning Research (11/2025)
After Warmup
200 Epochs
# Consistent Samples
2500
2000
1500
1000
500
Clean
Nolsy
0
plane auto
bird
cat
deer
dog
frog
horse
ship
truck
plane
auto
bird
cat
deer
dog
frog
horse ship
truck
Figure 5: The distribution clean and noisy labels among the samples from each class that are considered consistent, measured after the warm-up stage (30 epochs) and after 200 epochs. Furthermore, Figure in Appendix shows that CARoL effectively identifies the vast majority of noisy samples as inconsistent, reducing false negatives.
instance, the one-model variant ELR (ELR Table (Liu et al., 2020g) only achieves 80.7% on CIFAR- 10 (80% noise) and 30.3% on CIFAR-100 (80% noisc), significantly lower than CARoL's performance.
To further assess CARoL's 's applicability, evaluate it real-world noisy datasets (Table 3) CARoL demonstrates strong performance compared to prior works, providing significant gains in Web- Reb-rAirraftt and Web-Car while remaining competitive on Web-Bird. While real-world datasets introduce additional chal- lenges due to the non-unite uniform nature of label noise, CARoL's 's consistency-awaat learning mechanism enables it to remain robust across different noise structures.
Overall, CARoL's consistency- adaptation allows it effectively utilize all training samples without rigidly separating clean and noisy instances, making it highly robust across various noise conditions. Unlike methods that rely explicit noise assumptions strong dean/noisy sample separation, CARoL dynamically modulates learning signals, ensuring scalability and real-world applicability.
7 Ablation Study
To gain insights into the learning mechanisms in CARoL, conduct stepwise ablation, incrementally adding each objective function to the standard cross-entropy (CE) loss and evaluating its impact model performance. Table g shows that each component contributes positively, with their effect. becoming more pronounced at higher noise levels. In particular, the Aonsistency-Aware Error Adaptation loss (Eca) signifi- cantly mitigates memorization, underscoring its ability to identify noisy samples and suppress memorization by down-weighting inconsistent samples. The benefit further amplified when combined with LKD, showing the complementary role of consistency-based modulation and EMA-guided regularization.
The Consistency-Awant Error Adaptation loss significantly improves the model's ability to extract meaning- ful supervision from noisy labels while mitigating memorization. This is complemented by semi-supervised learning (SSL) and Consistency-Aware MixUp, both of which enable the model to leverage all training samples effectively. The SSL loss utilizes the EMA model to learn from confident predictions on the incon- sistent samples, improving overall performance. Notably, Consistency- Aware MixUp refines target labels by leveraging the EMA model's predictions, facilitating smoother interpolation between samples and further enhancing generalization. Overall, our ablation study demonstrates the benefits of dynamically adapting learning objectives based prediction consistency and error magnitude.
11

---

Published in Transactions on Machine Learning Research (11/2025)
8 Quality of Selection Criterion
One of the key challenges in learning under noisy labels is accurately distinguishing between clean and noisy samples to guide the learning process effectively. Therefore, we evaluate the effectiveness of our selection criterion in identifying clean samples. Figure plots the number of clean and noisy samples per class that are classified as consistent (i E () using our approach when training on CIFAR-10 with 50% noise. Even in the early training stages (after the warm-up phase), our method correctly identifies the majority of clean samples while minimizing false positives.
Furthermore, our class-wise consistency threshold accounts for varying class difficulty, leading to more balanced selection across classes. Unlike the small-loss criterion Karim et al. (022), which can exacerbate class imbalance by favoring easy-to-learn classes, our selection criterion ensures a more uniform distribution of selected samples across all classes. These results support our hypothesis that tracking prediction consistency over training provides a reliable signal for assessing supervision quality, offering a principled alternative to loss-based sample selection.
9 Conclusion
Our study analyzed the evolution of the predictions during training and compared the consistency of the predictions for clean and noisy samples to provide new insights into the behavior of the model under label noise. We then utilized these findings and insights from the brain to design novel method, Consistency- Aware Robust Learning, which provides a principled approach for utilizing the evolution of instance-level consistency in predictions of samples and error magnitude to estimate the quality of supervision and ac- cordingly employs different learning mechanisms for elficient learning under high degrees of label noise. The empirical evaluation demonstrated the effectiveness of our approach on various datasets with different types and degrocs of label noise and showed that improves precision detecting noisy labels and enables the model to learn effectively even under severe levels of label noise.
10 Broader Impact Statement
Learning under label noise is critical for real-world applications, where large-scale datasets are often con- structed through automated labeling, inevitably introducing annotation errors. Effectively handling such noise enables the utilization of vast, imperfect data sources, reducing dependence costly manual annota- tions and improving the scalability of DNNs. CARoL demonsttates the potential of incorporating biologically inspired mechanisms for robust learning by incorporating principles of error-based learning, where the brain dynamically modulates sensitivity to errors based on their magnitude and consistency. Our study makes a compelling case for integrating neuroscience-inspiree strategics into DNN design and learning mechanisms, encouraging further exploration of biologically plausible mechanisms to enhance the reliability and general- ization of deep neural networks in real-world scenarios.
12

---

Published in Transactions on Machine Learning Research (11/2025)
References
Eric Arazo, Diego Ortego, Paul Albert, Noel O'Connor, and Kevin McGuinness. Unsupervised label noise modeling and loss correction. In International conference machine learning, pp. 312 321. PMLR, 2019.
Devansh Arpit, Stanislaw Jastrzębski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kan- wal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization deep networks. International conference on machine learning, pp. 233-242. PMLR, 2017.
Luis Nicolas Gonzalez Castro, Alkis M Hadjiosif, Matthew A Hemphill, and Maurice A Smith. Environmental consistency determines the rate of motor adaptation. Current Biology, 24(10):1050 1061, 2014.
Sarah Criscimagna-Hemmiigg Amy J Bastian, and Reza Shadmchr. Size of crror affects cerebellar contributions to motor learning. Journal of neurophysiology, 103(4):2275-22)) 2284, 2010.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation policies from data. arXic preprint Xiv: nal:1805.09001 2018.
Yifan Ding, Liqiang Wang, Deliang Fan, and Boqing Gong. A semi-supervised two-stage approach to learning from noisy labels. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1215-1224. IEEE, 2018.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in neural information processing systems, 31, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645. Springer, 2016.
Dan Hendrycks, Mantas Mazeika, Dunean Wilson, and Kevin Gimpel. Using trusted data to train deep networks labels corrupted by severe noise. Advances in neural information processing systems, 31, 2018.
David J Herzfeld, Pavan A Vaswani, Mollie K Marko, and Reza Shadmehr. A memory of errors in sensori- motor learning. Science, 345(6222))134991188 1353, 2014.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge a neural network. arXie preprint arXiv:1503.0255 2015.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In International conference on machine learning, pp. 2304-2313. PMLR, 2018
Nazmul Karim, Mamshad Nayeem Rizve, Nazanin Rahnavard, Ajmal Mian, and Mubarak Shah. Unicon: Combating label noise through uniform selection and contrastive learning. In Proceedings of the IEEE/CVP Conference Computer Vision and Pattern Recogrition,, pp. 9676 9686, 2022.
Youngdong Kim, Juseung Yun, Hyounguk Shon, and Junmo Kim. Joint negative and positive learning for noisy labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9442 22--9451,44 2021.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. arXiv:1610.0224 2016.
arXio preprint
Fengpeng Li, Kemou Li, Jinyu Tian, and Jiantao Zhou. Regroup median loss for combating label noise, In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 18474-13482, 2024.
Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. Xiv preprint arXi:20022.0799, 2020.
13

---

Published in Transactions on Machine Learning Research (11/2025)
Sheng Liu, Jonathan Niles- Weed, Narges Razavian, and Carlos Fernandez- Early-learning regular- isation prevents memorization of noisy labels. Advances in neural information processing systems, 33: 20031220442, 2020.
Sheng Liu, Zhihui Zhu, Qing Qu, and Chong You. Robust training under label noise by over-parameterization International Conference on Machine Learning, pp. 14153 14172. PMLR, 2022.
Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah Erfani, and James Bailey. Normalized loss functions for deep learning with noisy labels. In International conference machine learning, pp. 6543-655x PMLR, 2020.
Ameesh Makadia, Vladimir Pavlovic, and Sanjiv Kumar. A new baseline for image annotation. In European conference on computer vision, pp. 316-329. Springer, 2008.
Eran Malach and Shai Shalev-Shwartz, Decoupling' when update' from' how to update'. In Advances in Newrul Information Processing Systems, pp. 960 9700 2017.
Mollie K Marko, Adrian M Haith, Michelle D Harran, and Reza Shadmehr. Sensitivity to prediction error in reach adaptation. Journal of neurophysiology, 108(6):1752-1763, 2012.
Barzan Mozafari, Purna Sarkar, Michael Franklin, Michael Jordan, and Samuel Madden. Scaling up crowd- sourcing to very large datasets: a case for active learning. Proceedings of the VLDB Endowment, 8(2): 125-136, 2014.
Diego Ortego, Eric Arazo, Paul Albert, Noel E O'Connor, and Kevin McGuinness. Multi-objective interpo- lation training for robustness to label noise. In Proceedings of the IEEE/CVF Conference Computer Vision and Pallern Recognilion, pp. 6606 6606-6615, 2021.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1944 1952, 2017.
Fahad Sarfraz, Elahe Arani, and Bahram Zonooz. Noisy concurrent training for efficient learning under label noise. In Proceedings of the (GE//CVF Winter Conference Applications of Computer Vision, pp. 3159-3168, 2021.
Mengmeng Sheng, Zeren Sun, Tao Chen, Shuchao Pang, Yucheng Wang, and Yazhou Yao. Foster adaptivity and balance in learning with noisy labels. In European Conference on Computer Vision, pp. 217-235. Springer, 2024.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with con- sistency and confidence. Advances in neural information processing systems, 33:596 608, 2020.
Hwanjun Song, Minseok Kim, and Jae-Gil Lee. Selfie: Refurbishing unclean samples for robust deep learning. International Conference on Machine Learning, pp. 5907-5909 5915. PMLR, 2019.
Zeren Sun, Huafeng Liu, Qiong Wang, Tianfei Zhou, Qi Wu, and Zhenmin Tang. Co-ldl: A co-training- based label distribution learning method for tackling label noise. IEEE Transactions on Multimedia, 24: 1093-1104, 2021a.
Zeren Sun, Yazhou Yao, Xiu-Shen Wei, Yongshun Zhang, Fumin Shen, Jianxin Wu, Jian Zhang, and Heng Tao Shen. Webly supervised fine-graines recognition: Benchmark datasets and an approach. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10602-10611, 2021b.
Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization framework for learning with noisy labels. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5552 35605 2018.
14

---

Published in Transactions on Machine Learning Research (11/2025)
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in newral informalion processing systems. 30, 2017.
Chih-Fong Tsai and Chihli Hung. Automatically annotating images with keywords: A review of image annotation systems. Recent Patents Computer Science, 1(1):55-68, 2008.
Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification, In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2691-2699 2015.
Kun Yi and Jianxin Wu. Probabilistic end-to-end noise correction for learning with noisy labels. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7017-7025, 2019.
Xingrui Yu, Bo Han, Jiangchao Yao, Gang NIu, Tvor W Tsang, and Masashi Sugiyama. How does disagree- ment help generalization against label corruption? arXiv preprint arXiv:1901. 19X.::1901.04115, 2019.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint Xi::
2016.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems, 31, 2018.
Tianyi Zhou, Shengjic Wang, and Jeff Bilmes. Robust curriculum learning: from clean label detection to noisy label self-correction. In International conference learning representations, 2020.
Xiaojin Jerry Zhu. Semiisuperrvsedd learning literabure survey. 2005.
15

---

Published in Transactions on Machine Learning Research (11/2025)
A
Appendix
B Implementation Details
Here, we provide additional implementation details for CARoL.
B.1 EMA Model Update
To accumulate knowledge across the different stages of training and avoid confirmation bias, CARoL maintain
an additional exponentially moving averaged (EMA) model Tarvainen & Valpola 2017), fg. which averages the model's weights during training.
i +- 0, if rp>>>> u
where is the decay parameter which is set to 0.99. CARoL employs stochastic update strategy for the EMA model, whereby instead of updating the weights of the EMA model, 0, at each iteration with the weights of the learning model, 0, we sample a variable u from a uniform distribution (between 0 and 1) and update the EMA model if u is less than the update frequency parameter rF. Additionally, to enforce the EMA model to aggregate more information from the early phase of training, where the model is less prone to memorization, we use a cosine ramp-down function for the update frequency:
Tmin + (Tmaz - Tminn ) cos( C.
(8)
E
where Tmar = 1, Tmin 0.05, e is the current epoch and E the total number of epochs. This ensures that the update frequency of the EMA model is progressively reduced during training (Figure left).
B.2 Uniform Selection Across Classes
To encourage more uniform selection of samples in the consistent set across classes, we take an average of the number of samples for each class in C and increase the number of samples for underrepresented classes in the consistent set. For this purpose,
assign score to each unselected sample (with index i) of the under-represented class ((1 c) 0)) and select samples with low scores so that the number of samples for the class is equal to the average number of samples. This provides a principled way to use both prediction consistency and error magnitude to determine the quality of supervision.
B.3 Error Memory
CARoL maintains an average error memory for each class, e E R by computing the exponential moving average of the E.A--odel's average cross-entropy loss during the training trajectory:
1 Σ (1() k) lad (f(x:0))))
e + ex + (1 (1-6h a) The
(9)
where a is the decay parameter set to 0.99.
To maintain the stability of error memory and prevent the statistic from being skewed toward large errors from noisy labels, we calculate the per-class average, Pk, and standard deviation, ak. of the loss values of the class samples in the batch and update e₁ with errors values less than p.ook) where p hyperparameter.
B.4 Dynamic Consistency Threshold
To account for the increase in consistency values training progresses and the model makes more predictions, also employ a dynamic consistency threshold using sigmoid ramp-up function, similar to [Sarfraz el al.
2021 Laine & Aila 2016) (Figure right):
w(e) e- -
=)²)
(10)
Hence, effectively at each epoch, the consistency threshold is given by w(e).
16

---

Published in Transactions on Machine Learning Research (11/2025)
1.0
1.0
0.8
0.9
0.6
W (
0.8
0.4
0.7
0.2
0.6
0.5
100
150
200
250
300
350
100
150
200
250
300
350
Epochs
Epochs
Figure 6: Ramp functions used for dynamic learning.
Table 5: Selected hyperparameters for the different settings.
DATASET
NOISE (%)
SYMM-20
0.3
3.0
2.5
CIFAR-10
SYMM-50
0.3
3.0
2.5
SYMM-80
0.3
0.75
0.1
SYMM-20
3.5
1.0
CIFAR-100
SYMM-50
0.1
1.5
0.3
SYMM-80
0.5
0.05
SYMM-20
0.3
1.0
1.0
TINY-MMGENNET
SYMM-50
0.3
0.5
0.1
ASsYM-45
0.6
0.1
0.1
WEB-A AIRCRAFT
0.1
1.0
2.5
WED-BIRD
1.0
2.5
WEB-CAR
0.1
1.0
2.5
B.5 Dynamic Weighting
We employ similar strategy to gradually shift the focus of learning from cross-entropy loss to knowledge distillation loss by increasing the 3 value using the Sigmoid function mentioned above. At each epoch, we use B = Smax w(e where Bmax = 0.95.
B.6 FixMatch
To further learn from samples that are considcred inconsistent, we adapt the FixMatch loss (Sohn et al. 2020) such that the EMA-model'ss predictions on the weakly augmented image, p = arg maxx(Fa( with confidencee score s, act as the pseudo labels. We then enforce consistency in predictions, by training the model to predict the same label as the EMA model when fed strongly augmented version of the same image X":
LSSl Cl -
(11)
where Ta is the confidence threshold set 0.6. This enables us to utilize the inconsistent samples effectively for learning.
17

---

Published in Transactions on Machine Learning Research (11/2025)
Table 6: Effect of different hyperparameters on the performance the model trained on CIEAR10 with 50% Symmetric Noise.
p
7
Ace (%)
1.0
95.08
0.1
2.0
95.32
3.0
95.29
1.0
94.67
1.0
0.3
2.0
95.51
3.0
95.51
94.71
0.5
2.0
95.78
3.0
95.64
1.0
15
0.1
2.0
95.17
3.0
95.10
1.0
95.55
2.5
0.3
2.0
95.56
3.0
95.88
94.97
0.5
2.0
95.69
3.0
95.83
C Training Details
For fair comparison, we experimental setup similar to previous methods Li et al. 2020 Karim ct al. 2022). Unless otherwise stated, we use PreActResNet-115 (He et al. 2016), SGD Optimizer with 0.9 momentum, 5e-4 weight decay, and an initial Ir rate of 0.02 with cosine annealing Ir scheduler, 64 batch size, and apply random crop and horizontal flip as weak augmentation. For CIFAR-10 and CIFAR-100 datasets, we Irain for 350 epochs and use CIFARIO AuloAugment (Cubuk et al. 2018) policy as strong augmentation. For Siny-ImageNNe
we train for 150 epochs and use the ImageNet policy as a strong augmentation. We use warm-up period of 30 epochs for all of our experiments and start tracking the predictions and loss values from Epoch 5. Following previous works (Li et al., 2020 Karim et al. 2022 we report the best accuracy for CIFAR10 and CIFAR100 datasets (the ablation study shows that there is minor difference bet the best performance and the last accuracy for CAROL; hence, it effectively mitigates forgetting) and for TinylmageNet, following (Han et al. 2018 Karim et al. 2022), we report the best and average accuracy for the last epochs. For Web datasets, we follow the settings in (Sheng el. al.) 2024) and use pre-trained ResNet-50 model with a learning rate of 0.005, batch size 32, and use SGD optimizer with weight decay 50-4 and cosine annealing scheduler. We train the model for 250 epochs.
C.1 Hyperparamter Tuning
A small clean validation set (5% of the training data) is used to finetune the hyperparameters with the best accuracy the metric of selection. Tuning CARoL primarily involves tuning the threshold parameter 7 e {0.1, 0.3, 0.6, 0.8), the scaling parameter y E (0.5, 0.75, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5} and loss filtering p E {0.05, 0.1, 0.3, 1.0, 2.5]. The parameters selected for each setting are provided in Table 5 For all experiments, we use T,=0.6, except for CIFAR- 100 80% where we set it to 0.2 to take into account the low confidence of the model when learning under such a high degree of noise and dataset complexity (with r=0.6, achieves 93.61%)
C.2 Sensitivity to Hyperparameters
Table further shows that our method is not highly sensitive a specific choice of hyperparameters and performs consistently across wide range of parameters. Importantly, the parameters exhibit synergistic
18

---

Published in Transactions on Machine Learning Research (11/2025)
After Warmup
200 Epochs
Clean
2500
Noley
InConsistent Samples
2000
1000
500
plane
bird
cat
deer
dog
frog
horse
ship
truck
plane
auto
bird
deer
frog
horse
truck
Figure 7: The distribution of clean and noisy labels among the samples from each class that are considered inconsistent. CARoL effectively distinguishes the vast majority of noisy samples inconsistent.
relationship, enabling us to keep certain parameters constant while tuning others, which greatly simplifies the process of hyperparameter optimization.
D Baselines Considered
Depending on the availability of results in similar settings, we compare CARoL with the following state-of-
the-art methods: M-Up (Zhang et al., 2018), PCTL (Yi & Wu 2019). JPL. (Kim et al. 2021 MOIT (Or-
tego et 2021), DMix (Li al. 2020), ELR+ (Liu et al., 2020), UniCon (Karim et al. 2022). Decou-
pling (Malach & Shalev-Shwartz 2017). MentorNet (Jiang et al. 2018 Co-teaching+ (Yu et al.
2019
M-correction (Arazo et al. 2019
NCT (Sarfraz et al. 2021 .SOP
Liu et al. 2022), RML-Semi LLi et al.
2024) SED (Sheng et al. 2024
E
Distribution of Consistent and Inconsistent Samples
we examine how CARoL partitions training samples into consistent and inconsistent sets when training CIFAR-10 with 50% label noise. Figures 5 and iillustrate the baasswiise distribution of clean and noisy samples in both sets, measured after the warm-up stage and after 200 epochs.
To further expand upon our analysis in Section g
As shown in Figure 5 the vast majority of clean samples are classified as consistent throughout training, with only a small fraction noisy samples entering this set. In contrast, Figure 7 shows that most noisy samples are reliably identified as inconsistent even in the early phase, and this separation is further sharpened training progresses. This demonstrates that CARoL not only minimizes false negatives (noisy samples mistakenly treated consistent) but also preserves high recall of clean samples.
Together, these results confirm the efficacy of our selection strategy: by leveraging prediction consistency and class-aware thresholds, CARoL achieves a robust scparation of clean and noisy samples across classes. This principled partitioning prevents memorization of noisy labels and ensures that learning signals are drawn primarily from reliable samples, thereby mitigating forgetting throughout training.
F Comparison with SELFIE
While SELFIE (ong et al. 2019) also uses the consistency of previous predictions, there are several fun- damental differenccs. It uses consistency correct the labels of the samples while still using the small-loss trick to distinguish clean and noisy samples (assuming access to the noise rate, which is a priori, and inferring it precisely using vols-validotion infeasible, as the range of possible values is huge).
19

---

Published in Transactions on Machine Learning Research (11/2025)
In contrast, study presents consistency in predictions more robust and reliable cue to identify clean and noisy samples, providing additional advantages, including estimating the transition matrix, correcting the labels, being more robust to different kinds of noise, etc. Therefore, our method uses consistent predictions to segregate clean and noisy samples instead of using the small-loss criterion.
Our main contributions are as follows: (1) A principled approach for mimicking error sensitivity modulation in the brain, whereby the model uses the history of predictions and loss values modulate the per-sample sensitivity that it learns more from small, consistent errors compared to abrupt, large errors. (2) Based on the characteristics of the sample, different learning mechanisms are utilized, iee, higher weight for supervised learning of consistent samples and consistency regularization using SSL on inconsistent samples. (3) Dynamic shift in learning from supervised to SSL as training progresses; and (4) CA-MixUp loss that utilizes label
refinement to correct the labels of inconsistent samples.
Our approach shows considerably higher performance: SELFIE achieves 86.5% accuracy on 40% symmetric noise (highest considered) on CIFARIO, which is considerably lower than CARoL even on double the noise (93.8% on 80% noise).
G Comparison with Robust Curriculum Learning (RoCL)
RoCL (Zhou al. 2020 focuses on the unreliability of instantancous loss, which exhibits high variance across epochs and leads to poor clean-label detection under severe noise. To reduce this variance, RoCL applies the small-loss criterion not on instantaneous loss but on EMA-smoothed losses, retaining small-loss filtering as its primary mechanism. Consistency in RoCL is defined as the discrepancy between the current modol's predictions and those of an EMA model across multiple augmentations, and it is used only to identify samples with reliable pseudo-labels during the sell-supervised phase. CARoL, by contrast, makes consistency the core learning signal. maintains explicit memories of past predictions and classwise error statistics, defines consistency as the fraction of predictions matching the training label across epochs, and leverages this for principled error modulation, adaptive thresholds, and multiple learning pathways. This allows CARoL to utilize all training samples effectively, rather than relying primarily on small-loss filtering.
CARoL demonstrates significantly better performance compared to RoCL, showcasing the effectiveness of the selection criterion and learning mechanisms employed in CARoL. For instance on CIFAR10 and CIFAR100 with 80% label noise, RoCT, achieves 85.76% and 53.89% respectively compared to CARoL's significantly higher performance 93.8% and 66.0%.
In summary, while RoCL and CARoL are motivated by related intuitions, approach introduces prin- cipled redefinition of consistency, memory-based mechanism for error sensitivity modulation, and multiple learning pathways that allow all samples to contribute effectively. These innovations lead to both new in- sights into training dynamics under noisy labels and significant empirical improvements, particularly under severe noise and on challenging datasets.
20


Reviews:
[{"review_id": "MktzovdVkN", "reviewer": "Reviewer_ZnWP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces Consistency-Aware Robust Learning (CARoL), a novel framework inspired by human brain error-based learning to enhance deep neural network robustness under noisy labels. By maintaining a memory of past predictions and errors, CARoL computes per-sample consistency scores and dynamically adjusts learning using a consistency-aware loss, semi-supervised learning, MixUp, and knowledge distillation. Empirical evaluations on CIFAR-10/100, Tiny-ImageNet, and real-world noisy datasets (Web-Aircraft, Web-Bird, Web-Car) demonstrate CARoL’s superior performance.", "claims_and_evidence_yn": "No", "claims_explanation": "1. The authors state that “the brain prioritizes learning from small, consistent errors over large, abrupt ones,” motivating their approach to assign a high weight to samples in the consistent set in Eq. 4. However, the weighting scheme may lead to unintended outcomes. Specifically, for inconsistent samples ($i \\notin C$ ), if the ratio $\\frac{e_{y_i}}{\\ell_{ce}}$  is sufficiently large, the weight for inconsistent samples could exceed 1, surpassing that of consistent samples. This contradicts the stated biological inspiration, as it may overemphasize noisy samples rather than prioritizing consistent ones. Additionally, in Eq. 4, why do the authors use the EMA model’s loss rather than the original model’s loss?\n\n2. Equation 2 defines the class-adaptive threshold as $\\tau_k = \\frac{\\frac{\\sum_{i=1}^N \\mathbb{I}(y_i = k) c_i}{n_k}}{\\frac{\\sum_{i=1}^N c_i}{N}} \\cdot \\tau$, where $\\frac{\\sum_{i=1}^N c_i}{N}$  is the global average consistency across all samples. Why is the global consistency average included in the denominator?", "audience_yn": "Yes", "audience_explanation": "This paper proposes a method for addressing the noisy label learning problem and offers very interesting insights. Moreover, the paper is well-written and easy to follow.", "requested_changes": "How is the consistency metric computed in Section 3?\n\nThe notation in Figure 4, such as $\\mathcal{M}_p$  and $\\mathcal{M}_e$ , is inconsistent with the main text's $M$  and $e$ . Additionally, blue and orange are used for clean and noisy samples without a clear justification. Adding an explicit explanation of the color scheme would improve clarity and reader understanding.\n\nIn Figure 5, the Warmup stage is referenced as the initial phase for evaluating the consistency-based selection criterion, but it is unclear whether this refers to a single epoch or multiple epochs, potentially confusing readers.  Additionally, Figure 5 would benefit from including the distribution of clean and noisy labels among samples identified as inconsistent for each class. This visualization could provide deeper insights into the selection criteria's behavior across classes.\n\nTable 4 should include an experiment with only $L_{CA}$ to validate its core role in CARoL’s motivation for robust noisy label learning.", "broader_impact_concerns": "There are no additional ethical concerns identified that require discussion.", "additional_comments": "N/A", "extra_scores": {}, "extra_text": {}}, {"review_id": "2zZyQp6Z3z", "reviewer": "Reviewer_3oSY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies the problem of noisy-label learning. Particularly, the paper proposes to use an instance-level consistency metric to identify noisy labels and then apply semi-supervised learning and mixup to learn the signals. The paper shows that the proposed method achieves improvements on datasets such as CIFAR10 and CIFAR100.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper adopts a clear intuition and conducts empirical studies to verify the claim. However, such observations/intuitions have already been studied in [1]. Although some detailed designs are different, the paper's approach largely overlaps with [1] (e.g., using a metric to identify noisy labels based on time consistency, and also utilizing semi-supervised learning). Therefore, my biggest concern is how this paper is different from [1], how the findings for the noisy-label training dynamics differ from [1], and what improvements have been made both in terms of the algorithm design and empirical performance.\n\n[1] Robust Curriculum Learning: from clean label detection to noisy label self-correction\n\nThe final loss contains four terms, and the paper uses beta to trade off between the first two terms following a specific schedule. Can the authors provide more justifications for the design of the schedule? Also, why don't we need additional trade-off factors for the other terms in the losses?\n\nMinor:\nAppendix: \"Table 6: ... CIDAR10\"", "audience_yn": "Yes", "audience_explanation": "Noisy-label learning is a real-world problem that has a rich literature.", "requested_changes": "Please address the comparison with [1].", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UmwxIRSHLH", "reviewer": "Reviewer_tuFR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "## Summary:\n\nThe authors introduce an algorithm to tackle label noise using a metric called the consistency score. They explain the different loss functions they combine and benchmark the algorithm’s performance on both synthetic (CIFAR-10 and CIFAR-100 with noise) and real-world datasets.\n\n## Strength:\n1. Benchmarking on real-world noisy datasets strengthened the claim for SOTA.\n2. The quality of the visualizations is good.\n\n## Weakness:\n1. The paper is not easy to follow. In sections 2,3, and subsections 4.2.4 and 4.2.5, it is not clear which of the ideas are the authors' and which are from related work. \n2. The component analysis table 4 is not sufficient to see which Loss contributes the most to performance.\n3. The results on the CIFAR 10 dataset are not favourable, but the authors don't explain. Does this have anything to do with the symmetry of label noise?", "claims_and_evidence_yn": "No", "claims_explanation": "The authors have to make the motivations clearer and the novelty of the paper more convincing.", "audience_yn": "Yes", "audience_explanation": "The idea of using instance-level consistency during training to identify noisy labels is interesting.", "requested_changes": "1. Fix the issues mentioned as weaknesses\n2. Consider adding a Motivation section/subsection at the start, or a Discussion section at the end, to make the paper clearer", "broader_impact_concerns": "N/A", "additional_comments": "n/A", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)