Uniform Noise Distribution and Compact Clusters: Unveiling the Success of Self-Supervised Learning in Label Noise
Authors: Pengcheng Xu
Li Yi
Gezheng Xu
Xi Chen
Ian McLeod
Charles Ling
Boyu Wang
Venue: Accepted by TMLR ()

Abstract: Label noise is ubiquitous in real-world datasets, posing significant challenges to machine learning models. While self-supervised learning (SSL) algorithms have empirically demonstrated effectiveness in learning noisy labels, the theoretical understanding of their effectiveness remains underexplored.
In this paper, we present a theoretical framework to understand how SSL methods enhance learning with noisy labels, especially for the instance-dependent label noise.
We reveal that the uniform and compact cluster structures induced by contrastive SSL play a crucial role in mitigating the adverse effects of label noise. Specifically, we theoretically show that a classifier trained on SSL-learned representations significantly outperforms one trained using traditional supervised learning methods. This results from two key merits of SSL representations over label noise:
1. Uniform Noise Distribution: Label noise becomes uniformly distributed over SSL representations with respect to the true class labels, rather than the noisy ones, leading to an easier learning task.
2. Enhanced Cluster Structure: SSL enhances the formation of well-separated and compact categorical clusters, increasing inter-class distances while tightening intra-class clusters.
We further theoretically justify the benefits of training a classifier on such structured representations, demonstrating that it encourages the classifier trained on noisy data to be aligned with the optimal classifier. Extensive experiments validate the robustness of SSL representations in combating label noise, confirming the practical values of our theoretical findings.

Published in Transactions on Machine Learning Research (05/2025)
Uniform Noise Distribution and Compact Clusters: Unveiling the Success of Self-Supervised Learning in Label Noise
Pengcheng Xu Western University
pru679uuo.co
Li Yi Western University
kyi7@uwo.cou
Gezheng Xu
gmw860uwo.co
Western University
Xi Chen McGill University
ri.chenl @mmogill.cc
Ian McLeod Western University
dimcleod@rwwo...
Charles Ling
linales.lingguuee co
Western University
Boyu Wang' Western University
bwang uovo.co
Reviewed OpenReview: oppnnrvew/e....veeee.. net/ forum? id= LDB
Abstract
Label noise is ubiquitous in real-world datasets, posing significant challenges to machine learn- ing models. While self-supervised learning (SSL) algorithms have empirically demonstrated effectiveness learning noisy labels, the theoretical understanding of their effectiveness remains underexplored. In this paper, we present a theorctical framework to understand how SSL methods enhance learning with noisy labels, especially for the instance-dependent label noise. We reveal that the uniform and compact cluster structures induced by contrastive SSL play crucial role in mitigating the adverse effects of label noisc. Specifically, thcoretically show that classifier trained on SSL-learned representations significantly outperforms one trained using traditional supervised learning methods. This results from two key merits of SSL representations over label noise: 1. Uniform Noise Distribution: Label noise becomes uniformly distributed over SSL representations with respect to the true class labels, rather than the noisy ones, leading to an easier learning task. 2. Enhanced Cluster Structure: SSL onhances the formation of well-soparated and compact categorical clusters, incrcasing inter-class distances while tightening intra-class clusters. We further theoretically justify the benefits of training classifier such structured representations, demonstrating that it encourages the classifier trained on noisy data to be aligned with the optimal classifier. Extensive experiments validate the robustness of SSL representations in combating label noise, confirming the practical values of our theoretical findings.
Corresponding author
1

---

Published in Transactions on Machine Learning Research (05/2025)
1 Introduction
Label noise is ubiquitous in the real world since acquiring accurately annotated datasets is expensive and Hime-consuming (Patrini et al. 2017 Xiao et al., of annotated images can be collected from unroliable sources such as non-expert annotators and image search engines, where label noise is inevitable (Xia et al.
2015 Natarajan et al.
2013 Alternatively, a large amount
2020a; el al.
2017).
Recent self-supervised learning
(SSL) with contrastive learning paradigms achieved great success in learning meaningful data representations
without label information (He al. 2020 Chen et al. 2020c Zbontar al. 2021 Caron et al. 2020
In SSL, any two augmented examples from the same image (referred to as positive pairs) are mapped to nearby locations in the embedding space, whereas two augmented images from different images (referred to as negative pairs) are mapped to a distant location (Oord et al. 2018 Purushwalkam & Gupta 2020 Chen
et al. 2020b0. Empirical evidence demonstrates that representations learned by SSL can be easily adapted to many downstream tasks such as image classification, objection detection, segmentation, and learning with
imbalanced datasets (Grill et al. 2020 Misra & Maaten 2020 Zhao et al. 2021 Xie al. 2021 Liu et al.
2021
Yang & Xu, 2020
Apart from these applications, in this paper, we show how SSL representations enhance learning with label noise. Specifically, we first construct motivating example of instance-dependent label noise, then we prove that a classifier trained on representations learned by SSL with noisy labels is optimal over clean data distribution. Then we systematically analyze the benefits of representations learned by SSL and find two merits of SSL representations: (1) Uniform Noise Distribution: The label noisc uniformly spreads over the learned SSL representations, (2) Enhanced Cluster Structure: The learned representations exhibit separated and compact cluster structure with respect to true labels. It increases the distance of clusters from different classes while reducing the variance of the cluster of the same class.
For point (1), we theoretically show that the label noise is uniformly distributed across the learned representa- tions by SSL in the motivating example, which is easier to address in practice (Cheng et al. 2020: Chen et al,, 2021a Zhang et al. 2020). We further extend the relationship between label noise and the representations learned by SSL to more general case and provide empirical validation. For point (2), we empirically and theoretically justify that representations learned by SSL exhibit cluster structure based true labels.
Furthermore, demonstrate that such a structure encourages the classifier trained noisy data to align more closely with the optimal classifier learned from clean data. In contrast, representations learned through supervised learning (SL) do not achieve uniform noise distribution or form a good cluster structure. In particular, representations learned through supervised learning still depend on noisy labels and they exhibit clusters with respect to noisy labels instead of true labels.
From the algorithmic perspective, we show that mixup over SSL representations boosts the property of cluster structure discussed in our theory. Specifically, following the common setting of SSL evaluation (Chen et al. 2020b He et al. 2020 Grill et al. 2020). we fix representations learned by SSL and then only train linear classifier on the frozen representations with different types of label noise methods, which show significant. improvements. The main contributions are summarized as follows:
We provide theoretical analysis to show that representations learned by SSL exhibit a more uniform and discriminative cluster structure under label noise, leading to better classifier than supervised learning.
We systematically show that SSL breaks the dependency of label noise and representation, resulting in uniformly distributed label noise over classes. Further, SSL enlarges the distance of clusters from different classes while tightening the cluster of the same class.
Empirically, we show that mixup based on conventional SSL augmentations satisfies our theoretical motivations and benefits cluster structure learning. Extensive experiments validate our analysis and the effectiveness of SSL in noisy label learning.
2

---

Published in Transactions on Machine Learning Research (05/2025)
2
Related Work
Learning with Noisy Labels. There are different approaches to addressing the issues caused by label noise. Commonly used loss functions such as cross-entropy loss (CE) are not robust to label noise. Therefore, noisc- robust loss function methods are proposed to make models fit clean examples but not mislabeled examples
(hang & Sabuncu 2018 Wang et al. 2019 Ma et al.) 2020 Englesson & Azizpour 2021). Alternative solutions design sclection strategics to improve the confidence of clean examples and filter them from noisy data[Han et al. 2018 Yu et al. 2019: Wei et al. 2020; Song et al.) 2019 Yang et al., 2023). Label correction
methods correct pseudo labels of noisy samples for computing their loss functions (Zhang et al. 2021 Liu
et al. 2020 Yi & Wu 2019 Zhang et al. 2020 Noise transition matrix methods estimate the underlying
label noise distribution and use it to correct the noisy labels and build a robust classifier, Other solutions
include using the neural symbolic system to model and reduce the noise Smmrnovaa et al. (002 filtering
noisy labels from sample and parameter levels (Wang ct al., 2023), and enhancing the label accuracy with
graph fusion (Xu et al. 2022). Cheng et al. (2021) provides a study on distilling the knowledge of SSL representations to supervised learning representations. In contrast, we focus on revealing and investigating how plain SSL representation can succossfully address the label noise issuc.
Self-supervised Learning. Representations of images learned by SSL have achieved remarkable success. SimCLR (Chen et al.) 2020b) requires a large batch size to contain sufficient in-batch negative pairs and domain-specific augmentations such as Gaussian blur, color distortions, and color jittering. However, a large batch size is infeasible. MoCo (He et al. 2020) solves this issue by introducing a memory bank to store representations of data from previous iterations. The later work this series uses the optimized large batch to remove the memory bank and introduce ViT into the framework (Chen et al., 2021b). BYOL (Grill et al.) 2020) and SimSiam (Chen & He 2021 propose new frameworks without using negative pairs so they can work with a reasonable batch size. On the other hand, SSL relies on domain-specifie image augmentation. That is to assume that image augmentations such as changing the colors of images should not affect labels of images in downstream tasks Tsai et al. 2020 DACL (Verma et al. 2021) and I-MIX (Lee et al. 2020 both leverage mixup augmentation (Zhang et al. 2017) as domain-aguostic angmentation and they find that SSL
methods with both domain-agnostic and domain-specific augmentations perform better.
Noisy labels with SSL. Applying SSL methods to mitigate label noise has recently been studied. Li et al. 2021 propose the contrastive learning loss in the principle subspace via autoencoder and the mixup in the low dimensional space to learn a robust representation. Ghosh & Lan 2021 empirically validates that the representations pretrained by SSL methods are beneficial for noisy label learning. C2D (Zheltonozhskii et al.,
2022) propose a contrast and divide method to empirically address the warm-up obstacle for memorixing the noise. Yi et al. (2022) analyze the memorization issue in cross-entropy issue and propose a contrastive regularization to mitigate it. Xue et al. (2022) shows that contrastive learning boosts robustness by analyzing the singular values of the representation matrix. However, these methods either lack systematic theoretical analysis or rely on extra assumptions of sub-class structure to show how SSL benefits noisy label learning.
Different from these discussed methods, our method provides theoretical analysis that SSL methods benefit noisy label learning by learning more separated and compact cluster structure. Concretely, we analytically show that SSL can enlarge the cluster distance from different classes and reduce the variance of each cluster. Further, we empirically prove that the built-on mixup can achieve our theoretical motivations and improve noisy label learning.
3 Analysis of SSL representations with noisy labels
We first. provide a motivating example to show that SSL can be significantly better than supervised learning, which enables us to explore and investigate the benefits of representations learned by SSL.
We first construct binary classification problem with two linearly separable clusters, where the samples from clusters are artificially flipped according to label noise function. We denote y, by the true label for x;
3

---

Published in Transactions on Machine Learning Research (05/2025)
and assume it is balanced sample from 1-1,+13 Then the instance 2; is decided in the following manner:
e14i + E2E,,
if y = +1
2i
cic e251:
if y/:
where Ç ~ Mo..j. & ~~ mm- b, m E R. b > m > 0 denote the interval without the loss of generity, and e1,e2 E R2 are two orthogonal unit-norm vectors. We assume ((x, v) = sign e2) as the instance-dependent label noise function. For each clean example (r), the corresponding noisy example (24, ūt), where T. = y: Vi). Then we can compute that there are 43.75% mislabeled examples if the noise function ((x, y)
is adopted.
We use a simple linear classifier parameterized by w and use the gradient descent algorithm learn the parameter over the noisy data {Iz, Vo De with logistic loss function. Thus, in conventional supervised learning, we have the loss:
((w)
Σ log (1 + exp (-Dww xx:)))))
In contrast, following the common practice in SST research Chen et al. (2020). we first learn a linear representation model with parameter W E Tixx from It in a suu---rieisied
(2020b); He et al. (2020) Grill et al.
manner. Specifically, we adopt the linear SSL objective function studied in Liu et al. 2022 HaoChen et al. 2021 which tends to pull two positive pairs (x + 7, + 7') to nearby locations in the embedding space:
WSSL arg min B[( + +7) Tw WWTTWWTW W(x++0)+++ W W WeRlkd
where É is an empirical expectation over the data, with 7 are independent and identical ( I) random variables. Once the optimal representation model Wsst. obtained, we fix WSSt and then learn a linear classifier parameterized by f on the top of representations with noisy labels {(WSSL H.). Analogous to supervised learning, we also use the gradient descent with a logistic loss function L()) to train the classifier.
((0)
1 n
Σ
n log (1 + exp (-0.0 --s..:Wssss):)) WssL.)))-
The following theorem states the behavior of linear classifiers on input data {1 ūi)); and representations of inputs data (Wsstt,r, }i) respectively.
Theorem 3.1. For a linear classifier trained with logistic loss function, let w, be normalized optimal paramelers via gradient descent with logistic loss over the data U1r ūi)} and {[Wssu. ūi) respeclively. Then the generalization accuracy in supervised learning is upper bounded by:
2d
Pr [Si xx) = y] < (x.3)
(2)
16
3n
While the generalization accuracy in SSL lower bounded by:
Pr (0 ToannO@Wsshe WSSLE) yl 2 2e -2/128
(3)
(a.y)
Remark Theorem 3.1 reveals interesting facts in the presence of label noise. (1) The prediction accuracy under SSL is guaranteed to be high value via a provable lower bound. The lower bound could further converge to (perfect prediction without error) when sample size n - too. (2) In contrast, in supervised learning, simply collecting more samples does not guarantee a high accuracy, where the upper bound of the accuracy converges to 9/16 as n + too. Theorem 3.1 is in proved in Supplementary Section B.
4 Why SSL Works
To show the benefits of SSL in noisy labels, we analyze the learned representation WSst. from Eq.
4

---

Published in Transactions on Machine Learning Research (05/2025)
(a) SL(noisy labels)
(b) SSL(noisy labels)
(c) SL(true labels) (d) SSL(true labels)
(c) SL(noisy labels) (f) SSL(noisy labels)
Figure 1: T-SNE of 60% instance-dependent label noise on CTFAR-10 and Clothing We train a 3esNet34
on the noisy data by supervised learning (SL), and we visualize the representations learned by SL in (a) and (c) with respect to noisy labels and true labels, respectively. Similarly, we also train an SSL ResNet34 and visualize data representations in (b) and (d). We highlight regions with solid polygons that suffer from the label noise in (a), where red points (label noise) represent wrong-labeled data. In (b), the red points are almost uniformly spread over the data representations. Similarly, we also show results on the real-world Clothing1M trained with ResNet50. The similar observation shows that the label noise is not uniformly distributed in SL (e) while almost uniformly distributed in all classes SSL (f).
Proposition 4.1. Let WSST. denole the feuture represenlation learned from SSL and en denole the discrimi- native feature-related true labels. The optimal solution WSSL in Eq. converges in probability to ke with the constant k 0.
The solution WSSL the span of the vector 61, which is crucial for learning an optimal classifier in Eq. (q. Note that only el determines the true labels of data In fact, the injected label noise depends the non-discriminative feature E2 but not the discriminative feature et. If we orthogonally project data onto the direction of 61, the label noise is independent of and is uniformly distributed over the projected data points spanned by E1. The representation model WSSL exactly maps the data x onto the direction of e1 orthogonally. Thus, the label noise is uniformly distributed over data representations WSSLE, which makes the label noise easier to address.
Remark The Proposition 1.1 indicates that the label noise dependent on the non-discriminative feature e2 is uniformly distributed over the projected data representation WESt. spanned by e₁-
Now, we discuss the benefits of uniformly distributed noise toward learning a generalized classifier from two aspects. First, if the label noise is uniformly distributed across the representation, the classifier trained on dala representation with this label noise can generalize better. Specifically, it can be verified that the optimal Bayes classifier, h(z)= = sign(e, z), is also the optimal classifier over the clean distribution. On the other hand, the classifier trained with the supervised learning mothod from Theorem 3.1 is forced to learn spurious correlations between the inputs e2 and the labels.
Second, estimating a noise transition matrix is easier when the label noise is uniformly distributed over inputs. In particular, the instanco-dependen label noise can be characterized by the noise transition matrix T(x) RDiXDD", where T(x),; measures the probability of observing corrupted label j given the true label and an instance v. The issue of label noise is then solved by estimating the noise transition matrix T(x) (Xiao et al. 2015 Liu & Tao 2015 Patrini et al. 2017 Goldberger & Ben- Reu-Re 2016). Estimating T(x) for vustance-dependenn label noise is practically challenging since T(x) can be different for different x and we may need parametrize n different T(x: from the noisy dataset of size n by a neural network (Cheng el al. 2020 Xia et al.) 2020b Berthon et al., 2021 In contrast, T is the same for all x for symmetric label noise (i.c, label noise uniformly distributed over data) (Patrini et al. 2017) and we only need estimate constant noise transition matrix. Therefore, by estimating single noise transition matrix instead of parameterizing n noise transition matrices by neural network, the label noise is casier to solve.
5 Generalized Observations in Real World
In this section, we validate our theoretical analysis of the SSL representations in the real-world data. Concretely, we visualized the feature representations learned with supervised learning and SSL in Fig 1
5

---

Published in Transactions on Machine Learning Research (05/2025)
The data representations of supervised learning for noisy labels are in Fig (aa while representations of SSL are in Fig [b). Fig 1 shows that in the representations learned by supervised learning, the label noise and representations are still dependent (solid polygons highlight the regions), whereas the SSL breaks such dependency and makes the label noise uniformly distributed across the data representations. We also get similar observations on the real-world Clothing1 M dataset as shown in (e) and (I). The representation of green and yellow classos have less label noise, which indicates that label noise is not uniformly distributed in the supervised learning classifier in Clothing This empirical comparison and evidence in real-world data confirms our analysis about uniformly distributed noise in Proposition 1. Quantitatively, results in Table show SSL clearly and consistently improves the performance of label noise, which concides with the implication of Theorem 3.1 These experiment results validate our experiments quantitatively and qualitatively.
Besides, we also visualize these representations with respect to their true labels in Fig ! c-d). We find that representations learned by SSL, exhibit an intrinsic cluster structure that is consistent with the true labels (Fig dd). In contrast, Fig cc) shows that representations learned by supervised learning do not exhibit a cluster structure with respect. to true labels. Thus, it further motivates explore how the cluster structure learned by SSL helps noisy label learning. We explore the theoretical properties of SSL on cluster structure in the next sections.
6 Compact Cluster Structure for Label Noise
In this section, we invostigate how the cluster structure can help mitigate the label noise. Concretely, we show that for fixed representations, good cluster structure encourages the classifier to be aligned to the optimal classifier, resulting in better generalization performance.
For simplicity, we use two-component Gaussian mixture model to describe the clusters of reprosentations, with each cluster representing one class. We assume that representations from class +1 are sampled from N(p, Σ) and representations from class - are sampled from (N-n,, Σ), where E Rd and Σ E Rd In this case, the distance between two clusters is controlled by [|l|, and the variance of each cluster is controlled by the sum of eigenvalues of Σ, which is equivalent to the trace of Σ.
We connect the cluster structure to - -V((wb) T. which can characterize the performance of the lincar V[(wa) classifier, where V[(wo) = [VC[[ j = Tमm and V((w)) is the gradient of the logistic loss computed by the linear classifier (initialized by wo). The normalized gradient of the loss VV((w) represents the direction of the steepest descent in the loss function calculated on the noisy data. Given that an optimal classifier obtained from the clean data is kj for any scalar k > 0, ((wo) is can measure the cosine similarity between the gradient descent direction and the direction where the optimal classifier points. After applying one-step gradient descent on the classifier, the updated classifier is more correlated to the optimal classifier if the cosine similarity is higher. More details can be found in the Appendix. This intuitively explains why - -V[(wb) ā can be used to measure the performance of the linear classifier. Now we focus on establishing the relationship between VV((w)) T j and the cluster structure.
As shown in Section i label noise uniformly distributed over representations. Thus, we define the symmetric
label noise function:
3(x,y) - -1, with probability +1, with probability 1 1-r
where 0 0< <<< controls the noise level. Note that for symmetric label noise, the label noise function ,(x, w) is independent of the data. The relationship between -[((w))T j and the cluster structure is presented in Theorem 1
Theorem 6.1. Let r denote the noise level of symmetric label noise function B(x, v), if at least half of examples are clean (r < 2)²
||
-C((u) 722 p2
2r) + onn onnnn11/ -1/3),
(4)
cIr(E) ||lll
where Tr(E.) is the frace of E and > O is conslant.
6

---

Published in Transactions on Machine Learning Research (05/2025)
Theorem 6.1 provides lower bound for -V[(wb) ft. The larger lower bound means the updated classifier is more correlated to the optimal one.
The lower bound can be affected by the noise level r and the following two cluster properties: 1) the distance between two clusters ||사 2) the variance of each cluster Tr(E). Without considering any label correction techniques, the noise level r is fixed given a dataset. Therefore, by learning clusters of data representations that are distant from each other (larger [| and/or by learning tight representation clusters (smaller Tr(2)), the classifier generalizes better.
Remark We remark that the spirits of encouraging good cluster structure are the same for other forms of label noise such as asymmetric label noise, though their expressions (wo)) j are different. Details are in Appendix C.2.
Clean Noisy
Clean
Noisy
2.0
1.-
10
We empirically justify that linear classifiers get better per- formance when -V((w)) p becomes larger. Fig shows the performances of classifiers trained on dala points with different cluster structures given fixed noise level 40%.
0.
-2.00 -0.79
50
-025 0.9
625
-1.0
-0.3
50
20
6.0
025
Sample Loss (In Log)
Sample Loss (in ingl
100
Text Rccurres Aaoeany
Specifically, with the same variance, the classifier trained on clusters with larger distances performs better (orange line v.8. red line). While with the same distance, the classifier trained on tight clusters performs better (orange line blue line). The two histograms show that the linear classifier with larger- - (W0))) já (orange) fits clean examples better, compared with the lincar classificr (red). It also highlights that representations with better cluster structure help the classifier generalize better clean data distribution.
80
||||
2.5
Nolsy Dataset
[ell
2.11
Σ.
I
Σ
1,251g
Epachs
Figure 2: Linear classifiers trained on synthetic datasets with 40% noise level. The dashed line is the performance of classifiers without label noise and the solid line is that with label noise. The histograms sample loss values at epoch 50 with respect to whether they are mislabeled.
7 Benefits of SSL on Cluster Structure
In this section, we rigorously justify that the cluster properties characterized Theorem 0.1 can be achieved by SSL. In particular, we focus the SSL objective function Eq. (5) that has been studied in Wang & Isola
(0200) Notably, the loss and its variants have been widely adopted in the SSL (Chen et al. 2020b He ct al.
2020
Tsai et al. 2021
Larls) =
1 Σ 121
E Eur of PP(N)Es))
[I] (u:)
(5)
(Alia(())
le- I1l (011
+ A Alog
B Bu.-Petulz)) Pa
11(1
L (b:j)
where À is hyper-parameter. (x1, ,nn input instances, P(u) ar) denotes the conditional distribution of an augmented instance gievn z, and f(-) is representation network that takes Eq. 5 minimizes the distance between two representations of different views from the same instance, and the Luniform makes representations uniformly distributed over the embedding space.
input. Intuitively,
To characterize the cluster properties studied Theorem 3.1 we introduce the notion of 8-cluster closeness.
Definition 7.1 (ō-cluster closeness). Let St be the support where Pr(ulz.) > 0 for any u S,, S, and S, are 8-cluster close if Pr[u E S, n S Sy[æ;] 2 5 for any two different instances F, 2y with y: = y/-
The notion of 8-cluster closeness is similar to the cluster assumption in Lafferty & Wasserman 2007 Rigollet reveals that the instance augmentations should be rich enough
2007); Singh al. 2008). Definition
7

---

Published in Transactions on Machine Learning Research (05/2025)
that any two different distorted augmentations from the same class can be overlapped. Following (Verma et al. 2021: Lee et al. 2020). apply mixup data augmentation augmentations (Chen et al. 2020bb in order to have richer instance augmentations, making Definition 7.1 hold with a large 6. The definition d-cluster closeness provides us a tool to investigate how Eq. the cluster properties: the distances between any two clusters and the variance of each cluster.
top of the conventional SST, data
F infiuence
To analyze the cluster properties, decompose (ft into three components and study their effects individually.
Larlf)
Σ mEy tEJm
Σ
Calign (1) Alog
Σ
Luntform (
n(n -
me nE JEVP
Luttfrrn (i.j)
n(n PEEEEEJ.
The following lemma indicates that a large distance between any two clusters can achieved by optimizing LUniform (i, for some i.J. We denote Jy by the index set corresponding to true class y.
Lemma 7.2. Let ia = ExEn Hupl, PJ EkeJ, [(uz] be sample meons of cluster i and cluster j with # j.
TJa
Without loss of generality, we assume [J&] = رر Then
B[lAA Asll 2 J&
Σ REJ:
log (Lunsform g()))(())))))))))))))){{))))(g
where the function J. Jj is any bijective function, and the expectation is over the dala augmentation.
Remark Lemma 7.2 indicates that the distance between the cluster and the cluster j can be lower bounded by - log (Luniform [kk)) for E J. Since - Luniform (k, g(k)) measures the distance between f(ux) from cluster i and f(ug(k)) from cluster j. minimizing Uniiform(ks g(k) for all e Jj increases the distance between the cluster and the cluster j.
On the other hand, the objective function Eq. (5) also controls the variance of each cluster. The following lemma helps us understand how the SSL objective function Eq. (5p controls the variance of each cluster.
(f(:)) - (f:)) - -a,) be the sample covariance matria. Suppose Definition 7.1 holds. Then for any fixed 8 E (0, 1), we have
Lemma 7.3. Let Σ. = J (f(u))
THE S J iEjy
CAtio(t))
where the expectation is over the data augmentation.
Remark Lemma 7.3 shows that the variance of cluster y is upper bounded by the term Σie,, Calign (i), where the variance Is measured by the sum of eigenvalues for the sample covariance matrix computed by representations from the cluster y. In other words, small variance of cluster y be achieved by minimizing Exe,, Latign (i).
Lemma 7.2 and Lemma have shown the effects of the first components in Eq. The last component in Eq. serves contradiction against the first component. We note that minimizing Luaiform (i. for i.j from the same cluster undesirably increases the variance of that cluster. This intuition is justified by the following proposition.
Proposition 7.4. Suppose Definition
7.1 holds with fixed 8. Then
logl n(n 1)
Σ
Σ
(in)) -a ΣΣ Alsggn (i),
mey kmj
mEy ieum
where -(1 8n(n-1) > 0.

---

Published in Transactions on Machine Learning Research (05/2025)
Table 1: Test accuracy on CIFAR-10 and C1FAR-100 with SYM label noise over different noise levels.
Datasct
CIFAR-10
CIFAR-100
Noise
20%
40%
60%
80%
90%
20%
40%
60%
80%
90%
SSL+CE
92.66:00006
92.60=0.06
92.5320.11
92.22+0.04
91.61+00.4
64.1620.15
63.7620.72
62.25+0042
60.56.0.27
57.07+0 96
GCE
93. 83.1662..18
90.11=0.27
02:35
74.95+0..51
54.34+0.51
71.71#0.00
67.72=0.19
59.30=0.40
35.80==0622
14.04+0.99
+ Mooo
05.74+0.07
90.67=000
95.58=0.01
95.36:+0.08
94.68+0.24
75.21+0.05
74.89:0.06
73.36=0000
71.91=0.91
68.224+0.7
BYOL
95.555:00.02
95.46-00.5
95.32:006
95.11100.00
94.66:0.10
73.5322000
72.0420001
71.43::0.09
69.40:0020
65.94.20 26
CT
93.66
92.22:0.16
70.51=0.22
09.75
27.34+0.98
72. 722669400144
68. 18.81100199
61.15+0.26
16.40=0.44
8.22
+MoCo
95.43:0000
95.37=0.05 0.00
95.19::0.25
91.97
87.65:1.66
73.86
73.37:00112
72.59:0.41
67.79=0.92
62.69+2
+BYOL
95.13=0.02
0.004
94.71#0.08
93.58+0.56
87.35=1137
72.19=0.05
71.33=0018
69.49:0008
5..55=3.28
52.65+1.22
ELR
83.53
93.
93.11:0.14
92. 02
85.7110..2
51.27:11.66
69. 0.9../1
65. 15.100
60.88
60.88:10.32
21.92=0.52
10.224...0
+MoCo 95.88
95.81=0.04
95.74=0.05
95.65++0.c2
95.60=0.00
72.74=0.06 72.89=0.39
71.74=0.11
70.47=0.19
66.75+0.22
+BYOL
95.55
95.13
95.30.00.0
95.11100.5
95.1210010
72.18.00.0
71.7310.06
10.35
68.15
63.70.0..1
TCL
93.53:20.10
94.73=0.11
03.2220:13
90.34:10.29
87.51
76. 76.2420025
73.1620025
69.46:20.33
62.71
63.58110.5
+MoCo
95.91.00.0
95.71=0003
95.82.0..3
96.6810002
95.3010.07
78.81
76.79.0016
72.6210.11
71.87 20.23
68.6311
BYOL
90.59
95.60=0.13
95.1520.0
95.0330.08
94.95520.1
78.6820.05
76.41:0018
72.25
71.65=0.22
66. 13:0
Table 2: Test accuracy on CIFAR-10 and CIFAR-100 with ASYM label noise over different noise levels.
Dataset
CIFAR-10
CIFAR-100
Noise
10%
20%
30%
40%
45%
10%
20%
30%
40%
45%
SSL+CE
92.25:20.06
07.63:0.21
86. 29.11
83.8620.22
79.13=a0.5
61.01=0.20
18344200.11
62:24=0.62
09.06
53.16120.54
GCE
93.0240.10
91.92:0.21
25
89.11:10.4
86.51
73.52200.0
70.0520.31
65.8020.35
53.19=0.5a
111088::.2
+ +MoCo
95. 95.63:#0.04
95.37=0005
95.00=0..1
03.30+0.26
88.31+0.41
74.54=0.04
73.60=0.12
72.63:0011
66.27=0.24
56.12+0.6
+BYOL
95.50.0002
95.23
94.83310.19
93.56100.6
90.69.1016
73:17.20.
72.12.0009
70.75.90.4
65.09.0.13
53.96.0..0
CT
94.40:2000
93.32
10
09.47:10:1
66.08:10.32
73.88220.0
00.8820:21
64.64:0.08
55.2 25.22
48.22:1:00
+MoCo
95.37 00007
95.25_000
94.33.10.16
92.29.10.32
86.7910.52
73.48.10.11
72.0210.26
69.3619.41
63.30
65.70.1..8
+BYOL
95.48&0.00
91.11=0.72
91.01110.4
90.72.10.2
87.3
72.01=0.05
70.161...4
66.22:0.37
51..7=00.44
46. 12
ELR
93 9..00
93.26 03
02.62210.13
90, 10.93:10.16
88.49110.24
73. 89
73.44.2000
72.90.0.11
70.02200.4
65. 05.62211.33
+ +MoCo
95.73:0002
95.69=0.01
94.83=0.12
92.6221.11
78.92:0..5
74.87=00.05
74.51:0010
73.75=20.08
72.26=0:05
67.1 28
+BYOL
95.59920.0
95.49_0.07
95.402000
94.7210.04
21.91
73.4422005
72.9520004
71.81 29.05
69. 1820.s
63. (3.27720.3
TCL
94.10=00:0
93.53=0.17
02.5220.11
91.14:2028
90.91
78, 78.2420.24
74.77=0008
73.98=0.22
71.62=0.52
67.38+2.35
+ +MoCo
95.18220.00
95.33
01.0720011
93.85:10.12 12
93.60:0.09
80.89
78.5422016
77.74
73.88
69.35520 25
+BYOL.
95.25=+0.12
95.38=0.18
95.10=0.11
93.21++0.00
93.17+a010
80.28=00.15
78.12
77.25
73. 03.152
68.65
Remark Proposition 7.4 indicates that minimizing the third term in Eq. 5 forces the first term to be larger, which makes the variance of clusters to be larger, where the strength is controlled by a factor a. We note that the third term is due to Luiiform of the instances from the same class and it cannot be eliminated since the label information is not leveraged. The constraint strength is mitigated when a decreases. It is small when instance augmentations are rich enough (5 is large), which also highlights the importance of data augmentations in learning SSL representations. In conclusion, our analysis for the SSL objective in Eq. reveals that SSL helps enlarge the inter-cluster distance (Lemma 7.2 and reduce intra-cluster variance (Lemma 3 and the data augmentation can help get a better trade-off (Proposition 7.4
8 Experiment
To validate our analysis of the cluster properties of SSL representations, we combine different SSL methods (i.e., MoCo and BYOL) as complementary with label noise methods. Compared to MoCov2, BYOL does not explicitly compute LUniform but implicitly computes it by the momenturn update of the network.
Datasets. Following previous state-of-the-arts (Hanng et al. 2023 Yang et al. 2023 we evaluate method on CIFAR-10 and CIFAR-100 (Krizhevsky et al. 2009) with different types of label noise: symmetric (SYM), asymmetric (ASYM), and instance-dependent label noise (IDN). We also evaluate our method on two real-world datasets ANIMAL-1ONN (Song et al. 2019) and Clothing M (Xiao et al. 2015).
CIFAR-10 and CIFAR-100 both consist of 50,000 training and 10,000 test color images whose size is 32 32. The difference that CIFAR-10 has 10 classes containing 5,000 training and 1,000 test images for each class while 100 has 100 classes containing 500 training and 100 test images for each class.
9

---

Published in Transactions on Machine Learning Research (05/2025)
ANIMAL-ION (Song et al. 2019 is datasct with animals of confusing appearances crawled from several online search engines including Bing and Google using the predefined labels as the search keyword. It includes 50,000 training and 5,000 test images. Since images are collected and labeled by online search engines, the resulting classification has an estimated crror from 6% to 10%.
Clothing1M MM (iaoo et al. 2015 has million training images and 10,000 test images from 14 classes. The noisy images are collected from online shopping websites, and there are many mislabeled samples since the labels are created by the surrounding text provided by the sellers. These texts can be coarse and induce realistic noisy labels.
Implementations. The implementation of MoCov2 and BYOL includes the backbone and a 2-layer MLP ad the projected head. For CIFAR datasets, the backbone is ResNet34 with a 2-layer MLP as the projection head. The input and output dimensions are 512. For ANIMAL-100 and Clothing the backbone is ResNet50, and the input and output dimensions are 2048.
For the data augmentations used in MoCov2 and BYOT, we use both strong image augmentation from Chen et al. 2020bb) and MixUp from Lee et al. (020)). MixUp (Zhang et al. 2017 has a hyper-parameter À that controls the strength of interpolation between data points, where we set À ==1 for CIFAR datasets and À = 2 for ANIMAL-10 and Clothing- IM. Once we train the representation network, we train a linear classifier by different label noise methods on this representation network.
The linear classifier is trained for 100 epochs using SGD, where the learning rate starts from {1. 5, 10, 20, 30} and it is reduced by a factor of 5 after 20, 30 and 40 epochs. For GCE method (Zhang & Sabuncu 2018), its parameter q is selected from {0.2, 0.4, 0.6, 0.8, 0.9); for Co-teaching method (Han et al.] 2018),
the warmup parameters is selected from (5, 8, 10]; for ELR method (Liu et al, 2020), the parameter B selected from {0.7, 0.9} and the parameter X is selected from f3, 5, 7}: for TCL method (Huang et al. 2023
two augmentations are used, and the temperature T of contrastive loss and the a of mixup are 0.25 and 0.1. All experiments including the SSL training are conducted on two Nivida A100.
Baselines. To show our analysis generally facilitating learning noisy labels, we combine frozen SSL representations with different types of algorithms: robust loss function GCE (Zhang & Sabuncu 2018).
sample selection Co-teaching (Han al. 2018), label correction ELR (Liu et al. 2020), and TCL (Huang
el al. 2023 that models and filter samples simultaneously, and the standard cross-entropy CS).+CE). GCE designs loss function to address memorization issues for incorrect labels. Co-teaching selects clean examples to update the neural network. ELR introduces a regularization for pseudo labels. TCL uses the GMM model to model the noisy label distribution and filter wrong labels as out-of- distribution examples.
Main Results. Tables 33 show the results for SYM, ASYM, and IDN label noise on CIFAR-10 and CIFAR-100, respectively. Table shows the results on ANIMAL- 10N and Clothing Both MoCov2 and BYOL SSL representations can improve efficacy to wide range of label noise methods: robust loss function methods, sample selection methods, and label correction methods. Results show that training a linear classifier on frozen SSL representations over noisy datasets is significantly better than training a whole neural network over noisy datasets. Asymmetric label noise is to flip labels between semantically-sinii example, cats are inherently more difficult to differentiate from dogs than trucks. To this end, we combine IDN with ASYM to generate more realistic label noise. Specifically, we choose similar images for cach class and then we flip their labels to the next. class. Results are reported in Table 1
classes. For
Following Zhang et al. 20200; Chen et al. (220aa) Lee et al. 2019 semantic label noise is type of instanco-dependent label noise that follows the intuition that hard instances are more likely to be mislabeled, where the hard instances are near the decision boundary of the model. To generate the semantic label noise, we train a VGG-13 (Sionnyan & Zisserman 2014h on training datasets for 30 epochs. Following Chen et al. (2020a), we select instances with the highest mislabeling scores to corrupt. For the first case, we corrupt these instances with random labels. For the second case, we corrupt these instances with predictions of the model VGG-13. We term the former TYPF-1 label noise and the latter TYPE-2 label noise. The results for the two types of label noise are reported in Table 5 Therefore, extensive experiments have demonstrated the effectiveness of applying frozen SSL representations.
10

---

Published in Transactions on Machine Learning Research (05/2025)
Table Test accuracy on CIFAR-10 10 and CIFAR-100 datasets with IDN label noise over different noise levels.
Dataset
CIFAR-10
CIFAR-100
Noise
20%
40%
60%
80%
90%
20%
40%
60%
80%
90%
CE
91. 36.66
30.79.0.1a
89.6520.25
87.80
80.01a1.34
65.78
63. 13.194011
61.47a005
58 38.8420.40
541.6620.57
GCE
20
80 50.35=0..4
06,91+n.51
19.38=0.66
31.19
69.58
60.4840.32
14.03
28.3 28.34=0.84
14.18
+MoCo
95.41=0.07
95.05=0.10
94.35
91.87=0.3
87.72:0.28
74.19:0.04
72.48:0.00
70.47=0.14
66.67
61.67=0.48
+ BYOL
95. 20.00s
94.80=0.co
04.26a.0.5
92.88=0.2
90.33
72.69
70.86:0.10
68.2 08.24:0010
65.20.0.20
60.06
CT
91. a0.50
85.95=0..8
74.09=n.62
30. 00.022
22.35=0.92
69. 81:01.
62. 22.59
52.11=0.65
16. 00.707
7.91
+ +MoCo
95.23
94.68 2013
94.07
77.87=22.0
73.39
72.15
70.14.0.54
66.26.1.08
58.342..08
+BYOL
94.97
94.52
94.11+0.14
89.00
00.0
71.72a1.47
71.59
70.15g0.15
66. :3:
57.79
49.88
ELR
33..54.0.04
93 98.20
92.07.0..0
73
41.39920.80
70.11
77
58.11 10
21.962277
10.28
+MoCo
95.77
95.70-0.10
95.65
98.58
91.35::1.9
72.74.0.44
71.56:0.12
69.69.0.22
65.94:0.69
59.80
+BYOL
95.45.10.2
95.25-00.3
95.0810.04
95.07
94.91::010
72.11=0:18
70.64:0.32
68.72:1.21
63.75:20.0
57.54110:8
Table 4: Test accuracy on CIFAR-10 10 and CIFAR-100 with IDN-ASYM label noise over different noise levels.
Dataset
CIFAR-10
CIFAR-100
Noise Level
10%
20%
30%
40%
45%
10%
20
30%
40%
45%
GCE
87.02+0.16
77.40=0029
08.63+0.05
57.85=0.11
34.01=0.92
71.01=0.12
02.42=0.15
52.48+a.5s
41.0920.7a
40.02+0.65
+MuCo
95.20
94.89=0.05
93.28:0.2:
84.26-004
71.66:1.10
73.90:0.01
71.52:0.57
69.12:::21
61.69:0.7
52.17+2.25
+DYOL
95.19+0.05
94.85=0010
93.66=0.21
85.57=0.77
70.14=1.44
71.08=00:4
69.61=0.13
67.22s0.0
59.33
50.37
CT
87.75
38
69.31
60.48.0.57
54.62au.80
71.97
64.332o.13
55.61
47.12s0.3
42. 22.233...s
+MoCo
94.73:1.45
23.25
88.92:0084
74.67
61.15
20.322
69.61
66.521:0.0
59.37
50.43
+BYOL
94.8810006
93.55
90.99....s n9
74.80
63.66
19.54
67.69.0.12
64.44.9.19
59.23..032
51.6510.8
ELR
94.0820.05
03.97
93.9110.14
93.79-0.20
77.76=2.24
:2.21
71 71.06:0.25
71.83
70.96
67_3310.45
+MoCo
05.72
95.60=0.04
95.16=0.0s
95.23
95.01=0.20
73.90
73. 73.16
+2.69
70.11
611.1#1..9
+BYOL
95.39
95.30=0.0s
95.20
94.99-0.14
85.19=0.32
70.58=0.12
89.8000..0
68. 60
66.95
63.32+..61
ASYM
SYM
IDN
SSL
CLEAN
1.0
0.8
Distance
Variation
015
0.4
0.10
02-
DEER
TRUCK
ARPLANE
DOG
AUTOMOBLLE
ARPLANE
DOG
HORSE
(a) Distances between different clusters
(b) Variation of clusters
Figure 3: Illustration of cluster structures for CIFAR-10 dataset. Representations are learned in different. label noise settings. ASYM (blue): 40% asymmetric noise; SYM (orange): 60% symmetric noise; IDN (green): 60% dnstenco-dependent noise. And we visualize distances between two clusters in (a) and the variance of each cluster in (b). The cluster structure (purple) serves a baseline for representations that are trained by supervised learning without label noise.
Cluster Structure. We evaluate our cluster structure of SSL representations learned by MoCov2 on CIFAR-10. Learning SSL representations do not leverage the label information, so the representations are invariant to label noise, whereas representations learned by supervised learning (ST.) are sensitive to label noise. We compare the cluster structure of SSL to that of SL in different label noise scttings Figure B We find that SSL representations (red) have better cluster structure than SL representations obtained with different label noise. We note that although the distances between clusters of SL representations (green) learned with IDN are slightly larger than that of SSL representations, the variance of each cluster is 10 times larger. We also highlight the baseline cluster structure with purple, which is trained by the SL method without. label noise.
Finc-tuned Performance. Following Chen et al. (2020b) we fino-tune the MoCov2 representation network on CIFAR-10 by GCE algorithm. For frise-free classification tasks, thnettunig usually outperforms linear evaluation various classification datasets (Cen et. al. 2020b Grill et. al. 2020). However, Figure
11

---

Published in Transactions on Machine Learning Research (05/2025)
Table 5: Test accuracy on CIFAR-10 semantic label noise over different noise levels.
Dataset
TYPE-1
TYPE-2
Noise Level
20%
10%
60%
80%
90%%
10%
20%
30%
10%
15%
GCE
88.6 38.61
79.01=0.0
08.77:0..3
63.21=0:2
33.66
85.31
76.08
70.44
37 37.820..7
+MoCC
95.391004
94.80.0.10
89. 89.94.9016
83.13-095
71.84:00:5
90.92:0.0
13.10.0.12
16.41
69.60.0..
66.68100.5
+HYOL
04
23.
89.19
83.53
75.77.1-0
89.64.0
83.34.0.m
75.6519.37
69.32
66.861004
CT
01306
02.78:0035
44.30
20137=0018
07.90
85.52
76.1
65.361
55.5420.1
48, 48.6410038
+MoCo
95.02+0.43
89.84=0.47
81.70=00.0c
73.34=1.84
59.22
89.00=0:45
83. 0.773
76.55+0.22
72.41
66.66#1.98
+BYOL
94.58+0.07
91.56:0042
86.19+0.70
73.27
6..75
91.06:0.47
10.00
76.50
68,57
63.82
ELR
91.19
91.75=00.11
82.72+0.4
70.86=0.40
19.00=e0.72
86, 00.00
79. 19.06e.0.0
71.02sa.11
62.09
58.02+0.25
+ MoCo
94.96.0.6
84.30.9..
79.99
63.63 1.0
88.27
2..53
78.45
77.57
69.27
+DYOL
95.6620.02
95.35=20.7
88.91#0.3:
77.83=0.4
76.98
89.41=0.57
80,83
78.30
77.10
71.64=1.50
Table 6: Test accuracy on ANIMAL-1ON and Clothing 1M
Dataset
Animal-10N
Clothing-
Method
GCE
CTr'
ELR
TCL
GCE
ELR
'CLL
Origin
84.58
86.93
86.52
87.90
71.34
71.68
71.89
73.81
+MOCO
87.35
87.66
88.51
89.51
72.61
72.41
72.71
74.31
+BYOL
88.42
88.36
88.68
89.43
72.90
72.63
72.98
74.38
100
Teat Accuraty
Accuracs
Text Accu
lvea Fostustion
SCE Nouip
GCE HXX. MET-moWllll
2CI-notesclp ellp
50%
ELS Hapup
ELB-
L1S-notteclp
Epochs
Epochs
CIFAR-10
CIFAR-100
CIFAR-I0
CIFAR-100
Figure 4: Left Two: comparison of linear eval and fine-tune on GCE. Right Two: comparison of SSL methods with and without MixUp component enabled on 80% symmetric label noise.
illustrates that linear evaluation (orange) performs better than finc-tuning (bluc) in the presence of label noise. When label noise exits and the representations are not. frozen, fine-tuning degrades the performance of the neural network. We hypothesize that fine-tuning the learned SSL representations destroys the cluster structure regarding true labels, leading to poor performance.
The effects of MixUp augmentation. We study the importance of mixup data augmentations. Our analysis indicates the importance of keeping larger d in Lemma 7. and Proposition 7.4 by data augmentation. Without MixUp, the Definition 7.1 holds with smaller 8. With MixUp enabled, the bound in Lemma 7.3 is tighter and the negative effects in Proposition MixUp augmentation significantly improves the performance on noisy datasets.
7.4 are mitigated. Results in Figure ! indicate that applying
9 Conclusion
We provide simple but effective method to address label noise. We first construct motivating example to theoretically show that the classifier Icarned on SSL representations generalizes better than that from supervised learning. By further investigating the SSL representations under label noise, we find that: (1) The label noise uniformly distributed over the data representations. (2) Representations learned by SSL exhibit good cluster properties, which encourages the linear classifier be aligned with the optimal classifier. From the algorithmic perspective, we show that SSL representations can be applied as a strong complementary various label noise methods by extensive experiments.
12

---

Published in Transactions on Machine Learning Research (05/2025)
References
Antonin Berthon, Bo Han, Gang Niu, Tongliang Liu, and Masashi Sugiyama. Confidence scores make instance-deeendent label-noise learning possible. In International Conference Machine Learning, pp- 825-836. PMLR, 2021.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint ar Xrx:200006 09882, 2020.
Pengfei Chen, Junjie Ye, Guangyong Chen, Jingwei Zhao, and Pheng- Ann Heng. Beyond class-conditional assumption: A primary attempt to combat instance-dependent label noise. arXiv preprint ar 0rXiv:2012.05458, 2020a.
Pengfei Chen, Guangyong Chen, Junjie Ye, jingwei zhao, and Pheng-Amn Heng. Noise against noise: stochastic label noise helps combat inherent label noise. In International Conference on Learning Representations, 2021a. CRL
Ting Chen, Simon Korublith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, In International conference on machine learning, pp. 1597-1607. PMLR, 2020b.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi-supervised learners. orXiv preprint arXiv:2006.1000 2020c.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750-15758, 2021.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CV /CVF international conference on computer vision, pp. 9640-9649) 2021b.
Hao Cheng, Zhaowei Zhu, Xingyn Li, Yifei Gong, Xing Sun, and Yang Liu. Learning with instance-dependent label noise: A sample sieve approach. arXiv preprint or arXiv:2010.03347, 2020.
Hao Cheng, Zhaowei Zhu, Xing Sun, and Yang Liu. Demystifying how self-supervised features improve training from noisy labels. arXiv preprint 201000000902, 2021.
Erik Englesson and Hossein Azizpour. Generalized jensen-shannon shannon divergence loss for learning with noisy labels. arXiv preprint or orXin:2105. 04.522 2021.
Aritra Ghosh and Andrew Lan. Contrastive learning improves model robustness under label noise. In Proceedings of the IEEE/CV Conference on Computer Vision and Pattern Recognition, pp. 2703 2708, 2021.
Jacob Goldberger and Ehud Ren-Reuven. Training deep neural-networks using a noise adaptation layer. 2016.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. Xrv preprint arXiv:2006.077:: 2020.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. arXiv preprint 1rXiv:1804.000 2018.
Jeff 7. HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self-supervised deep learning with spectral contrastive loss. arXiv preprint arXio:2106.04115 2021.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xic, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Paltern Recognation, pp. 9729 9738, 2020.
13

---

Published in Transactions on Machine Learning Research (05/2025)
Zhizhong Huang, Jumping Zhang, and Hongming Shan. Twin contrastive learning with noisy labels. In Proceedings of the IBEE/CVF Conference on Computer Vision and Pallern Recognition, pp. 11661-11670, 2023.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
John Lafferty and Larry Wasserman. Statistical analysis of semi-supervised regression. 2007.
Kibok Lee, Yian Zhu, Kihyuk Sohn, Chun-Liang Li, Jinwoo Shin, and Honglak Lee. I-mix: A domain-agnostic strategy for contrastive representation learning. arXiv preprint arXiv:2010.0880 2020.
Kimin Lee, Sukmin Yun, Kibok Lee, Honglak Lee, Bo Li, and Jinwoo Shin. Robust inference via generative classifiers for handling noisy labels. In International Conference on Machine Learning, pp. 3763 3772.
PMLR, 2019.
Junnan Li, Caiming Xiong, and Steven CH Hoi. Learning from noisy data with robust representation learning. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 9485-9494, 2021.
Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Luc Van Gool. Webvision database: Visual learning and understanding from web data. arXiv preprint orXiv: 708. 02800222, 2017.
Hong Tiu, Jeff 7. HaoChen, Adrien Gaidon, and Tengyu Ma. Self-supervised learning is more robust to dataset. imbalance. Xiv preprint mrXiv:211000502. 2021.
Sheng Liu, Jonathan Niles-Weed, Narges Razavian, and Carlos Granaad Early-learning regulariza- tion prevents memorization of noisy labels. arXiv preprint arXiv:2007.00151, 2020.
Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE Transactions on pattern analysis and machine intelligence, 38(3):447 461,, 2015.
Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah Erfani, and James Bailey. Normalized loss functions for deep learning with noisy labels. In International Conference on Machine Learning, pp. 6543 6553. PMLR, 2020.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invarianl representations. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognétion, pp. 6707-6717, 2020.
Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. Advances in neural information processing systems, 26:1196 1204, 2013.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint vrxii:1807.077/ 2018
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 1944-1952, 2017.
Senthil Purushwalkam and Abhinav Gupta. Demystifying contrastive self-supervised learning: Invariances, augmentations and dataset biases. arXio preprint wXXi:2007.13911 2020.
Philippe Rigollet. Generalization error bounds in semi-supervised classification under the cluster assumption. Journal of Machine Learning Research, 8(7), 2007.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiu preprint
2014.
Aarti Singh, Robert Nowak, and Jerry Zhu. Unlabeled data: Now helps, now it doesn't. Aduances in neural information processing systems, 21:1513 1520, 2008.
14

---

Published in Transactions on Machine Learning Research (05/2025)
Alisa Smirnova, Jie Yang, Dingqi Yang, and Philippe Cudre-Mauroux. Nessy: A neuro-symbolic system for label noise reduction. TEEE Transactions on Knowledge and Data Engineering, 2022.
Hwanjum Song, Minseok Kim, and Jae-Gil Lee. Selfie: Refurbishing unclean samples for robust deep learning. In International Conference on Machine Learning, pp. 5907 5913. PMLR, 2019.
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasckar, and Nathan Srebro. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research, 19(1):2822 2878, 2018.
Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis-Philipee Morency. Self-supervised learning from multi-view perspective. arXiv preprint or orXic:2006.0557 2020.
Yao-Hung Hubert Tsai, Martin Q Ma, Muqiao Yang, Han Zhao, Louis-Philippe Morency, and Ruslan Salakhutdinov. Self-supervised representation learning with relative predictive coding, ar preprint arXiv:2108.11991 2021.
Vikas Verma, Thang Luong, Kenji Kawaguchi, Hieu Pham, and Quoc Le. Towards domain-agnostic contrastive learning. In International Conference on Machine Learning, pp. 15500-10541. PMLR, 2021.
Haixin Wang, Huiyu Jiang, Jinan Sun, Shikun Zhang, Chong Chen, Xian-Sheng Hua, and Xiao Luo. Dior: Learning to hash with label noise via dual partition and contrastive learning. IEEE Transactions on Knowledge and Data Engineering, 2023.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity the hypersphere. In International Conference Machine Learning, pp. 9929-9939. PMLR, 2020.
Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In Proceedings of the IEEE/CVF International Conference on Compuler Vision, pp. 222-330,440 2019.
Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. Combating noisy labels by agrcement: A joint training method with co-regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp.
2020.
Xiaobo Xia, Tongliang Liu, Bo Han, Chen Gong, Nannan Wang, Zongyuan Ge, and Yi Chang. Robust early-learning: Hindering the memorization of noisy labels. In International Conference on Learning Representations, 2020a.
Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. Advances in Newral Information Processing Systems, 33, 2020b.
Tong Xiao, Tian Xia, Yi Yang. Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 2691 2699, 2015.
Eaze Xie, Jian Ding, Wenhai Wang, Xiaohang Zhan, Hang Xu, Peize Sun, Zhenguo Lú, and Ping Luo. Detco: Unsupervised contrastive learning for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3392-8401. 2021.
Ning Xu, Yong-Di Wu, Congyu Qiao, Yi Ren, Minxue Zhang, and Xin Geng, Multi-view partial multi-label learning via graph-fusteeeee fusion- based label enhancement. IEEE Transactions on Knowledge and Data Engineering, 2022.
Yihao Xue, Kyle Whitecross, and Baharan Mirzasoleiman. Investigating why contrastive learning benefits robustness against label noise. In International Conference on Machine Learning, pp. 24851-24871 PMLR, 2022.
15

---

Published in Transactions on Machine Learning Research (05/2025)
Hao Yang, You-Zhi Jin, Zi-Yin Li, Deng- Bao Wang, Xin Geng, and Min-Ling Zhang. Learning from noisy labels via dynamic loss thresholding. IEEF Transactions on Knowledge and Data Engineering, 2023.
Yuzhe Yang and Zhi Xu. Rethinking the value of labels for improving class-imbalanced learning. arXiv preprint 2rx::20006.070 07529, 2020.
Kun and Jianxin Wu. Probabilistic end-to-end noise correction for learning with noisy labels. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 2017-7025, 2019.
Li Yi, Sheng Liu, Qi She, A Ian McLæod, and Boyu Wang. On learning contrastive representations for learning with noisy labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16682-166000 16691, 2022.
Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does disagreement help generalization against label corruption? In Internalional Conference Machine Learning, pp- 7164 7173. PMLR, 2019.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. arXiv preprint (Xiu:2103.03250 2021.
Hongyi Zhang, Moustapha Cisse, Yann Dauphin, and David Lopez-Paz mixup: Beyond empirical risk minimization. arXiv preprint crXiv:1110.09441 2017.
Yikai Zhang, Songzhu Zheng, Pengxiang Wu, Mayank Goswami, and Chao Chen. Learning with feature- dependent label noise: A progressive approach. arXiv preprinl Kki:21033077666 2021.
Zhilu Zhang and Mert R Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In 32nd Conference on Neural Information Processing Systems (NeurIPS), 2018.
Zizhao Zhang, Han Zhang, Sercan Arik, Honglak Lee, and Tomas Pfister. Distilling effective supervision from severe label noise. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9294-93900 9303, 2020.
Xiangyun Zhao, Raviteja Vemulapalli, Philip Andrew Mansfield, Boqing Gong, Bradley Green, Lior Shapira, and Ying Wu. Contrastive learning for label efficient semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10623-10633, 2021.
Evgenii Zheltonozhskii, Chaim Baskin, Avi Mendelson, Alex M Bronstein, and Or Litany. Contrast to divide: Self-supervieed pre-training for learning with noisy labels. In Proceedings of the TEFE/CVF Winter Conference on Applications of Computer Vision, pp. 1657 1657-1667, 2022.
16


Reviews:
[{"review_id": "vZzzeaDJsI", "reviewer": "Reviewer_Njpr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n1. The paper addresses an important problem in machine learning - understanding why SSL works well with noisy labels, which has both theoretical significance and practical applications.\n2. The work combines theoretical rigor (e.g., Theorems 3.1 and 6.1) with algorithmic insights, offering a well-rounded contribution to the field.\n\n## Weakness\n\n1. Theoretical gaps: Equation (1) is a simplified SSL objective using linear transformations, while practical methods like SimCLR employ non-linear MLPs. This discrepancy between the theoretical model and practical implementations limits the direct applicability of the paper's insights to SSL.\n2. Experimental analysis: It would be better if the authors can discuss the computational cost or hyperparameter sensitivity of SSL methods.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a theoretical framework explaining why self-supervised learning (SSL) succeeds with label noise, especially instance-dependent noise. The authors propose that SSL mitigates noisy labels by creating uniform noise distribution and forming compact, well-separated clusters in representation space. Theoretical proofs show classifiers trained on SSL representations outperform supervised learning approaches, with provided generalization accuracy bounds. The authors enhance SSL with mixup augmentation and validate their findings through extensive experiments on CIFAR-10, CIFAR-100, ANIMAL-10N, and Clothing1M datasets, demonstrating robustness against various noise types.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Add a discussion about the gap between the SSL objective (1) and common practices in SSL methods.\n2. Add more results about the computational cost and hyperparameter sensitivity analysis.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vlBvSeJhWT", "reviewer": "Reviewer_FCwC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "S1: Section 7 provides novel insights into the loss function proposed by Wang & Isola, investigating the mean and covariance of learned clusters and how these relate to the balance between alignment and uniformity losses.\n\nS2: Empirical evaluations are very thorough. 4 datasets are used, with CIFAR-10 and CIFAR-100 being evaluated across varying types and strengths of label noise. A variety of methods are used and almost all results include some measure of variability, although to my knowledge it is not specified what this measure is and how it was calculated.\n\nS3: Sections 3-4 utilize a toy problem for label noise to show an advantage in terms of accuracy bounds for supervised learning with SSL over supervised learning alone. This is then generalized to real data via Section 5, which shows via Figure 1 that the desired clustering according to true labels does occur.\n\nW1: Lemma 8 (also called Lemma B.1?) has flaws in both sentences. As written, the first sentence is incorrect because there exists intervals such that no $\\xi$ are within them. The proof calculates the probability of this occurrence. Therefore the first sentence should be reworded to not be a statement of fact, but a statement to be evaluated. For example, “We consider the probability that the statement ‘...’ is true.”\n\nThe second sentence only holds in the case where $n = d^{10}$, so this should either be stated in the lemma or the lemma should be generalized for $n$ equal to any polynomial of $d$.\n\nW2: Theorem 3.1 (also called Theorem 1?) has 2 issues. I believe the first sentence should refer to $zeta$ and specify the differing range for the intervals.\n\n“By Lemma B.1 with probability at least $1 - 2 e^{d^{-5}}$” is technically true, but I do not see why the true probability from applying the product rule, $(1 - e^{d^{-5}})^2$, is not used instead.\n\nW3: The first paragraph of Section 7 misunderstands the relationship between prior SSL papers. Chen et al. is a precursor to He et al., Wang & Isola, and Tsai et al. Wang & Isola’s loss, given in Eq. 5, has not been widely adopted, to my knowledge, and is not cited in any of the papers used to support that claim. It is fine to focus on this loss, but its relevance should not be overstated as SimCLR variants are far more common.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper analyzes how SSL can assist with handling labeled noise for supervised learning. A toy problem is used to show how SSL compensates for label noise, leading to improved supervised learning accuracy when using frozen SSL features. This theoretical contribution motivates a comparison between supervised and SSL+supervised features, finding that SSL+supervised clusters data with respect to true labels and properly spreads noisy examples across classes. The clustering properties of the alignment and uniformity losses are then investigated, characterizing the mean and covariance of clusters with later empirical validation. Empirical evaluations investigate a variety of label noise setups on CIFAR-10 and CIFAR-100, as well as accuracy on two additional datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "C1: [MoCo v3](https://arxiv.org/pdf/2104.02057) should be discussed in the SSL related works section in addition to the original MoCo paper.\n\nC2: Where to find the proof of Theorem 1 should be stated near the end of Section 3.\n\nC3: Proposition 2’s third sentence should change “depended” to “dependent”.\n\nC4: It’s unclear whether ANIMAL-10 and Clothing-1M have label noise, so please specify in the “Datasets” part of Section 8.", "broader_impact_concerns": "I do not have any broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QtwF5HllsC", "reviewer": "Reviewer_xBbb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n- This paper aimed to theoretically analyze how self-supervised learning can address the label noise issue in machine learning, which is an important research question.\n- This paper constructed a motivating example to demonstrate the benefits of self-supervised learning.\n- This paper presented empirical evidence showing representations learned by self-supervised learning help improve robustness against noisy labels, especially instance-dependent noise.\n\n## Weaknesses\n\n- The definitions and theorems lack clarity and consistency (see below), significantly weakening the theoretical contributions.\n- The first two theoretical results (Theorem 3.1 and Proposition 4.1) are very specific to a given problem: both the data distribution and the learning methods are fixed. Usually, stronger assumptions can lead to stronger theorems but weaker generalization ability. In this case, these results cannot even be generalized to any other data. All the design choices ($[-1.75, 2.25]$, $43.75%$) and results ($\\frac{9}{16}$, $128$) are oddly specific because they are only applicable in this synthetic setting. Such theoretical results have little to no practical implications for real-world problems.\n- The author claimed that SSL works because of \"Uniform Noise Distribution: The label noise uniformly spreads over the learned SSL representation\". However, I cannot see the correspondence between theoretical results and this claim.\n- I cannot see how the analyses in Section 7 (Lemma 7.2, Lemma 7.3, and Proposition 7.4) are specific to noisy label learning. Even if they are true, they are just properties of SSL methods.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigated how self-supervised learning (SSL) could address challenges posed by label noise in machine learning, especially instance-dependent noise. Most theoretical results were for a very specific synthetic noisy label data distribution, or only for a specific SSL method (Wang & Isola, 2020) not related to noisy label learning. Some empirical evidence on image datasets showing the benefits of SSL was presented.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The numberings of theorems are not consistent: Theorem 3.1 and Proposition 4.1 in the main body; Theorem 1 and Proposition 2 in the appendix. In appendix B, Lemma 8 and Lemma B.1 are just next to each other. You can use $\\LaTeX$ packages `cleveref`, `thm-restate`, or `thmtools`.\n- The theorems are not self-contained. The whole paragraphs before Theorem 3.1 are its assumptions. The same applies to Proposition 4.1 and Theorem 6.1.\n- Section 5 is oddly short, and its claim is unclear.\n- The notation should be rechecked. $\\mathrm{Pr}[]$, $P()$, $P[]$, and $p()$ seem to be used interchangeably.\n- Definition 7.1: Why it's asymmetric? Why $P[u | x_i]$ and not $P[u | x_j]$?", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission made the first attempt to theoretically explain why self-supervised representation learning can help label-noise learning, focusing on the properties of the noise distribution in the learned representations. In the beginning, all reviewers have major concerns about the theoretical analysis; after the rebuttal, two reviewers commented that their concerns had been addressed. Specifically, two positive reviewers said the following in their official recommendations:
> This paper focus on an important problem of why SSL succeeds with label noise and combines theoretical rigor (e.g., Theorems 3.1 and 6.1) with algorithmic insights, offering a well-rounded contribution to the field. My concerns about the theoretical results and experiments are also addressed during rebuttal. Therefore, I recommend an acceptance for this work.

> The claims made in the submission supported by accurate, convincing and clear evidence and individuals in TMLR's audience interested in contrastive learning would enjoy the findings of this paper.

while the negative reviewer said
> It is unclear if the proposed theoretical results for a very specific problem are insightful in practice.

I feel that this paper fits TMLR well and we should accept it for publication. I don't know if there are still addressable concerns now (from Reviewer xBbb), so I think "accept with minor revision" may be better.
Decision: Accept (poster)