How does Labeling Error Impact Contrastive Learning? A Perspective from Data Dimensionality Reduction
Authors: Jun Chen
Hong Chen
Yonghua Yu
Yiming Ying
Venue: ICML 2025 poster (2025)
Keywords: Contrastive learning
labeling error
data dimensionality reduction
data augmentation
Primary area: deep_learning->selfsupervised_learning

Abstract: In recent years, contrastive learning has achieved state-of-the-art performance in the territory of self-supervised representation learning. 
Many previous works have attempted to provide the theoretical understanding underlying the success of contrastive learning. Almost all of them rely on a default assumption, i.e., the label consistency assumption, which may not hold in practice (the probability of failure is called labeling error) due to the strength and randomness of common augmentation strategies, such as random resized crop (RRC). This paper investigates the theoretical impact of labeling error on the downstream classification performance of contrastive learning. We first reveal several significant negative impacts of labeling error on downstream classification risk. To mitigate these impacts, data dimensionality reduction method (e.g., singular value decomposition, SVD) is applied on original data to reduce false positive samples, and establish both theoretical and empirical evaluations. Moreover, it is also found that SVD acts as a double-edged sword, which may lead to the deterioration of downstream classification accuracy due to the reduced connectivity of the augmentation graph. Based on the above observations, we give the augmentation suggestion that we should use some moderate embedding dimension (such as $512, 1024$ in our experiments), data inflation, weak augmentation, and SVD to ensure large graph connectivity and small labeling error to improve model performance.

How does Labeling Error Impact Contrastive Learning? A Perspective from Data Dimensionality Reduction
Jun Chen Hong Chen 12
Yonghua Yu Yiming Ying*
Abstract
1. Introduction
In recent years, contrastive learning has achieved of-th-of-the-art performance in the territory of self- supervised representation learning. Many previ- ous works have attempted provide the theo- retical understanding underlying the success of contrastive learning. Almost all of them rely on a default assumption, i.e., the label consisteney assumption, which may not hold in practice (the probability of failure called labeling error) due to the strength and randomness of common aug- mentation strategies, such random resized crop (RRC). This paper investigates the theoretical im- pact of labeling error on the downstream classi- fication performance of contrastive learning. We first reveal several significant negative impacts of labeling error on downstream classification risk. To mitigate these impacts, data dimensionality reduction method (e.g., singular value decompo- sition, SVD) is applied on original data to reduce false positive samples, and establish both theoreti- cal and empirical evaluations. Moreover, it is also found that SVD acts as a double-edged sword, which may lead to the deterioration of down- stream classification accuracy due to the reduced connectivity of the augmentation graph. Based the above observations, we give the augmen- tation suggestion that we should use some mod- erate embedding dimension (such as 512, 1024 in our experiments), data inflation, weak augmenta- tion, and SVD to ensure large graph connectivity and small labeling error to improve model perfor- mance.
Contrastive learning, as an emerging celf-supervise learn- ing paradigm, has achieved remarkable performance by leveraging data without label information (He et al., 2020; Chen et al., 2020c; Jang & Wang, 2023; Wang et al., 2023b; al., 2023). Typically, this learning framework entails for- mulating an auxiliary contrastive task endowed with pseudo- labels, which aims to maximize the similarity between two samples augmented from the same original sample while minimizing the similarity between samples augmented from different original samples (Chen al., 2020b)
Recently, some studies have delved into the theoretical mechanism underlying the empirical success of contrastive learning (Mikolov et al., 2013; Pagliardini et al., 2018; Jean et al., 2019; Arora et al., 2019; Jing et al., 2022; Wang et al., 2022; Lei et al., 2023). Generally speaking, they acquiesce that the labels of two augmented samples generated from the same original sample remain consistent, which is referred to as the label consistency assumption (Wang et al., 2022), Particularly, Wang et al. (2022) stated that label consistency is a natural and minimal assumption that is likely to hold in practice, and established the upper and lower bounds of downstream classification risk for contrastive learning only requiring intra-class samples have similar augmented views (called intra-class augmentation overlap, see Figure (b)). Nevertheless, given that data augmentation process is random, some strong augmentation strategies like RRC may lead to the lose of semantic- et al., 2024), which undermines label consistency and gives rise to labeling error (Tamkin al., 2023). For instance, Figure 1 (a) shows that several images augmented from dog image using RRC possess different labels (dog, ship, fog and blanket). In such a situation, it can be observed that inter-class samples also exhibit the augmentation over- lap phenomenon (see Figure (c)). This inconsistent label phenomenon (called labeling error) motivates us to develop new theoretical analysis for contrastive learning to better understand the interplay among data augmentation, labeling error, and downstream classification performance.
information (Zang
Colleee of Informatics, Huazhong Agricultural Univer- sity, Wuhan, China 'Enginerrigg Research Center of Intelli- gent Technology for Agriculture. Ministry of Education, China College of Engineering. Huazhong Agricultural University, Wuhan, China "School of Mathematics and Statistics, Univer- sity of Sydney, NSW, Australia. Correspondence to: Hong Chen <chennh @ mail hzw..cdu.cn>.
Recently, the analysis framework of HaoChen et al. (2021) pioneered the consideration of the labeling error caused by data augmentation, which revealed the first dependencies
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

How does Labeling Error Impact Contrastive Learning? A Perspective from Data Dimensionality Reduction
Dog
Ship
Augmentation
Augmentation
Fog
Blanket
(a) Different Labcls
(b) Intra-class Overlap
(c) Inter-class Overlap
Figure 1. (a) Four images augmented from a single dog image using RRC have different labels (dog, ship, fog and blanket). (b) Augmented views of different intra-class samples may overlap. For instance, views labeled "dog" might be augmented from different dog images. (c) Augmented views different inter-class samples may also overlap. For instance, two views labeled as "ship" might be augmented from a dog image and ship images, respectively.
on labeling error and the connectivity of the augmentation graph (Equation (2) HaoChen al. (2021)) for the down- stream linear probe error. While matching the experimental observations, they didn't offer any suggestions to reduce the dependency on labeling error. Subsequently, considering that generated data may sometimes even harm contrastive learning, Wang et al. (2024) formulated novel analysis strategy to explore the reasons underlying this labeling error phenomenon from the perspective of data inflation !. Theo- retical and empirical results in Wang al. (2024) illustrated that stronger data inflation would bring larger graph connec- tivity, which decreases the upper bound of the downstream linear probe error. They also validated that stronger data augmentation would affect the trade-off of labeling error and graph connectivity to bring the phenomenon that model performance first rises and then falls. Consequently, they suggested obtaining large graph connectivity along with small labeling error through the combination of strong data inflation and weak augmentation. However, good down- stream performance is primarily ensured by large graph connectivity. They didn't verify whether the labeling error caused by the weak augmentation is sufficiently small.
overlap. From the data dimensionality reduction perspective, our theoretical and empirical analyses validate the effective-
ness of some useful suggestions for decreasing labeling enror to enhance downstream classification performance. Our main contributions are summarized as follows.
Theoretical guarantees of classification risk for both intra-class and inter-class augmentation overlap cases. Beyond the intra-class augmentation overlap consid- ered by Wang et al. (2022), we establish the first upper and lower bounds of downstream classification risk for inter-class augmentation overlap case. It is discovered from these bounds that there several significant de- pendencies associated with labeling error. Following the above analysis. a new perspective of data dimen- sionality reduction is introduced to reduce these depen- dencies. As illustrated by singular value decomposition (SVD), both theoretical analysis and empirical obser- vations demonstrate that this dimensionality reduction technique is capable of suppressing classification risk by diminishing labeling error.
Theoretical guarantees of classification error while us- ing SVD. Except for downstream classification risk, we provide an upper bound classification error while using SVD. This bound uncovers that SVD can also reduce classification error by decreasing labeling error. Besides, SVD might lead small graph connectivity, thereby potentially worsening this bound. We sug- gest that adopting moderate embedding dimension contributes to the decrease of classification error by
In this paper, we investigate the negative impacts of labeling error. Specifically, we first provide the theoretical upper and lower bounds of downstream classification risk and error considering both intra-class and inter-class augmentation
'Data inflation defined by Wang al. (2024) the process of using generative models (e.g.. denoising diffusion probabilistic model (DDPM) (Ho et al., 2020)) to generate lot synthetic samples. Wang et al. (2024) performed contrastive learning on the combination of the real and generated data.
2 "Classification error measures the proportion of misclassified samples. Thus, the summation of classification error and accuracy equal to 100%.
'The word "moderate" means that the embedding dimension k is not small too large. In experiments, 512 1024 may be suitable.
2

---

How does Labeling Error Impact Contrastive Learning? A Perspective from Data Dimensionality Reduction
increasing graph connectivity, which is validated by our data experiments.
2020; Barbano al., 2023) and weak supervised progresses (Zheng et al., 2021; Tsai et al., 2022, Cui et al., 2023) that combine available label information to guide model training.
Some useful suggestions guided by our analyses. Under the experiment setting of Wang et al. (2024), our empir- ical evaluations indicate that their weak data augmen- tation still brings an unignorable labeling error. The aforementioned analyses suggest that we should use moderate embedding dimension, data inflation, weak augmentation, and SVD to achieve two critical ob- jectives: ensuring large graph connectivity and small labeling error, thereby enhancing the downstream clas- sification accuracy.
3. Preliminaries
This section initiates our analysis with a comprehensive overview of contrastive learnin.. We first denote D the finite but exponentially large set set of all unlabeled original sample E E R and denote y : D - [K] the ground-truth (deterministic) labeling function, then ys E (1.....k) Let P be the population distribution of the original samples. Contrastive learning initializes and trains the model parame- ters in an unsupervised fashion, laying a foundational under- standing of the underlying structure and patterns of original data. The subsequent supervised fine-tuning stage further adapts the pre-trained model parameters to some specific downstream task on the test data drawing from 'P to enhance model performance.
2. Related Works
Theoretical Understanding of Contrastive Learning. Ex- cept for the above theoretical studies, there are also many understanding of contrastive learning from other perspec- tives (Tian et al., 2020; Chen et al., 2020a; Zimmermann et al., 2021; Saunshi et al., 2022; Waida et al., 2023; Ji et al., 2023; Zhang et al., 2023b; Zou& & Liu, 2023; Wen et al., 2024). From the perspective of information theory, Tian et al. (2020) theoretically and empirically showed that data augmentation can decrease mutual information and improve downstream classification accuracy. Saunshi al. (2022) empirically presented that different function classes and algorithms bring different behaviors and suggested the consideration of inductive biases in theoretical analysis. Ji et al. (2023) proved two conclusions: I) contrastive learning outperforms some other supersuper learning paradigms such as autoencoder and generative adversarial network; 2) label information benefits in-domain downstream task and harms out-domain downstream task. Different from previ- ous theories on data augmentation (Dao et al., 2019; Rajput et al., 2019; Wu et al., 2020), Chen et al. (2020a) pioneered connection between data augmentation and the performance of downstream task using group theory under approximate equality condition (similar to label consistency). These works all hinge upon the label consistency assumption as a fundamental premise. In contrast, our analysis is free from the label consistency assumption.
For the first learning stage, an encoder projector f E Fi Rd + gk is pre-trained to map a d-dimensional input vec- tor T to an embedding vector 2 = fxx in a k-dimensional unit hypersphere, where the inequality k < d generally holds which is validated by our experiments. This pre- training process is composed with several steps including data augmentation, contrastive representation, and loss cal- culation. Due to the unavailability of true label, contrastive pre- pre-training constructs a surrogate task via some sample aug- mentation strategies. Then, this task is learned by minimiz- ing the distance between similar samples and maximizing the distance between dissimilar samples in embedding space. For example, we first select an original sample T. and make data augmentations t, t, E T(T = {t|t Rd + R to obtain two independent augmented samples x = 2 t (i). respectively. We use p(- ( to denote the distribution of the augmented sample for J. Then, M negative sam- ples x; , I,.., Ad, are randomly augmented from other original samples. We assume x. x+ x= follow the same population distribution, which is the same as previous work like Wang et al. (2022). We use these augmented samples to calculate the most common contrastive loss, InfoNCE loss (van den Oord et al., 2018), defined as follows
Applications of Contrastive Learning, Contrastive learn- ing has achieve the empirical success in many fields, includ- ing time series prediction (Nonnenmacher et al., 2022; Lee et al., 2024; Xu et al., 2024; Zheng et al., 2024), graph learn- ing (Xia et al., 2022; Ghose al., 2023; Lin et al., 2023: Yu et al., 2023b; Liu et al., 2024), federated learning (Yu et al., 2023a; Louizos et al., 2024), multi-modal learning (Lin et al., 2022; Wang et al., 2023a; Huang et al., 2023a; Fang et al., 2023; Xia et al., 2023), adversarial learning (Xu et al., 2023a; Zhang et al., 2023a; Xu et al., 2023b; Luo et al., 2023), etc. Not only self-supervised contrastive leam- ing, there are also some supervised works (Khosla et al.,
LinfoNCE(f)
(x)) ((x)T((>') f(xt)
-E n.a +..t.(7770
log exx) s(at) +M ( M(x7)
In InfoNCE loss, the similarity between two samples is quantified by the inner product f(x) f(x') of two vectors (a' denotes any sample in 2t, ...., MM) Considering that f is a projector mapping from Rd to a k-dimensional unit hypersphere, the inner product f(x) f(x) represents
3

---

How does Labeling Error Impact Contrastive Learning? A Perspective from Data Dimensionality Reduction
cosine similarity. To sum up, minimizing InfoNCE loss is equivalent maximizing the similarity of positive pair and minimizing the similarity of negative pairs. Generally, we consider the empirical version of InfoNCE loss
and the quality of the learned representations. This study primarily focuses on the potential labeling error caused by false positive augmented samples (Assumption 3.1).
Assumption 3.1 (Labeling Error (Wang al., 2024)). For any I P, its latent label Va. and its augmented sample x ~ p(-[z), we assume that the true label of x: is not con- sistent with ya with the probability E (0, 1). That is, E E [I [y + yall a.
LinfoNCE(f)
(zx)) f(x))
log
(2)
S(a)) + Em= ((,)
1=
and its minimizer
4. Main Results
E' e arg min LinsoNCE(f). IEFi
4.1. Theoretical Impact of Labeling Error
Definition 4.1 (Augmentation Overlap). Given collection of augmentation strategies T, we say that two original sam- ples ह, E' ~ P are T Taugmentatiion overlapped if they have overlapped views, i.e., 3t, l' E T such that ((z) = ('(x'.
For the second learning stage, the parameters of the pre- trained encoder remain unaltered. We retrain linear projec- tion head g E F2= R$ - .RK with the weight W E R using the cross entropy (CE) loss conduct downstream classification task. For labeled sample ~ P, the CE loss and the mean CE loss 4 are calculated via
In the analysis of Wang et al. (2022), they proposed the concept of augmentation overlap. Owing to the label con- sistency, the positive augmented sample pair (æ, x+) has the same label yr yu+. Therefore, they analyzed the model performance in the case that different intra-class samples could have overlapped augmented views (Figure (b)). While under Assumption 3.1, there arises the phe- nomenon that different inter-class samples may also exhibit overlapped augmented views (Figure (c)). Our first theo- rem (Theorem 4.2) takes these two augmentation overlap cases into account (the proof can be found in Appendix C).
exp (f(x) wyy)
CCE(DR.) = Ez log
(3)
Σ exp
K
and
exp (f(x)] My.))
LcE(95,)) = Ex,
log
(4)
E expl (f(x) [4))
where the linear classifier predicts 9J.W arg | (FTW), W = [wi, ..... wk], and the param- eter of mean projection head is μ = [uI...,,44 whose
Theorem 4.2 (Bounds of Mean Classification Risk). Let Assumption 3.1 hold. For any f E Fi,g E Fz, the gap between the mean downstream classification risk and the contrastive risk fCE(gga)) LinfoNCE(() can be upper bounded by
element μ4 denotes the mean of the representations for the inputs with the label i E [K]. i.e., 14, = B( [x)] Define the downstream classification error as
E(S, W) = Pr (or, (x) + ya]-
(5)
VV((())) + VV-
O ( - logg
M
x>p
K
This work uses linear probing in fine-tuning stage, rather than full fine-tuning which updates all model parameters.
and lower bounded by
As previously stated, contrastive learning employs data augmentation methods to make preparation for unsuper- vised pre-training. There are two default assumptions: 1) the positive samples pair (x, have the same label, i.e., y> Ux : 2) any negative sample x; has a label differ- ent from Vr- However, the inherent randomness of both traditional data augmentations and negative sample sam- pling may violate these two assumptions in practice. As a consequence, the pre-training process may be led astray, po- tentially undermining both the overall training effectiveness
VV([(x)) VV- (f(x))
1 V(((()))
2
- (M-+)) -log
(M+1)
( (M+1) +1)
K
where V(((x) = [(xxt)e
+ [1((x) - 1/y 112
V- (f(x)) = B(a,a)ex-- [[[((zt) (({xt=- - -Myy
v(((=)) = v(zl= E (s(x)), (((x),,(e)) )), N1+t
yz) = (( (.s( (0M,+++,,,- [| |1al1 5
"Mean classifier was first considered by Arora et al. (2019). They stated that the mean classifier could achieve comparable performance to learned weights. Note that we don't use the mean classifier in our experiments. It is only available an intermediate result (Theorem 4.2) in our theoretical analysis.
are the intra-class variance of the representations for true positive augmented samples, the variance for
'1y2 and My, denote the mean of these representations with the same label as sample x and embedding 2, respectively.
4

---

How does Labeling Error Impact Contrastive Learning? A Perspective from Data Dimensionality Reduction
false positive augmented samples, and the intra-class variance for negative and true positive augmented samples, respectively, X = ((za=))z = Vut 3 and x = {(xxxx²) + Yat ) denote the sets of true positive sample pair and false positive sample pair, respectively.
To sum up, Theorem 4.2 provides more detailed and prac- tical bounds of the mean downstream classification risk (Cr(grw) under milder assumption compared to the re- sults in Wang et al. (2022). Our analysis indicates that the labeling error a exerts negative impacts on the theoretical downstream classification risk. Thus, we aim to put forward some suggestions to mitigate these impacts.
From the above results, it can be observed that there are mul- tiple terms determining the upper and lower bounds of the mean classification risk (ge(gr.), i.e., 1) LInfoNCE( (f): the contrastive risk; 2) V (f(x)): the intra-class variance of
4.2. Dimensionality Reduction as A New Perspective
This subsection investigates the impact of labeling error on contrastive learning through a new perspective of data dimensionality reduction. Specifically, we use data dimen- sionality reduction individual original sample to obtain the corresponding compressed sample which is then aug- mented to pre-train the model. We take Singular Value Decomposition (SVD, Wall et al. (2003)) as an illustrative example. For ease of calculation, the randomized SVD (RSVD, Halko et al. (2011)) algorithm is employed in our experiments.
the representations of true positive augmented samples; 3) VT (f(z)): the variance of false positive augmented sam- ples; 4) V)f(==) = V(=|> E {f(xz) ff )} Va+ = Va): the intra-class variance of negative and true positive augmented samples; 5) O >) the order of the ap- a constant depending on M and K. In terms of the form of these bounds, our re- sult is more refined than that of Theorem 4.2 in Wang et al. (2022) as evidenced by these terms V (f(z)) V (f(x), and (f )). It should be noted that the assumption of our Theorem 4.2 is milder than the corresponding result in (Wang et al., 2022), which leads to some latent differences in these terms. We carry out the following analyses with respect to these terms.
proximation error; 6) log MM
Definition 4.3 (SVD). For a matrix X E Rmxm' (without loss of generality, we let m S m²)) its SVD is expressed as X = USVT, where U = [ul,..., u.m] E RV [UI...., Un/ E Rm xm" " is the left (right) singular ma- trix, consisting of mnmm) orthonormal column vectors (eigenvectors of X XXT(X" (XTx)()))))) S = [diag(s), ..., sm), O] is composed of a diagonal matrix diag(s),... Sm) E R= and zero matrix 0 with size m (m' m), the values si 1,..., ..) are the singular value, arranged in descending order such that 2 82 2 ... 2 8m 2 0.
(1) V (((r)): Although this term looks like the variance term in the bound of Theorem 4.2 for Wang et al. (2022),
its definition (ex++ [l)
11] demonstrates
the clustering property of true positive sample pair, not including false positive sample pair. If the value of a eqals to zero, the variance term in Wang et al. (2022) will be equivalent to our V (f()),, as shown in Figure (a).
In general, we use the truncated version of SVD which is ex- pressed as X, = U,S.V.T. where U, = [u1...., Mg] E Rmxx,, Sq = diag(si,... sq), Vo = [Ul...., Uiyl RT ne''X9gg E [m]. The data distribution after conducting truncated SVD is written as Pgg then the corresponding labeling error is ag = EE ..Eppp..)p(()1)) [Lly + y21]. The fol- lowing lemma provided by Eckart & Young (1936) proved that Xg the best least squares lower rank approximation with rank q for given matrix X.
(2) V- (f())): Wang et al. (2022) mistook false posi- tive augmented samples for true positive augmented sam- ples, consequently resulting in a large conditional posi- tive intra-class variance. Upon revisiting the definition (f)))], [z)], these false positive augmented
My, = E samples do not have an impact on My.- When we discard these false positive augmented samples, the remaining posi- tive augmented samples appear more concentrated around Mya in comparison to those of Wang et al. (2022). There- fore, as the value of a decreases, this term will progressively decrease until it reaches O.
Lemma 4.4 (Eckart- Young Theorem). Let X be x m' matrix of rank r E [m] with complex elements. Define Pa as the set of all m x ml matrices with rank g E [r). Then,
|X xx-x SI1X B MIXXBBIp,BBC YBE
(3) V (((xT) )): Considering the sampling randomness and the unavailability of the true labels for negative samples, we let the label yー denote the latent label of the negative augmented sample x=, irrespective of the class to which its original sample h- pertains. Therefore, compared with V- (f(z)), this term remains unaffected by the labeling error a. We provide Figure 2 to facilitate the understanding of the difference between the analyses of V (f))) and
As stated by Kilmer et al. (2021), Lemma 4.4 implies that the majority of the informational content is captured by the dominant singular subspaces, i.e., the span of the singular vectors corresponding to the largest singular values. By de- fault, we assume a positive correlation between the amount of information and its semantic relevance. It follows that the information content corresponding to the largest singular value 81 should represent the most crucial information in
V (f(x)))
5

---

How does Labeling Error Impact Contrastive Learning? A Perspective from Data Dimensionality Reduction
a
(a) Positive Augmented Samples
(b) Negative Augmented Samples
Figure 2. The impact of the labeling error the positive (a) and negative (b) augmented samples. Brown dots, blue dots and grey dots denote the false positive samples, true positive samples and true negative samples, respectively. Blue arrows denote the decrease of a.
Table Downstream classification lop-1 accuracies (%) of SimCLR (LInfoNCE) on CIFAR-IO 10 using truncated SVD which discards two singular values ($t.441 denotes discard the i-th and the + 1-th singular values 8,, 8 via SVD, Ti {Random resize crop (RRC), Color jitter, Random horizontal flip, Random grayscale, Gaussian blur), bold number indicates the optimal performance of each experimental group).
T
Encoder
$1,2
$2.3
83,1
$4,5
$5.6
$6,7
$7,8
$8,9
Ti
Resnet-18
57.31
64.14
65.63
67.20
66.93
67.77
67.93
68.44
T
Encoder
$9,10
$10,11
$11,12
$12,13
$15,16
$21,22
$31,32
T1
Resnet- 18
67.97
68.08
68.59
68.96
68.61
69.20
69.48
Table 1 and Figure empirically validate that the larger the singular value the more semantic-related information it encompasses. Figure 4 visually presents the differences between original samples and the compressed samples.
Let sample and its corresponding sample after applying SVD be represented as the matrices X and sin- X,, E mm respectively. Assume that there gular values associated with semantic-related information. When q 2 q², under Assumption 3.1 and the augmentation collection T. the true label of the augmented sample of X, is inconsistent with the latent label of with the probabil- ity 0g I a. Conversely, when q < q", the corresponding probability satisfies 0zg Ot--
Assumption 4.5.
512
50.3
56A
5655
St.t St.11
Rt..2 Ra.13
StLae Stu.2
Figure 3. Curve of classification top-1 accuracies (%).
The truncated SVD with q = q" is defined as the optimal truncated SVD. As indicated by Assumption 4.5, we assume the trend where a initially decreases and then increases as the decrease of q. In the first stage (q 2 q²), the decrease of derives from the removal of semantically irrelevant infor- mation. Once q < q", semantic-relate information begins to be discarded, giving rise to the increase of a. Table and Figure (Appendix G) present empirical performances applying the truncated SVD with different values g, align- ing with our analysis. Notably, the values of q vary across different settings. Table further validates the effectiveness of SVD across various augmentation strategies
BOSAL
Figure 4. Examples from STL-10. The first row shows the original images and the second shows the images after taking SVD.
X. For example, an image is intrinsically represented as a matrix X. The information associated with 81 most significant for distinguishing the true semantic of the image.
"Explantiions of these augmentation strategies in Table provided in Appendix
6

---

How docs Labeling Error Impact Contrastive Learning? A Perspective from Data Dimensionality Reduction
Table Downstream classification top-1 accuracies (%) SimCIR (LInfoNCE) using the truncated SVD with different truncated
parameter g.
T
Encoder
Dataset
w/o SVD
9
30
9
25
20
15
q
10
1
Resnet-18
CIFAR- 10
68.82
69.48
69.75
69.87
69.01
68.26
7
Resnet-50
CIFAR-10
63.20
63.36
63.96
62.23
60.97
60.06
RRC
Resnet-18
CIFAR-10
58.56
58.83
58.67
58.61
58.54
58.32
T
Resnet-18
CIFAR-100
38.48
38.81
40.10
39.05
38.98
38,10
T
Encoder
Dataset
w/o SVD
9
90
9
70
9
50
9
30
q 10
T
Resnet-18 STL-10
71.54
73.12
72.29
71.10
70.04
67.52
Table Downstream classification top-1 accuracies (%) of SimCLR (CInFONCE) on CIFAR-1O using the truncated SVD with different augmentations (Th = (Ti+ Cutout]: T- = {RRC, Cutout, Hide patch;; Ta = {RRC, Cutout, Color jitter): T5 = {RRC, Cutout]:
To {RRC(0.08, 0.5), Cutout): {RRC(0.08, 0.5), Cutout(O.5, 1.0))).
SVD
Encoder
T2
T
T4
Ts
Ta
T7
RRC(0.08,0.5)
w.o. SVD
Resnet-18
62.90
50.53
60.00
56.67
54.97
54.09
57.11
q = 30
Resnet-18
64.86
51.00
61.57
57.85
55.69
54.75
58.10
Due to the unavailability of q², we select a relatively large value of q, specifically q = 30 for CIFAR-I0 (image size is 32 32) in Table 3. Even though we merely discard the in- formation corresponding to the two smallest singular values 831, 832, the results of multiple augmentation strategies in Table 3 all exhibit some non-negligible improvements, with an average increase of 0.97%.
s(+t|| S egr Seg.
Theorem 4.7, stated as below, is derived based on Assump- tion 4.6, which theoretically clarifies the effectiveness of data dimensionality reduction.
Theorem 4.7. Given the conditions of Theorem 4.2 and As- sumption 4.6, after taking the truncated SVD on DD the mean downstream classification risk LCE(gs. LinfoNCE' (f) with the encoder I can be upper bounded by
Further, we introduce an assumption related to SVD, i.e., ( aalggmmett for any false positive sample pair.
EqT + Eq +0 + (M-3) (M(M))
M K
Assumption 4.6 ((Eg,, k AAligmment for Any False Pos- itive Sample Pair). Let Assumptions 3.1, 4.5 hold and X- = =(xx, "yy= * }. When performing SVD with the truncated value q. the encoder ʃ with the empirical InfoNCE loss LInfoNCE(f) (2) align any false posi- tive sample pair (x, X-, such that their distance in the embedding space lies within (e(ag)) ), e(ag)), i.e., ((ag)) = min lIf(x) - f(x+)] and e(Or)) = (ax(t,t)) [f(x)) f(*))]l .. For simplicity, let Eg- =
-log
and lower bounded by
cr- Co 2 - (M- -log
M 1
K
When 9 =q, the two bounds are Ey. + O (M-+)
log k and Cop V (M-#) - log k
e(arr)) 6, =()g))
4.3. Further Understanding of Labeling Error
Intuitively, labeling error can affect the relationship between the embedding vectors of positive augmented sample pair. Thus, we introduce the (Egr; Ex)llimmmenm assump- tion, which is inspired by the weak alignment definition proposed by Wang et al. (2022) (Definition 1 in Appendix B.I). Furthermore, Assumption 4.6 means that the encoder f. optimized under the loss (InfoNCE(f), can align any (z, xt) E (tr, = ya++ } with the distance at most Er in embedding space, that is, max +ex (xr)
Next, we will delve deeper into the theoretical impact of SVD on the downstream classification error E(S, W) in (5) through the lens of spectral graph theory. In the subsequent analysis, the spectral contrastive loss Lape (f) is adopted,
Lupe (f)
2 [f(x] f(x*)] + E,,,r- [f(x)] f(x-))
7

---

How does Labeling Error Impact Contrastive Learning? A Perspective from Data Dimensionality Reduction
Table Downstream classification top-1 accuracies (%) SimCLR (Cepe) on CTFAR-IO using truncated SVD with different or the data inflation strategy under the weak data augmentation adopted by Wang et al. (2024) (7s {RRC(0.2, 1.0), Color jitter(O.., 0.4). Random horizontal flip, Random grayscale, Gaussian blur}).
T
Encoder
Inflation
w/o SVD
q -= 30
25
20
q 15
q 10
Ts
Resnet-18
71.54
71.21
71.64
71.65
71.11
70.41
67.83
T
Encoder
Inflation
Inflation (g = 30)
Inflation + (g = 25)
Inflation + (g 20)
7s
Resnet-18
71.54
71.64
72.55
71.19
Table Downstream classification top-1 accuracies (%) SimCLR (CInfoNCE) on CIFAR-10 using the truncated SVD with different q or the data inflation strategy under the weak data augmentation adopted by Wang et al. (2024) (Ts (RRC(0.2, 1.0), Color jitter(O..5, 0.4), Random horizontal flip, Random grayscale, Gaussian blur)).
T
Encoder
Inflation
w/o SVD
q q= 30
25
q= = 20
q
15
q -= 10
Ts
Resnet-18
70.87
70.11
70.92
70.67
70.97
70.41
69.03
T
Encoder
Inflation
Inflation (g = 30)
Inflation + (q = 25)
Inflation + (g 20)
Ts
Resnet-18
70.87
70.98
71.21
70.48
which is proposed in HaoChen et al. (2021) and similar to the InfoNCE loss. The augmentation graph, defined as below, is involved in our analysis.
analysis framework, Wang et al. (2024) suggested using strong data inflation and weak data augmentation to guaran- tee a small value of a and a large value of Ax+1. We aim to replace the complex data inflation with SVD.
Definition 4.8 (Augmentation Graph, HaoChen et al. (2021)). Given an augmentation collection T, there exist n augmented samples that form the augmentation dataset
Our experimental results in Table 4 and Table demonstrate that the benefit of employing SVD can catch up with that of data inflation. Moreover, we theoretically provides the upper bound of the downstream classification error under the application of the truncated SVD with the hyper-paramete q.
D Daug = [a| = a,,ET=
An augmentation graph G is obtained by taking the n aug- mented samples as the graph vertices and assuming there exists an edge between two vertices z, x' E D Daug (if they P).
Theorem 4.9 (Bounds of Classification Error). Let Assump- tion 3.1 hold. For the empirical optimal encoder f taking the truncated SVD with hyper-parameter q on unlabeled original dataset, there exists linear head W E RR with norm [|* 1/(1 >kkg) such that
can be generated from a random original sample
According spectral graph theory, we define A guxn as the adjacency matrix of the augmentation graph G. For two augmented samples x. E 'Duug. the element A(x, x²) denotes the marginal probability of generating a, x' from a random original sample F ~ P. Formally, A(z, x') = Ez~p ((=]))p(<'|a)) [z)]. The corresponding nor- malized graph Laplacian matrix is L = 1 - D- AD- where D denotes a diagonal degree matrix with the diagonal element Dz,,= = Ev'D,.., A(x, x²). The eigenvalues of T, denoted {A, 1. where 0 \1 S ... S A, 2.
40g
(US,,")+
80kg1
Ak+1.g
where l denotes the dimension of embedding space and Ak+1, denotes the k 1-th eigenvalues of Liq.
Note that there are two differences between our Theorem 4.9 and Theorem C.3 of HaoChen et al. (2021): 1) we take the truncated SVD on original samples; 2) we evaluate the classification error for the linear classifier 9y.w rather than the majority voting classifier of HaoChen et al. (2021).
HaoChen et al. (2021) first established an upper bound related to and AR+1 for the downstream classifica- tion error of the majority voting classifier (),w(a) = Pr (g..(x)) i]. Building upon their arg max (x)(--1))
Wang et al. (2024) concluded that stronger data inflation solely enhances graph connectivity (larger A without affecting labeling error. When the graph connectivity is sufficient, only weak data augmentation necessary to
When taking the truncated SVD with hyperparameter the unlaheled original sample the corresponding symbols Ggs Aq. Tog and d E [n].
8

---

How does Labeling Error Impact Contrastive Learning? A Perspective from Data Dimensionality Reduction
Table Downstream classification top-1 accuracies (%) of SimCI.R (Lope) using the truncated SVD (g = 30 for CIFAR-10 and CIFAR-100 y 90 for STL-10) with different embedding dimension k.
Embedding Dimension
T
Encoder
Dataset
k = 128
k = 256
k == 512
k 1024
k =
2048
T
Resnet-18
CIFAR-I0
67.71
68.51
68.54
69.09
68.65
1
Resnet-50
CIFAR-10
67.43
63.99
66.50
66.83
66.22
T
Resnet-18 18
CIFAR-100
35.00
36.68
36.78
37.78
37.18
T
Resnet-50
CIFAR-100
35.46
35.42
35.39
35.59
35.53
T
Resnet-18
STL-10
72.35
72.42
73.12
73.88
73.47
T Resnet-50 STL-10
74.68
74.94
75.01
76.26
75.57
achieve a small labeling error. However, Table 4 and Table 5 reveal that the weak augmentation employed by Wang negliggli labeling error.
reasons for future work. In this paper, we propose selecting moderate value of k, such 512 or 1024. Additionally, recommend adopting data inflation similar to that in Wang et al. (2024) to mitigate the negative impact of SVD on X, which is validated by the experiments in the second rows of Table 4 and Table 5.
et al. (2024) still induces Compared with Theorem 4.1 in Wang et al. (2024), Theorem 4.9 further introduces SVD to offer an improvement by reducing a to Og-- In the following, we will analyze the impact of SVD on Oi and AR+1.q involved in Theorem 4.9.
In summary, an augmentation strategy proposed and vali- dated to ensure large graph connectivity and small labeling error, where the key building blocks include using moderate embedding dimension, data inflation, weak augmentation and SVD.
(1) agz As discussed behind Assumption 4.5, ag exhibits trend of first decreasing and then increasing the decrease of q E [m]. The value of Og reaches its optimal value Oxg' when 9 = =q"
(2) Agt1.z As mentioned earlier, {>i)== are the eigenval- ues of the normalized graph Laplacian matrix L. Based on the definition of T,, it can be known that {1 > are the cigenvalues of the adjacency matrix A, where 1 AR+1 the k 1-th largest eigenvalue. The definition of A(x, x²) demonstrates that the implementation of SVD on origi- nal samples will not lead to a decrease in the elements on the main diagonal of A. In other words, the trace tr(A) is non decreasing and remains less than 1, formally, tr( tr(A)= = Σx= (1 )r.g) S Therefore, the application of SVD may lead to Ax+1. S A deteriorating the bound of Theorem 4.9. It appears that choosing a large dimension k of embedding space may alleviate this deterioration much as possible by increasing graph connectivity. When the value of k is not sufficiently large, such as k < 512, the corresponding results in Table may reveal this trend. However, Table also indicates that an overly large value of h might cause the degeneration of the downstream classi- fication performance. Interestingly, similar optimal values of k have been observed across experiments on different datasets (CIFAR-10 10, C00 and STL-10) and different encoders (Resnet-18 and Resnet-50). Experimental results with InfoNCE loss (Table 8 in Appendix G) also reveal this phenomenon. Therefore, there are still some results that do not align with our analysis. As stated in Limitation (Appendix F), leave the exploration of the underlying
5. Conclusions
This paper investigated theoretically the impact of labeling error on the downstream classification performance of con- trastive learning. The derived upper and lower bounds of the downstream classification risk are both affected by the labeling error. To mitigate these negative impacts, we first propose removing the semantically irrelevant information of the original data from the perspective of data dimension- ality reduction. Specifically, the classical SVD method is employed to offer both theoretical and empirical evidence to support the effectiveness of dimensionality reduction. Except for the advantages of conducting SVD the orig- inal data, we also theoretically find that SVD may cause the deterioration of downstream classification accuracy by decreasing the graph connectivity of augmentation graph. Based on the aforementioned analysis, we provide an aug- mentation strategy that we should use moderate embedding dimension (such as 512, 1024), data inflation, weak augmentation and SVD to ensure large graph connectiv- ity and small labeling error, ultimately improving model performance.

---

How does Labeling Error Impact Contrastive Learning? A Perspective from Data Dimensionality Reduction
Impact Statement
Eckart, C. and Young, G. The approximation of one matrix by another of lower rank. Psychometrika, 1:211-218, 1936.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Fang. Q. Zhang, X., Hu, J., Wu, X., and Xu, C. Contrastive multi-modal knowledge graph representation learning. IEEE Transactions Knowledge and Data Engineering, 35(9):8983-899) 2023.
Acknowledgments
This work is supported by the National Natural Sci- ence Foundation of China (NSFC) (Nos. 62376104 and 12426512) and the Open Research Fund of Engineering Research Center of Intelligent Technology for Agriculture, Ministry of Education (No. ERCITA-KFOR KFO02).
Ghose, A., Zhang. Y., Hao, J., and Coates, M. Spectral augmentations for graph contrastive learning. In Interna- tional Conference on Artificial Intelligence and Statistics (AISTATS), volume 206, pp. 1 11111-122000 2023.
Halko, N., Martinsson, P., and Tropp. J. A. Finding structure with randomness: Probabilistic algorithms for construct- ing approximate matrix decompositions. SIAM Review, 53(2):217-288, 2011.
References
Arora, S., Khandeparkar, H., Khodak, M., Plevrakis, O., and Saunshi, N. A theoretical analysis of contrastive unsupervised representation learning. In International Conference on Machine Learning (ICML), volume 97, pp- 5628-5637, 2019.
HaoChen, J. Z., Wei, C., Gaidon, A., and Ma, T. Provable guarantees for self-supervised deep learning with spec- tral contrastive loss. In Advances in Neural Information Processing Systems (NeurIPS), pp. 5000-50000000 5011, 2021.
Barbano, C. A., Dufumier, B., Tartaglione, E., Grangetto, M., and Gori, P. Unbiased supervised contrastive learning. In International Conference on Learning Representations (ICLR), 2023.
He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. B. Mo- mentum contrast for unsupervised visual representation learning. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9726-9733 2020.
Budimir, I., Dragomir, S. S., and Pecaric, J. Further reverse results for jensen's 's discrete inequality and applications in information theory. Journal of Inequalities in Pure & Applied Mathematics, 2, 2000.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion prob- abilistic models. In Advances in Neural Information Processing Systems (NeurlPS), 2020.
Huang, R., Wu, S., Jie, L., Zuo, X., and Zhang, H. Siamelim: Text-based pedestrian search via multi-modal siamese contrastive learning. In IEEE International Conference on Image Processing (ICIP), pp- 1800-1804, 2023a.
Chen, S., Dobriban, E., and Lee, H. A group-theoretic framework for data augmentation. Journal of Machine Learning Research, 2 21(245):1-11, 20220 2020a.
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. E. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning (ICML), volume 119, pp. 1597-1607, 2020b.
Huang, W., Yi, M., Zhao, X., and Jiang, Z. Towards the generalization of contrastive self-supervised learning. In The Eleventh International Conference on Learning Rep- resentations (ICLR), 2023b.
Jang, T. and Wang. X. Difficulty- based sampling for debi- ased contrastive representation learning. In Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14039-24048, 2023.
Chen, T., Kornblith, S., Swersky, K., Norouzi, M., and Hinton, E. Big sule-supervised models are strong semi- supervised learners. In Advances in Neural Information Processing Systems (NeurIPS), 2020c.
Jean, N., Wang. S., Samar, A., Azzari, G., Lobell, D. B., and Ermon, S. Tile2vec: Unsupervised representation learning for spatially distributed data. In The Thirty- Third AAAI Conference on Artificial Intelligence (AAAI), pp. 3967-3974, 2019.
Cui, J., Huang, W., Wang, Y., and Wang, Y. Rethinking weak supervision in helping contrastive learning. In In- ternational Conference on Machine Learning (ICML), volume 202. pp. 6448-6467, 2023.
Dao, T., Gu, A., Ratner, A., Smith, V., Sa, C. D., and Ré, C. A kernel theory of modern data augmentation. In International Conference on Machine Learning (ICML), volume 97, pp. 1528-1537, 2019.
Ji, W., Deng, Z., Nakada, R., Zou, J., and Zhang, L. The power of contrast for feature learning: A theoretical anal- ysis. Journal of Machine Learning Research, 24:330:1 330:78, 2023.
10

---

How does Labeling Error Impact Contrastive Learning? A Perspective from Data Dimensionality Reduction
Jing, L.,, Vincent, P., LeCun, Y., and Tian, Y. Understand-
Pagliardini, M., Gupta, P., and Jaggi, M. Unsupervised learning of sentence embeddings using compositional n-gram features. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACI-HLTT), pp. 528- 540, 2018.
ing dimensional collapse in contrastive self-supervised learning. In International Conference on Learning Rep- resentations (ICLR), 2022.
Khosla, P. Teterwak, P., Wang, C., Samna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., and Krishnan, D. Supervised contrastive learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Rajput, S., Feng, Z., Charles, Z., Loh, P., and Papailiopoulos, D. S. Does data augmentation lead to positive margin? In International Conference on Machine Learning (ICML), volume 97, pp. 5321-5330, 2019.
Kilmer, M. E., Horesh, L.,, Avron, H., and Newman, E. Tensor-tensor algebra for optimal representation and com- pression of multiway data. Proceedings of the National Academy of Sciences, 118, 2021.
Saunshi, N., Ash, J. T., Goel, S., Misra, D., Zhang, C., Arora, S., Kakade, S. M., and Krishnamurthy, A. Un- derstanding contrastive learning requires incorporating inductive biases. In International Conference on Machine Learning (ICML), volume 162, pp. 19250-19286, 2022.
Lee, S., Park, T., and Lee, K. Soft contrastive learning for time series. In International Conference on Learning Representations (ICLR), 2024.
Tamkin, A., Glasgow, M., He, X., and Goodman, N. D. Feature dropout: Revisiting the role of augmentations in contrastive learning. In Advances in Neural Information Processing Systems (NeurlPS)), 2023.
Lei, Y., Yang, T., Ying, Y., and Zhou, D. Generalization analysis for contrastive representation learning. In In- ternational Conference on Machine Learning (ICML), volume 202, pp. 19200-19221 2023.
Tian, Y., Sun, C., Poole, B., Krishnan, D., Schmid, C., and Isola, P. What makes for good views for contrastive learning? In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Lin, M., Xiao, T., Dai, E., Zhang, X., and Wang, S. Certi- fiably robust graph contrastive learning. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Lin, Z., Zhang. Z., Wang, M., Shi, Y., Wu, X.. and Zheng, Y. Multi-modal contrastive representation learning for entity alignment. In International Conference on Computational Linguistics (COLING). pp. 2572-2584, 2022.
Tsai, Y. H., Li, T., Liu, W., Liao, P., Salakhutdinov, R., and Morency, L. Learning weakly-supervised contrastive representations. In International Conference Learning Representations (ICLR), 2022.
Liu, J., Tang, H., and Liu, Y. Perfect alignment may be poisonous to graph contrastive learning. In International Conference on Machine Learning (ICML), 2024.
van den Oord, A., Li, Y., and Vinyals, O. Representation learning with contrastive predictive coding, 2018.
Louizos, C., Reisser, M., and Korzhenkov, D. A mutual information perspective on federated contrastive learning. In International Conference on Learning Representations (ICLR), 2024.
Waida, H., Wada, Y., Andéol, L.,, Nakagawa, T., Zhang, Y., and Kanamori, T. Towards understanding the mech- anism of contrastive learning via similarity structure: A theoretical analysis. In Machine Learning and Knowl- edge Discovery in Databases: Research Track European Conference (ECML PKDD), volume 14172, pp. 709-727, 2023.
Luo, R., Wang, Y., and Wang, Y. Rethinking the effect of data augmentation in adversarial contrastive learning. In International Conference on Learning Representations (ICLR), 2023.
Wall, M. E.,, Rechtsteiner, A., and Rocha, M. Singular value decomposition and principal component analysis, pp. 91-109. Springer US, 2003.
Mikolov, T.. Sutskever, I., Chen, K., Corrado, G. S., and Dean, J. Distributed representations of words and phrases and their compositionality. In Advances in Neural In- formation Processing Systems (NIPS), pp. 3111-3119, 2013.
Wang. Y., Zhang, Q., Wang, Y., Yang. J., and Lin, Z. Chaos is ladder: A new theoretical understanding of con- trastive learning via augmentation overlap. In Interna- tional Conference on Learning Representations (ICLR), 2022.
Nonnenmacher, M. T., Oldenburg, L.., Steinwart, I., and Reeb, D. Utilizing expert features for contrastive learn- ing of time-series representations. In International Con- ference on Machine Learning (ICML), volume 162, pp. 16969-16989, 2022.
Wang, Y., Zhang, J., and Wang, Y. Do generated data always help contrastive learning? In International Conference on Learning Representations (ICLR), 2024.

---

How does Labeling Error Impact Contrastive Learning? A Perspective from Data Dimensionality Reduction
Wang, Z., Zhao, Y., Cheng, X., Huang, H., Liu, J., Yin, A., Tang, L.., Li, L.,, Wang, Y., Zhang, Z., and Zhao, Z. Connecting multi-modal contrastive representations. In Advances in Neural Information Processing Systems (NeurIPS), 2023a.
Zang, Z.,, Luo, H., Wang, K., Zhang, P., Wang, F, Li, S. Z., and You, Y. Diffaug: Enhance unsupervised contrastive learning with domain-knowlldde-free diffusion-based data augmentation. In International Conference on Ma- chine Learning (ICML), 2024.
Wang. Z., Zhao, Y., Cheng, X., Huang, H., Liu, J., Yin, A., Tang, L., Li, L., Wang, Y., Zhang, Z. and Zhao, Z. Connecting multi-modal contrastive representations. In Advances in Neural Information Processing Systems (NeurIPS), 2023b.
Zhang, A., Sheng, L.., Cai, Z.., Wang. X., and Chua, T. Empowering collaborative filtering with principled adver- sarial contrastive loss. In Advances in Neural Information Processing Systems (NeurIPS), 2023a.
Zhang, Q. Wang. Y., and Wang, Y. On the generalization of multi-modal contrastive learning. In International Conference on Machine Learning (ICML), volume 202,
Wen, W., Li, H., Gong, T., and Chen, H. Towards sharper generalization bounds for adversarial contrastive learning- In International Joint Conference Artificial Intelli- gence (ICCAT), pp. 5190-5198, 2024.
pp. 41677-41693, 2023b.
Zheng, M., Wang, F., You, S., Qian, C., Zhang, C.,, Wang, X., and Xu, C. Weakly supervised contrastive learning. In IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10022-10031, 2021.
Wu, S., Zhang. H. R., Valiant, G., and Ré, C. On the generalization effects of linear transformations in data augmentation. In International Conference on Machine Learning (ICML), volume 119, pp. 10410-10420, 2020.
Zheng, X., Wang, T., Cheng, W., Ma, A., Chen, H., Sha, M., and Luo, D. Parametric augmentation for time se- ries contrastive learning. In International Conference on Learning Representations (ICLR), 2024.
Xia, J., Wu, L., Wang, G., Chen, J., and Li, S. Z. Progel: Rethinking hard negative mining in graph contrastive learning. In International Conference on Machine Learn- ing (ICML), volume 162, pp. 24332-24346, 2022.
Zimmermann, R. S., Sharma, Y., Schneider, S., Bethge, M., and Brendel, W. Contrastive learning inverts the data generating process. In International Conference on Machine Learning (ICML), volume 139, pp. 12979- 12990, 2021.
Xia, W., Wang, T., Gao, Q. Yang, M., and Gao, X. Graph embedding contrastive multi-modal representation learn- ing for clustering. IEEE Transactions Image Process- 2023.
ing,
Zou, X. and Liu, W. Generalization bounds for adversar- ial contrastive learning. Journal of Machine Learning Research, 24:1 14:1-114:54, 2023.
Xu, M. A., Moreno, A., Wei, H., Marlin, B. M., and Rehg. J. M. REBAR:
retrieval-based reconstruction for time- series contrastive learning. In The Twelfth International Conference on Learning Representations (ICLR), 2024.
Xu, X., Zhang, J., Liu, F. Sugiyama, M., and Kankanhalli, M. S. Enhancing adversarial contrastive learning via ad- versarial invariant regularization. In Advances in Neural Information Processing Systems (NeurIPS), 2023a.
Xu, X., Zhang, J., Liu, F., Sugiyama, M., and Kankan- halli, M. S. Efficient adversarial contrastive learning via robustness-awan coreset selection. In Advances in Neu- ral Information Processing Systems (NeurIPS), 2023b.
Yu, Q., Liu, Y., Wang, Y., Xu, K., and Liu, J. Multimodal federated learning via contrastive representation ensem- ble. In International Conference on Learning Represen- tations (ICLR), 2023a.
Yu, Y., Wang, X., Zhang, M., Liu, N., and Shi, C. Prov- able training for graph contrastive learning. In Advances in Neural Information Processing Systems (NeurlPS), 2023b.
12

---

How does Labeling Error Impact Contrastive Learning? A Perspective from Data Dimensionality Reduction
A. Notations
The main notations this paper are summarized Table
Table 7. Summary of main notations involved in this paper.
Notations
Descriptions
D
the original unlabeled dataset drawing from the distribution P
p(x)z)
the distribution of the augmented sample conditioned on the unlabeled original sample 2 the joint distribution of the positive augmented sample pair (z, xt)
x*)
(07 Pim
M negative augmented samples
F2
the functional spaces of the encoder and the linear projection head g
k
the dimension of embedding vector
K
the number of labels for downstream classification task the data augmentation set defined as {tlt IR + RR population InfoNCE loss empirical InfoNCE loss
T
LinfoNCE(f)
LinfoNCE(f)
fr
the empirical optimal encoder for min LinfoNCE(f) cross entropy (CE) loss
LCE(g)
95.w,9f. W
linear classifier and mean classifier
the weight of the linear projection head defined as [w], wkl the parameter of mean projection head defined [Mi,...: [kl the downstream classification error defined as Pr [gr. (I) # yul T>
μ
E(f.W) w) a
the labeling error defined as Ea-P,r<p(112 [I [yv + yall
m, ml
the size of the matrix X Rummm (for example an image)
U(V)
the left (right) singular matrix with m(m') orthonormal column vectors the diagonal matrix whose diagonal elements are singular values the truncated parameter of truncated SVD and its optimal value
s
4,4
Cape
the spectral contrastive loss
Daug; n
the augmentation dataset defined as (œ|z t(z), D. T and its number augmentation graph, its adjacency matrix and normalized graph Laplacian matrix the eigenvalues of L
G,A, L
A
B. Lemmas
Lemma B.1 (Lemma A.2 Wang et al. (2022)). For LSE log Ex()) eexp (f(x) T g(2))], we denote its (biased) Monte Carlo estimate with M random samples 2; ~ p(z), 1..., M as LŠE,s log 1 ΣM, exp (f(x)T9(zz)) Then, approximation error can be upper boundedd in expectation as
M
LSEM
LSE| 0 (M-
E E
13

---

How does Labeling Error Impact Contrastive Learning? A Perspective from Data Dimensionality Reduction
Lemma B.2 (Equation (11) in Wang al. (2022)). Let a projector J F Rd + s*-1 map map from the d-dimensonal inpud space unit hypersphere in the k-dimensional space. For I, xt E RRd, we have
f(x) (f(xt) - S f(xt)- 1( My [1 myll
) (f(xt)- [11((xt) -uyl
where My (f denotes label.
Lemma B.3 (Corollary 3.5 in Budimir al. (2000)). Let the function g TRd + R be differentiable convex and L-smooth mapping. Then, for any E Rd, we have
(x(s) [o(=)] ((Ep()) [=]) S L Σ
where z00 denotes the j-th dimension of Vz26)) denotes the variance of z(J),
Lemma B.4 (Theorem 4.5 in Arora et al. (2019)). With probability at least 8, for any f arg minfen,,,, LinfoNCE(f) and g E F2, there holds that LCE I Lcelgs-.u
Lemma B.5 (Theorem C.3 HaoChen et al. (2021)). Assume the set of augmented data Daug is finite. Let f E arg min J.D wuy -RR be a minimizer of the population spectral contrastive loss Lspe(f) with g Then, there exists a linear head W E R&XK with norm [W*]] || S 1/(11 Aτ) such that
Pr - (x) + Vnl S 4a
+ 8a.
R p.p(1pp((1))
Act1
C. Proofs of main results
Theorem C.1 (Theorem 4.2 (restated)). Let Assumption 3.1 hold. For any f Fi,g F2, the gap between the mean downstream classification risk and the contrastive risk Lor(gs.) LinfoNGA([] can be upper bounded by
(V + VV- (f(x)) +0 (M- (0 -log
/
K
and lower bounded by
VV(x( - VV- -
logg - - (M-3))
M +
K
where V (f(x)) E (a.at)ex [15(z) Mp V- (f(x)) B(axatex-- [11((x)) Myy Mya IP]] v(f(=-) V
13] are the intra-class variance of the representa- tions for true positive augmented samples, the variance false positive augmented samples, and the intra-class variance for negative and true positive augmented samples, respectively.
(f(xt), (ff(z)))00 Da. yz) E(spe(((att)) (,,,, 1la-
Proof of Theorem 4.2: Assume positive augmented samples are t and M negative samples are ;; 1, M (x; belongs to any class in the K classes). We let μy, = E= [f)).. Under our assumptions, the labels of x and x+ may be different from each other. In other words, data augmentation may change the semantics of the original samples due to the intrinsic randomness. Thus, we denote the sets of true positive sample pair and false positive sample pair as X+ = ((x, x |uz Uz and X- = {(x, x+ My # Uz J. respectively. Then, we have the following lower bounds of
14

---

How does Labeling Error Impact Contrastive Learning? A Perspective from Data Dimensionality Reduction
the InfoNCE loss
LinfoNCE(f)
e (a) s(at)
-E n.t (x
log
ef(a) f(xt) + Ei=l ef(o)
log 1+ EM exp (f ((,,))
= E r.rl
dz, 1
exp (f(z) f(xt))
2 2- - f(x*)) +B o.
JM
1i
log
exp (f(x) Tf((T)()))
i=1
M
Bn [f(x)]((x*) E Be h
log
1
exp (f(x) ff(())
+ log M
M
(1)
Σ exp ([(())((()))) /10731 0 (xr-2) + log M
M
2 - [f(x) f(xxt) + Ex
log M
E
Ee, [f()] [(x*)] + En log Ee- [exp (J(x) [(x [ (00---0(--)
(M-
+ log M
B(x,ot)ex- [fxx]](x*)) [(x))] E( [(x))
E log Eay-
[e 10-- -v-- [exp (fx)) f(x []1]] O (M (M- + log M
(2)
2- B(axl)e [f(x)] (My, + +((x)) - ky.,]] - -B(zzI JEX [f(x) (My + f(xx) - mm-)
+ Ex [log E-y-
[exp (f(x) [yy)]] )]] - 0 (M (M-b) log M
-Ba,,ttex- [f(x)] T My + (x)) (s(xt) )- [yy,)] - -Bx [fxx) My + s(x) (s Myy))]
E, log EBy [exp (fx) py myy))] 0 (M) + loggM M
(3)
2- - ((xx [f(x)] My, - [|((xx]()++))))x)}()))) - My ll! + Ex [log 18y- [exp (f(x) yy-)]l
- 0 (Mr + log M -E(eott [s(x) T Nv + [||((xxt)> - kv
(
K
2- B(axo)Ex-- [f(x)] [m.]] V ((..t))xx [I1((x)) -Moll + Ex log K
Σ exp (f(x) mx))
k=1
- 0 (M- + +log M -E (x, [f(x] [0 - VBReetttex [Ils(ct) M 11²)
Eg [(x) 1ay - loog Σ
exp (f(x) (xx) - /Beestteex [IL((0) - -My. myall"
[1L(())-Mwx11))) 11]
k=1
0 (M-=) + -oog
M
[[lf(ot] - MPa
K
=Lor(gru)) -
- ( [1[(()) - -yy -
- O --000M + log
(M) K
[[l)(ot] - Myy 11].
where inequality (1) derives from Lemma B. inequality (2) is due to the Jensen's inequality of convex function (ex- ponential function exp), incquality (3) follows Lemma B.2, and inequality (4) is from the Cauchy-Schweer incquality.
Let V (f(x)) Dex+ [l](()) - My, [?]] and V- (f(z)) (xx,
[Ilf(x)) - P l1]] We can get the upper
bound of (cE(grw) + log (4) - [In)oNCE(f) as
(oe(grw) +log ( IM (M)
+0 (M- -
K
15

---

How does Labeling Error Impact Contrastive Learning? A Perspective from Data Dimensionality Reduction
Next, we will prove the corresponding lower bound.
(Celdru)
K exp (f(z) px)
- [f(x)] Hyal + Ex
log
k=1
K
Eg [f(z)] [yv + +E
log
exp (f(x)] ((x))))m))))))))))))))
+ log K
K
R=1
( text [f(x)* Mya] + EE [log By [exp (f(x) My )]] log K - (x, jex [f(z)) p
(1)
2 - B(axxt)xx [f(x) ((xt)= + f(x) (my-- [xx
B Berity Wes log M+1 exp (f(x))((u))) + Σ exp (f(z) myz) )]
M
+ Ez
1=1
o (M- + log K - ((xx)))E-- [f(x)] f(x)) + f(x) (Nya [xx)))
(2)
2 B(xxt)Ex [f(x)]((xt)) Bla [llf(z) My. III
E Borly, 1i log M+1 exp (f(x) (yyx)= + Σ exp (f(D)TMu: (vx) )
M
+ Eg
- o (M-+) + log K - B(axo))E---- [f(x) s(xt)] -B(o,t))xx [1f(xtt) My
M
E --Br,et [f(x)T((xt)) + Ey Prentorr Jan, log M+1
1
exp (J(x) [y.)(yy))) + Σ
exp (f(z) My:
1=1
-(nxt))x [1lf(x) Mva 111 E(aat)Ex [|]f(x*) - -M 0 (M--) + log K
-gg,et [(x)]] (((,ot)Ex+ [II[(x) pyy 111 B(xx []]](xt) M 11 - O (M- + log K
M
) -M :)
+ Eg
E Bv:
log M +1
B. (aatin Met -M.M =Mi
exp (f(x) f(+))(())))))))))))))}{)))))ff (f(x)T((at)) + exp (f(x)
((x (f(o)T((15))))
(3)
Ey,t [f(x)] [f(x)]((xt)] (((xxt))x [Ilf(x)) Hy, 11]
- [[n,ttex-- [1](xt)] - Hyr 12] - -
2
+ log K
+ Eg
E Benii 1
E
log M +1
1
exp (f(x) (f(x))((x())) + Σ exp (f(x) ((1)))
a .T 18 M, m
1 V(elze E (f(x+), f(z (f(fz())00 N1, = yv)
2
M
E,,t [f(x) f(xt)] Exp
E
log exp (expp (f(x) (((x)T((²))) f(xt)) Σ exp (f(x) f(x)))
-V(((=)) (f(x))
2 V(lk (f(xt), f(x f( = vx)) O ( [MI (M-+) log
M +
K
=CInfoNCE(f)- VV((()) VV-
2 v(l2z e (f(x*),f(a 11 Nz+ = yz) -o (M- llog
M +
K
where the inequalities (1), (2) are similar to the proof of the above upper bound, the inequality (3) follows Lemma B.3 and Wang et al. (2022) proved the convex function logsumexp 1 -soooh.
16

---

How does Labeling Error Impact Contrastive Learning? A Perspective from Data Dimensionality Reduction
the Cauchy-Seti Bluchy-Schwarz inequality. We can get the lower bound of (CE(D)))) log M+
LInfoNCE(())
M [M -++11 1
) - LinsONCE(f)
cCE( + log
K
VV(((( VV- (f(z))
2 V(slz (((+t),(o Ux)) -=)- (Mr-
By integrating the Theorem 4.5 of Arora et al. (2019), we can directly derive Corollary C.2 which measures the upper bound of the classification risk LCE LCE(gs- (9)
Corollary C.2. Under the condition of Theorem 4.2, the downstream classification risk LCE(gg .w) + log ( (x) can be upper bounded by LInSANCE(()) + /V- (f*(z)) + vV([(()) + O O (M-
Proof of Corollary C.2: According to Lemma B.4, we can know that, for any f E arg m LInfoNCE(f) and g E F2, the downstream classification risk (gg w) with the linear classifier W can be bounded by the mean downstream classification risk Lc (gi-,p) .a) with the mean classifier μs. And, from Theorem 4.2, we can obtain the upper bound of the mean
downstream classification risk Lo n)) as (In) VV- (f*()) + (V((())) + O (M-+) log #
Therefore, the upper bound of Corollary C.2 is proved.
Assumption C.3 (Assumption 4.5 (restated)). Let a sample and its corresponding sample after applying SVD be represented as the matrices& X and Xg E Rm respectively. Assume that there are 9 singular values associated with semantic-related information. When g 2 q", under Assumption 3.1 and the augmentation collection T, the true label of the augmented sample of Xg is inconsistent with the latent label of X with the probability ag S Conversely, when q q", the corresponding probability satisfies Og >> Og*
Statement of Assumption 4.5: practice, each sample possesses unique amount of semantic-related information content. While for convenience, we assume that each sample can be decomposed into q" singular values capturing semantic-related information. Under Lemma 4.4 and the default assumption that there is a positive correlation between the amount of information and the importance of information, more semantic-related the information captured in the corresponding subspace. Table show the trend of increase of downstream classification accuracy from the experiment discarding the singular values $1, 2 to the one discarding $31, 832. which empirically validates this demonstration. Based on the above analysis, we can derive that truncated SVD firstly discards semantically irrelevant information that leads to labeling error. When takes the value q", samples don't 't have any semantically irrelevant information. When < q", labeling error will increase due to the loss of semantic-related information.
demonstrate that the larger the singular value, the
Theorem C.4 (Theorem 4.7 (restated)). Given the conditions of Theorem 4.2 and Assumption 4.6, after taking the truncated SVD on D, the mean downstream classification risk LCE (9) - LInfoNCE(f) with the encoder f can be upper bounded by
Eq- t O (M-
- log
-log
M
K
and lower bounded by
Eg' Eg (ff())))-0f (M--)) (w- -log
IM + 1
K
When g = q², the two bounds are eq- + O (M1-b) log ( and -Eg- v(f(x))) - O (M--) log ( (1+1)
Proof of Theorem 4.7:
17

---

How does Labeling Error Impact Contrastive Learning? A Perspective from Data Dimensionality Reduction
Upper bound: From the proof of Theorem 4.2, can get that
LinfoNCE(f)
2 E(xx [5(z) Myr + +11[(x*)) - Mys |1] + Ex [log Ey- [exp (f(z) My- )]]
o (M + log M - Blax))e [f(x)) T Mys + ||f(xx) - -My=ll Ill
>ccE(D5 Blax [1lf(x)) Mv. I1I - O (M-+ + log
MM
- [e (ex [1] Mve
K
-[CE(95.) ((ax))ex
f(x +1 -0 (M- + log
M
K
-Bi,,tt)Ex- BR [s(x*) [(x)]ll
1 LCE(DFN) B(ax)))x [1] f(xx) o (2) + log
M
((x []][(x)))))])))))))(()))))))[[ f(x))
K
Lor(DsN) -n- -G -0 -M + log
M
K
where the inequality (I) is derived from the Cauchy- inequality and the inequality (2) due to Assumption 4.6. Then.
M
LinfoNCE(f) co + cn + O (M-
LcElgga) + log
K
Lower bound:
Similarly, we can get that
foE(ggu) LinfaNCE(() - ((xx)))x (1) - My. Bea,tt)e-- [x Myalll II]
2 - (f(xt), ), yz) O 0^M (MI+) -log M +
log
K
ZimJoNCE(f) - - v(el= ff f(=))(((/))))
}/. Ux- yz) - O (M--) -log -log yz)
M + 1
K
Corollary C.5. Under the condition of Theorem 4.7, the downstream classification risk LCE + log K
M
can be
upper bounded by LIntoNCE("*) + + + (M=3) When = =4" the bound is (f +0 M (M-1)
Theorem C.6 (Theorem 4.9 (restated)). Let Assumption 3.1 hold. For the empirical optimal encoder f taking the truncated SVD with aaper-paraneter q on unlabeled original dataset, there exists linear head W e B with norm
||W*|| < 1/(1 - )n.) such that
40g + 80g.
E( W*) S
dt+1.g
where & denotes the dimension of embedding space and Acti.g denotes the k + 1-th eigenvalues of Lg:
Proof of Theorem 4.9: Lemma B.5 states that
4a
Pr S~P
los. (x) + Val
+ 8a.
Ab+1
We rewrite the definition E(f, W)
Pr (x) + Uzl (xP.)xp(E(E))
Pr
(gr,w(x) (x) # Uz]. Thus, after taking the truncated
a-p
SVD with hyper-parameter q E [m] on unlabeled sample, the following bound holds
4ag + 80kg
8(8", W*) = Pr Z~P P=Pp2))(E
[a (x) + Uzl I
Pr aaP ponp((p()
[g (x) + Uzl S
Axt1.9
18

---

How does Labeling Error Impact Contrastive Learning? A Perspective from Data Dimensionality Reduction
D. Experimental setting
Details of models and datasets. For the backbone structure, we use three variants of Resnet, i.e., Resnet-18, Resnet-50, and Resnet- 152, where the dimensions of embedding are chosen from 128, 256, 512, 1024, and 2048. We use two-layers multilayer perceptron (MLP) be the projection layers. For the datasets, we employ three benchmark datasets, i... CIFAR-10, CIFAR-100 100, and STL-10.
Details of pre-training and fine-tuning, We do pre-training for 100 epochs with batch size 256 using the optimizer Adam. The optimizer has the weight decay parameter 0.0004 and learning rate 0.0003. The learning rate decreases following the cosine schedule. In this stage, the dimension of the final output of the model is set as 128. For the downstream fine-tuning process, we train with a small number of labeling samples for 100 epochs with batch size 256 using the optimizer Adam with weight decay parameter 0.0008 and learning rate 0.0003. We use RTX 2070 GPU for all experiments.
Details of these experiments with inflation Considering fair comparison, we do pre- pre-training for 33 epochs with batch size 256 when we make data inflation. The settings of fine-tuning are the same for all experiments.
Details of augmentations In the main text, we provide several groups of augmentation strategies whose components are listed. We need to make the explanations that these abbreviations of augmentations without specific parameter settings adopt their default settings. For example. "RRC" (instead of "RRC(0.08, 0.5)") adopts the default parameters "(0.08, 1.0)", where the range *(0.08, 1.0)" denotes the proportion of preserved area after cropping. "Color jitter" (instead of "Color jitter(O.55, 0.4)") adopts the default parameters "(1.0, 0.8)", where "1.0" denotes scale parameter and "0.8" denotes probability parameter. "Cutout" (instead of "Cutout(O.5, 1.0)") adopts the default parameters "(0.1, 1.0)", where the range "(0.1, 1.0)" denotes the coordinate parameters used to cut samples.
E. Discussions with related works
Huang et al. (2023b) proposed the concept of augmented distance and provided some upper bounds revealing the theoretical effect of augmented distance on understream classification performance. Specifically, they found that the classification performance of contrastive SSL is related to three key factors: alignment of positive samples, divergence of class centers, and concentration of augmented data. Theorem 2 in our work provided both upper and lower bound, which can not only give similar conclusions but also reveal some additional factors. Firstly, the term V(J(z))vz) implies the alignment of positive samples. Secondly, the term Vyz (f(x) (ya) stems from labeling eтror caused by data augmentation, which is similar to the concentration of augmented data. Thirdly, the term V(f(=-)ly implies the alignment of negative samples, which is not considered by Huang et al. (2023b). More importantly, we further improved the bounds of Theorem 2 via data dimensionality reduction and provided the corresponding thcoretical analysis and empirical observations (Section 4.2).
Cui et al. (2023) established theoretical framework for weakly supervised contrastive learning for the first time. Their results revealed that 1) semi-supervised information improves the error bound compared with purely unsupervised contrastive learning by using all labeled samples; 2) joint training of supervised and unsupervised contrastive learning does not improve the error bound compared with purely supervised or purely unsupervised contrastive learning. Although weakly supervised contrastive learning is not the topic of our work, the labeling error considered by our work is analogous to a type of weak supervision, i.e., noisy-labeled information. Therefore, we will extend the theoretical analysis of this work to weakly supervised contrastive learning in our future work. Besides, Cui al. (2023) and our work both gave the suggestion that we should choose moderate feature dimension k. which enhances the credibility of our suggestion.
F. Limitations
The limitations of our work are listed as follows:
This paper selects the embedding dimension k = 512 or k = 1024 as an example of a moderate choice. Nevertheless, we've ve observed that the optimal values of l vary somewhat across different experimental settings, showing a degree of inconsistency. Delving into the underlying reasons for this inconsistency holds great interest since it has the potential to provide valuable insights that would be beneficial for the design of more effective algorithms.
This paper pioneers in theoretically elucidating the role of labeling error in contrastive learning from a novel perspective, data dimensionality reduction. Algorithmically, we utilize the classical SVD for preliminary empirical validation. However, considering the unavailability and diversity of p" across different settings, the fixed g in the traditional SVD
19

---

How does Labeling Error Impact Contrastive Learning? A Perspective from Data Dimensionality Reduction
fails guarantee sufficiently small labeling error. Thus, future research will focus developing more flexible low-rank image approximation method, which assigns distinct truncated parameter q to each sample.
Even though our work mainly concentrates on the study the potential false positive augmented samples, conducting research on false negative augmented samples remains of great necessity.
G. Other Experimental Results
CIFAR- 10, Resnet-18
CIFAR-100 Resnet- 18
SVD
SVD with 9=30
SVD with 30
SVD with 25
SVD q-2h
Classification Risk
SVD with 20
Classification Risk
SVD with q-
SVD with 15
SVD with 9-15
SVD with q-10
q-10
088
0.
50
100
20
100
Epoch
Epoch
Figure 5. Classification risks of SimCLR using the augmentation and the truncated SVD with different values of q. The corresponding top-1 accuracies are shown in Table 2.
Table Downstream classification top-1 accuracies (%) of SimCLR (CInfoNCE) using the truncated SVD (g 30 for CIFAR-10 10 and CIFAR-100, 4 90 for STL-10) with different embedding dimension k.
Embedding Dimension
Encoder
Dataset
k 128
k 256
A: 512
k
1024
k
2048
T
Resnet- 18
CIFAR-10
68.12 12
69.11
69.48
69.27
68.84
T
Resnet-50
CIFAR-10
67.72
68.29
69.12
67.83
63.36
T
Resnet-155
CIFAR-10
66.75
67.64
68.05
65.98
62.50
Resnet-18
CIFAR-100
37.68
38.49
39.59
39.42
39.19
7
Resnet-50
CIFAR-100
38.96
39.50
39.83
38.10
32.23
T
Resnet-16 18
STL-10
71.28
71.37
71.93
71.49
71.34
T
Resnet-50
STL-10
72.72
73.76
72.53
71.73
70.28
20

---

How does Labeling Error Impact Contrastive Learning? A Perspective from Data Dimensionality Reduction
Table Downstream classification lop-1 accuracies (%) of SimCLR (LIffoNCE) using the truncated SVD (4 30 for CIFAR-10 - with different epochs.
Epochs
T
Encoder
Dataset
SVD
100
200
300
400
500
T
Resnet-18
CIFAR-I0
w.o. SVD
68.82
70.91
71.05
72.97
74.54
Ti
Resnet-18
CIFAR-I0
q 30
69.48
71.06
71.48
73.43
74.94
Table 10. Downstream classification top-1 accuracies (%) of MaCo (CinfoNCE) using the truncated SVD with different q.
T
Encoder
Dataset
w/o SVD
q =
30
9
25
9
20
=
15
9
Ti
Resnet-18
CIFAR-10
72.69
73.02
72.48
71.58
71.30
69.87
Table Downstream classification top-1 accuracies (%) of SimCLR (CrofoNCR) using truncated SVD on TinylmageNet-200
(image size 64 x 64, 50 pre-training epochs) and different backbones (VIT, ConvNeXt, 10 pre-training epochs) with different truncated parameter q.
T
Encoder
Dataset
w/o SVD
q = 60
q = 50
g =
T
Resnet-18 18
TinyImageNet-200
28.72
29.38
28.44
27.94
T
ViT
CIFAR-10
42.22
42.98
42.87
38.44
T
ConvNeXt
CIFAR-10
55.75
56.37
55,87
55.59
Table 12. Downstream classification top-1 accuracics (%) of SimCLR (CinfaNcE) CIFAR- 10 using the truncated SVD with Random Erasing, GridMask and HidePatch.
SVD
Encoder
Random Erasing GridMask HidePatch
W.o. SVD
Resnet-18 18
49.27
50.33
23.05
q =
Resnet-18 49.65
51.20
26.22
Table 13. Downstream classification top-1 accuracies (%) SimCLR (CInfONCEs pre-training epochs) on ConvNeXi using the truncated SVD (q 30 for CIFAR 10) with different embedding dimension (- represents that does not converge).
Embedding Dimension
Encoder
Dataset
k = 128
k = 256
512
==
== 1024
k = 2048
k == 3072
h: = 4096
T
ViT
CIFAR-10
41.62
41.28
42.22
43.14
40.18
Ti
ConvNeXt
CIFAR-10
53.48
55.58
55.75
55.89
56.71
55.84
55.13
Table 14. Downstream classification top-1 accuracies (%) of BYOL (LinfONCR, 10 pre-training epochs) using the truncated SVD
30 for CIFAR-10) with different embedding dimension
Embedding Dimension
T
Encoder
Dataset
k 128
k 256
k k=511
512
k 1024
k: = 2048
k = 3072
k 4096
Ti
BYOL
CIFAR-10
33.71
33.85
34.08
34.09
34.32
33.11
33.00
21

---

How does Labeling Error Impact Contrastive Learning? A Perspective from Data Dimensionality Reduction
Table 15. Downstream classification top-1 accuracies (%) of SimCLR (CInfONCE) CIFAR-100 and STL-10 using the truncated SVD with different q or the data inflation strategy under the weak data augmentation adopted by Wang et al. (2024) (Ts = (RRC(0.2, 1.0). Color jitter(O.5,, 0.4), Random horizontal flip, Random grayscale, Gaussian blur)).
Dataset
Encoder
Inflation
w/o SVD
q 90
9 70
g 50
STL-10
Ts
Resnet-18
70.86
70.51
71.49
71.23
69.26
Dataset
T
Encoder
Inflation
Inflation (g 90)
[y 70)
(q 50)
STL-10
Ts
Resnet-16 18
70.86
71.75
71.21
69.85
Dataset
T
Encoder
Inflation
w/o SVD
q = 30
g = 25
g = 20
CIFAR-100
Ts
Resnet-18
42.84
42.29
42.93
42.72
42.5
Dataset
T
Encoder
Inflation
Inflation
(g 30)
(q 25)
(q 20)
CIFAR-100
Ts
Resnet- 18
42.84
43.04
43.08
42.92
Table 16. The estimated frequencies (%) of labeling for CIFAR-10 with multiply augmentation strategies.
Augmentation
RRC
Ti
Ta
Ta
Frequency
27.9
37.5
48.3
42.1
Table 17. The cost of SVD different datasets.
Dataset
CIFAR-10
TinylmageNet-200
STL-10
Size
32 32
64 64
96 96
Truncated parameter q
30
60
q 90
Time of per image/s
0.001994
0.004985
0.007978
Time of all images /s
99.7 99.7186
498.5094
797.7724
22


Reviews:
[{"review_id": "VROeNKfMsa", "reviewer": "Reviewer_JnUb", "summary": "This paper investigates the impact of labelling errors introduced by augmentation in contrastive learning. The authors first demonstrate labelling errors in terms of positive and negative pairs and theoretically prove that these errors affect the upper and lower bounds of the downstream classification risk. Furthermore, they propose adopting Singular Value Decomposition (SVD) in the augmentation strategy to reduce irrelevant semantic features and minimise classification errors.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to you for your valuable comments and constructive suggestions.\n\n**Q1:** ImageNet?\n\n**A1:** Thanks. Considering the size of the original ImageNet is too large, it is hard to complete the experiment on it before the deadline (March 31 (AoE)). We have added experiments on TinyImageNet (anonymous link: https://imgse.com/i/pEsQWVK).\n\n***\n**Q2:** Discontinuous singular value in Table 1?\n\n**A2:** We have added experiments for $s_{i,j}$ (link: https://imgse.com/i/pEsasuF).\n\n***\n**Q3:** Are improvements meaningful?\n\n**A3:** Although a few experimental results did not demonstrate significant improvement, our all experiments exhibited a consistent trend of progress across different settings. In some cases, the improvement exceeded 1%, even when q=30 (such as columns 3,5,6 in Table 4). Further reductions in the q value may lead to additional performance gains.\n\n***\n**Q4:** Why q=30 not q=25?\n\n**A4:** Since the optimal q varies across different experimental settings (Table 2), we fixed q=30 for subsequent experiments to ensure consistent improvement in downstream performance across varying conditions, thereby validating our theory, even though the improvement might be relatively limited in some settings when adopting q=30. Note that, this paper aims not to obtain optimal downstream performance but to investigate how dimensionality reduction affects labeling error.\n\n***\n**Q5:** Other augmentations?\n\n**A5:** We have added experiments with Random Erasing, GridMask, and HidePatch (link: https://imgse.com/i/pEs15jA). They also support the claims made in this paper.\n\n***\n**Q6:** Discussion of dimensionality reduction?\n\n**A6:** Thanks. We have supplemented the discussion of various dimensionality reduction techniques. Beyond SVD, there are various techniques for feature extraction and dimensionality reduction, including matrix and tensor decomposition (PCA[1], NMF[2]), dictionary learning[3], compressed sensing[4], deep learning (Autoencoder[5], GAN[6]). Since we use SVD as a simple example to study the effect of dimensionality reduction on contrastive learning, we didn’t carefully make a survey about various dimensionality reduction methods. Our empirical observations and theory show that an effective data augmentation based on dimensionality reduction is necessary. We will conduct research on dimensionality reduction to develop a flexible low-rank approximation as a new augmentation to achieve greater model performance, which was mentioned in the second point of Limitation (Appendix E).\n\n[1]Y. Ren, et al., Hyperspectral Image Spectral-Spatial Feature Extraction via Tensor Principal Component Analysis, Geoscience and Remote Sensing Letters, 2017.\n\n[2]M. Chen, et al., Feature Weighted Non-Negative Matrix Factorization, Transactions on Cybernetics, 2023.\n\n[3]P. Song et al., Multimodal Image Denoising Based on Coupled Dictionary Learning, International Conference on Image Processing (ICIP), 2018.\n\n[4]T. Hong, et al., A Complex Quasi-Newton Proximal Method for Image Reconstruction in Compressed Sensing MRI, Transactions on Computational Imaging, 2024.\n\n[5]Y. Shen, et al., DRACO: A Denoising-Reconstruction Autoencoder for Cryo-EM, NeurIPS, 2024.\n\n[6]D. Chen, et al., SSL: A Self-similarity Loss for Improving Generative Image Super-resolution, ACM MM, 2024.\n\n***\n**Q7:** Experimental settings?\n\n**A7:** To ensure reproducibility, we have updated the experimental details in Appendix D of our revised manuscript.\n\n**Hardware:** All experiments are executed on an RTX 2070 GPU with an Intel(R) i7-10750H CPU.\n\n**Software:** Python 3.7.4 with PyTorch 1.13.1.\n\n**Optimizer, epoch, batchsize:** We provided them in the second point of Appendix D. Note that we pre-train STL-10 with 50 epochs due to its substantial data volume.\n\n**Augmentation:** We provided the parameter of every augmentation in the fourth point of Appendix D.\n\n**Supplementary experiments in our response:** Complete implementation details are included in their respective response.\n\nShould any additional technical specifications be required? Please inform us and we will promptly provide the requested information.\n\n***\n**Q8:** Cost from SVD.\n\n**A8:** We have added the cost of SVD on different datasets (link: https://imgse.com/i/pEsQfUO).\n\n***\n**Q9:** Validation on different backbones and datasets.\n\n**A9:** We have added experiments on TinyImageNet and ViT, ConvNeXt (link: https://imgse.com/i/pEsQWVK).\n\n***\n**Q10:** Other augmentation?\n\n**A10:** Please see **A5**.\n\n***\n**Q11:** Define moderate embedding dimensions.\n\n**A11:** We have added experiments on ViT, ConvNeXt, and BYOL (link: https://imgse.com/i/pEswcf1). We find that the optimal dimension is usually in $[512, 2048]$. The impact of the moderate dimension is valuable to be further explored in our future work, which was mentioned in the first point of Limitation (Appendix E).\n\n***\n**Q12:** Are the improvements significant?\n\n**A12:** Please see **A3**.", "claims_and_evidence": "Please see comments regarding the experiment designs and weaknesses.", "methods_and_evaluation_criteria": "I would appreciate experiments on ImageNet as well to demonstrate the generalisability of the proposed SVD-based augmentation.", "theoretical_claims": "I reviewed the theoretical claims in Sections 3, 4.1, and 4.2, and they appear to be correct.", "experimental_designs_or_analyses": "- In Table 1, why do the authors present the performance of discontinuous singular value pairs (e.g., no evaluation of $s_{2,3}$, $s_{12,13}$)\n- Is a 0.27% (row 4 column 4&5 in Table 2 results) statistically significant? In other words, are the improvements, which often appear to be less than 1%, when adopting SVD truly meaningful?\n- For the ablation studies in the Appendix, why was $q=30$ used for all experiments when Table 2 shows that $q=25$ performs better on CIFAR-10?\n- Have other augmentation combinations (besides RRC, Cutout, Color Jitter, and their combinations) been tested, and do they also support the claims made in this paper?", "relation_to_broader_scientific_literature": "This paper contributes to the growing body of research on contrastive learning by addressing the realistic challenge of labelling errors, which has been largely overlooked in previous studies. The key contributions align with and extend prior findings in the following ways:\n\n- The paper underscore the impact of labelling error in contrastive learning, providing a more realistic assumption compared to many existing works. \n\n- The authors theoretically prove the negative impacts of labelling errors, which raise awareness for future research on to robustness and reliability of contrastive learning applications.\n\n- This paper is closely related to existing works that investigate the effectiveness of adopting generated data and weak augmentation in contrastive learning. It extends these observation by demonstrating the benefit of using SVD is on par with adopting data inflation.", "essential_references_not_discussed": "The paper discusses the significance of applying dimensionality reduction to contrastive learning embeddings to mitigate the impact of labelling errors. Beyond SVD, various feature extraction and dimensionality reduction techniques exist, which should have been discussed in the related works.", "other_strengths_and_weaknesses": "Strengths:\n\n- The study examines the effects of varying the top q singular values, labelling error rates, model architectures, and embedding dimensions across three ResNet architectures and on three benchmarks.\n- The authors provide theoretical proofs to support their claims.\n\n---\n\nWeaknesses:\n\n- Some results require further justification (see comments on Experimental Designs).\n- Certain experimental settings remain unclear, making the claims less convincing.\n- There is no justification for the additional computational cost incurred by computing SVD as part of the proposed augmentation strategy.\n- Not enough validation to demonstrate the generalisability of proposed method on different architectures and datasets.", "other_comments_or_suggestions": "Please see the weaknesses above.", "questions_for_authors": "1. Is it possible to test your method on other augmentation combination (except RRC, Cutout, Colout jitter, and their combinations) to support the claims in this paper?\n2. How to define *moderate* embedding dimensions on different architectures (for example the ViTs) and different contrastive learning methods (except SimCLR and MoCo)\n3. Are the improvements of using truncated SVD considered significant?\n\nI am open to revise my recommendation if the authors can provide more justification of the generalisability and significance of their method.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K8CtXcmEh4", "reviewer": "Reviewer_339q", "summary": "This paper theoretically analyzes the effect of labeling errors, particularly cases where an augmented example may belong to a different class than the original example. Based on this, the authors further study how performing dimensionality reduction on representations can mitigate the negative impact of labeling errors and support their findings with experiments.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to you for your valuable comments and constructive suggestions. \n\n**Q1:** Since the labeling issue ultimately stems from imperfections in the augmentation process, could it be addressed simply by making more careful choices in augmentation? For instance, setting the cropping ratio appropriately or relying on smarter augmentation techniques like AutoAugment.\n\n**A1:** Thanks for your constructive comment.\n\n**Firstly**, data augmentation presents a dual challenge in deep learning systems: excessive augmentation introduces label noise due to over-distorted samples, while insufficient augmentation compromises model performance through limited diversity. **Secondly**, the optimal augmentation method varies across different datasets. Optimal augmentation selection requires non-trivial domain-specific expertise and extensive empirical validation.\n\nBesides, some smarter augmentation techniques like AutoAugment can improve model performance by dynamically adjusting augmentation strategies. However, their time-consuming and resource-intensive nature limits practical implementation. In contrast, this work pioneers a novel paradigm by employing data dimensionality reduction as a pre-process strategy to improve model performance effectively. Empirical and theoretical results validate the effect of dimensionality reduction. We will conduct research on dimensionality reduction to develop a more flexible low-rank approximation method as a new augmentation method to achieve greater model performance, which was mentioned in the second point of Limitation (Appendix E).\n\n***\n**Q2:** Additionally, I feel there should be experiments to justify how frequently this type of labeling error occurs in real-world scenarios.\n\n**A2:** Thanks. We have added experiments on multiply augmentation combinations to justify how frequently this type of labeling error occurs in real-world scenarios (anonymous link: https://imgse.com/i/pEsQo2d).", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The evaluation makes sense for the problem.", "theoretical_claims": "I didn’t go through all the proofs in detail, but the theoretical results seem convincing to me.", "experimental_designs_or_analyses": "The experimental design is valid and demonstrates the effect of applying SVD to the inputs.", "relation_to_broader_scientific_literature": "The paper is related to the literature on contrastive learning. It adds a new perspective by considering the possibility that data augmentation may transform an example into something that semantically belongs to another class.", "essential_references_not_discussed": "I didn’t notice any.", "other_strengths_and_weaknesses": "I think the paper makes a novel point by considering labeling errors caused by augmentation that might alter an example’s semantic meaning.\n\nHowever, I find the practical contribution somewhat limited. First, demonstrating the negative impact of labeling errors seems quite intuitive—everyone would expect performance to worsen when this issue is present. So, the more practical contribution is likely the authors’ demonstration that applying SVD to inputs can mitigate the issue. However, it seems that there could be other straightforward solutions. For example, since the labeling issue ultimately stems from imperfections in the augmentation process, could it be addressed simply by making more careful choices in augmentation? For instance, setting the cropping ratio appropriately or relying on smarter augmentation techniques like AutoAugment. \n\nAdditionally, I feel there should be experiments to justify how frequently this type of labeling error occurs in real-world scenarios.", "other_comments_or_suggestions": "I don’t have other comments.", "questions_for_authors": "My main questions are listed in the Strengths and Weaknesses section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u0bItaYHPp", "reviewer": "Reviewer_nEce", "summary": "This paper investigated theoretically the impact of labeling error on the downstream classification performance of contrastive learning. The authors demonstrate—both theoretically and empirically—that employing a moderate embedding dimension, data inflation, weak augmentation, and SVD fosters greater graph connectivity and reduces labeling error, ultimately improving downstream classification accuracy.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to you for your valuable comments and constructive suggestions. \n\n**Q1:** Still no evidence showing whether the labeling error is sufficiently small in this paper.\n\n**A1:** Thanks for your constructive comment. Our work didn’t guarantee that the label errors are sufficiently small, which was discussed in the second point of the Limitations section in Appendix E (line 984). The primary objective of this study is to demonstrate that the label error under the setting of prior work [1] is not negligible. Our approach further reduced labeling error, as validated in Tables 5 and 6. Achieving sufficiently low label error remains an ongoing research objective.\n\n[1]Wang, Y., et al., Do generated data always help contrastive learning? ICLR, 2024.\n\n***\n**Q2:** No clear definition of $\\epsilon$.\n\n**A2:** Thanks. We have supplemented their clear definitions as follows. $\\epsilon(\\alpha_{q})$ is defined as the maximum distance between $f(x),f(x^+)$ when taking the truncated parameter $q$, where $(x,x^+)$ is a false positive sample pair. And, $\\epsilon(\\alpha_{q^*})$ is defined as the minimum distance between $f(x),f(x^+)$ for any truncated parameter values. In summary, $[\\epsilon(\\alpha_{q^*}), \\epsilon(\\alpha_{q})]$ is the range of value of $||f(x)-f(x^+)||$ for $(x,x^+)\\sim p(x,x^+, y_{\\bar{x}}^\\neg)$.\n\n***\n**Q3:** Why does the formula hold in Theorem 4.9?\n\n**A3:** The proof of the result for Theorem 4.9 was provided at the end of Appendix C. This result is quite similar to Lemma B.5, with the key difference being the definition of $E(f,W)$. In the proof, we have rewritten the form of this definition as $\\underset{\\bar{x}\\in\\bar{D},x\\in p(\\bar{x}|\\bar{x})}{\\mathrm{Pr}} \\left[g_{f^*,W^*}(x) \\neq y_{\\bar{x}}\\right]$ to facilitate the comparison with Lemma B.5. Obviously, $x\\in p(\\bar{x}|\\bar{x})$ is included in $x\\in p(\\cdot|\\bar{x})$. According to Lemma B.5, the result of Theorem 4.9 holds. \n\n***\n**Q4:** The improvements on Table 4 and Table 5 appear to focus exclusively on CIFAR-10, which may seem inconsistent given that CIFAR-100 and STL-10 were utilized in the earlier experiments.\n\n**A4:** Thanks. We have added related experiments on CIFAR-100 and STL-10 (anonymous link: https://imgse.com/i/pEsQI8H). \n\n***\n**Q5:** The conclusions of this paper may not be universally applicable, as certain factors still involve heuristic choices, including the degree of data inflation, the extent of weak augmentation, the number of dimensions selected for data reduction, and the domain of the datasets used.\n\n**A5:** Thanks. We have made some explanations about the several factors you mentioned.\n\n**1) Data Inflation:** Data inflation is not the focus of this paper. Previous work [1] has shown that the more similar the distribution of inflated data is to the one of original data, the better the model performance. Therefore, this paper adopts the optimal inflation setting from [1].\n\n**2) Weak Augmentation:** We also adopt the weak augmentation strategy suggested in [1].\n\n**3) Data Dimensionality Reduction:** The SVD truncation parameter used in this work is manually set to verify the effectiveness of SVD, not to obtain optimal model performance. Therefore, we set a large $q$ value in most experiments. In the future, we will adaptively learn the optimal truncation value $q^*$, which is mentioned in the second point of Limitation in Appendix E.\n\n**4) Data Domain:** The experiments and theoretical analysis of this work are focused on the field of computer vision. Exploring whether the findings generalize to domains such as NLP and graph learning is an interesting and open question. Future work will extend this investigation to NLP and graph learning to assess the broader applicability of the proposed framework.\n\n[1]Wang, Y., et al., Do generated data always help contrastive learning? ICLR, 2024.\n\n***\n**Q6:** \"ultimately improving improve model performance\" to \"ultimately improving model performance\"\n\n**A6:** Thanks. We have corrected this issue. \n\n***\n**Q7:** Still no evidence showing whether the labeling error is sufficiently small in this paper.\n\n**A7:** Please see **A1**.\n\n***\n**Q8:** Will the conclusions of this paper also hold for other contrastive learning domains, such as NLP and graph learning?\n\n**A8:** Thanks. The experiments and theoretical analysis of this work are focused on the field of computer vision. Exploring whether the findings generalize to domains such as NLP and graph learning is an interesting and open question. Future work will extend this investigation to NLP and graph learning to assess the broader applicability of the proposed framework.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence except for the claim in line 87 and line 88:\n“They didn’t verify whether the labeling error caused by the weak augmentation is sufficiently small”.\nIt seems that there is still no evidence showing whether the labeling error is sufficiently small in this paper.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense for the problem.", "theoretical_claims": "I have examined the theoretical claims and identified the following issues:\n\n1.\tIn Assumption 4.6, there seems to be no clear definition of $\\epsilon(\\alpha_{q*})$  and  $\\epsilon(\\alpha_{q})$\n\n2.\tIn Theorem 4.9, there is no clear explanation for why the formula $\\varepsilon(f^*,W^*)\\leq \\frac{4\\alpha_q}{\\lambda_{k+1,q}}+8\\alpha_q$  might hold.", "experimental_designs_or_analyses": "I have carefully examined the experimental designs and analyses in Table 4 and Table 5, which incorporate various augmentations and data inflation. These experiments are both appropriate and necessary to illustrate the importance of combining data inflation, SVD, and weak augmentation. However, they appear to focus exclusively on CIFAR-10, which may seem inconsistent given that CIFAR-100 and STL-10 were utilized in the earlier experiments.", "relation_to_broader_scientific_literature": "1.\t A notable aspect of this paper is its use of SVD to specifically address label mismatch in self-supervised settings.\n2.\t This paper extends prior efforts that combine generative models (e.g., DDPMs) with contrastive learning but emphasizes that simply generating more data does not solve the labeling error problem unless mislabeling is curbed.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths: This paper has the potential to inspire further exploration of labeling errors in contrastive learning across other domains, such as NLP and graph learning.\n\nWeaknesses: The conclusions of this paper may not be universally applicable, as certain factors still involve heuristic choices, including the degree of data inflation, the extent of weak augmentation, the number of dimensions selected for data reduction, and the domain of the datasets used.", "other_comments_or_suggestions": "There is an issue in the conclusion section (line 437):\n\"ultimately improving improve model performance.\" \nIt is recommended to revise it to \"ultimately improving model performance.\"", "questions_for_authors": "--There is still no evidence showing whether the labeling error is sufficiently small in this paper. (Comparing the comment “They didn’t verify whether the labeling error caused by the weak augmentation is sufficiently small” in line 87 and line 88)\n\n-- It appears that the experiments are conducted on only three datasets, all within the computer vision domain. Will the conclusions of this paper also hold for other contrastive learning domains, such as NLP and graph learning? Given the scope of the experimental results, they may not be sufficient to generalize across all contrastive learning methods.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IZJpbCl9el", "reviewer": "Reviewer_ABgp", "summary": "In this paper, the authors provide a detail theoretical analysis on the effect of data augmentation on the downstream classification performance in contrastive learning, where the intra-class and inter-class augmentation overlap are considered. Based on these results, the authors propose to apply SVD on the input images and verify experimentally and theoretically that this could reduce the negative effect from inter-class augmentation overlap. Meanwhile, the authors also show that adopting SVD would also decrease the connectivity of the augmentation graph and then hurt the downstream classification performance. As a remedy, the authors propose to use moderate embedding dimension, which may increase the connectivity of the augmentation graph and then counteract the negative effects of SVD. The experimental results on benchmark datasets support the theoretical findings.\n\n***\n**Update after Rebuttal**\n\nThanks the authors for their detail responses, which have adequately addressed my concerns on the theoretical results. I encourage the authors to include these modified results and the discussions about those related work in the final version.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to you for your valuable comments and constructive suggestions. \n\n**Q1:** A constant 2 appears suddenly. Inequality (4) holds only when inner product is greater than 0.\n\n**A1:** Thanks. The constant 2 is not necessary. We have deleted this constant factor. Therefore, inequality (4) holds without the requirement that the inner product term is greater than 0. We have also corrected the corresponding result in Theorem 4.2.\n\n***\n**Q2:** Line 902-904: why does inequality (3) hold?\n\n**A2:** Thanks. We have corrected the inequality (3) in the anonymous link: https://imgse.com/i/pEsQ5Pe. We have also corrected the corresponding result in Theorem 4.7.\n\n***\n**Q3:** Line 923-925: $1/2\\epsilon_q^2$ appears suddenly.\n\n**A3:** Thanks. We have corrected the formula from line 918 to 925 in the anonymous link: https://imgse.com/i/pEsQ5Pe. We have also corrected the corresponding result in Theorem 4.7.\n\n***\n**Q4:** Some experiments of domain generalization ability.\n\n**A4:** Thanks. In our opinion, this question is about the domain transfer ability from pre-training data to downstream classification data. However, our work assumed the distribution of downstream classification data is the same as the distribution of pre-training data. Therefore, we were unable to study domain transfer ability. We will attempt to mild this assumption to analyze the impact of SVD on domain transfer ability from empirical and theoretical perspectives.\n\nIf our understanding is wrong, please do not hesitate to contact us. We will make some more clear explanations at once.\n\n***\n**Q5:** Cite some works.\n\n**A5:** In our modified manuscript, we have made some discussions about these works you mentioned.\n\n**[1]** proposed the concept of augmented distance and provided some upper bounds revealing the theoretical effect of augmented distance on understream classification performance. Specifically, they found that the classification performance of contrastive SSL is related to three key factors: **alignment of positive samples, divergence of class centers, and concentration of augmented data**. Theorem 2 in our work provided both upper and lower bound, which can not only give similar conclusions but also reveal some additional factors. **Firstly**, the term $V(f(x)|y_{\\bar{x}})$ implies the alignment of positive samples. **Secondly**, the term $V_{y_{\\bar{x}}^{\\neg}}(f(x)|y_{\\bar{x}})$ stems from labeling error caused by data augmentation, which is similar to the concentration of augmented data. **Thirdly**, the term $V(f(x^-)|y^-)$ implies the alignment of negative samples, which is not considered by [1]. **More importantly**, we further improved the bounds of Theorem 2 via data dimensionality reduction and provided the corresponding theoretical analysis and empirical observations (Section 4.2).\n\n**[2]** established a theoretical framework for weakly supervised contrastive learning for the first time. Their results revealed that 1) semi-supervised information improves the error bound compared with purely unsupervised contrastive learning by using all labeled samples; 2) joint training of supervised and unsupervised contrastive learning does not improve the error bound compared with purely supervised or purely unsupervised contrastive learning. Although weakly supervised contrastive learning is not the topic of our work, the labeling error considered by our work is analogous to a type of weak supervision, i.e., noisy-labeled information. Therefore, we will extend the theoretical analysis of this work to weakly supervised contrastive learning in our future work. Besides, [1] and our work both gave the suggestion that we should choose a moderate feature dimension $k$, which enhances the credibility of our suggestion.\n\n[1]Huang et al., Towards the Generalization of Contrastive Self-Supervised Learning. ICLR 2023.\n\n[2]Cui et al., Rethinking Weak Supervision in Helping Contrastive Representation Learning. ICML 2023.\n\n***\n**Q6:** Whether the combination of SVD and moderate embedding dimension would bring positive or negative effects.\n\n**A6:** Thanks. This question is the concern stated in the remark of Theorem 4.9. We use a moderate embedding dimension to mitigate the reduction of the connectivity for the augmentation graph. According to our empirical observations, if we use a moderate embedding dimension, the positive effect of SVD on the downstream classification performance is not offset by the reduction of the connectivity for the augmentation graph at least. We don’t know whether a moderate embedding dimension can completely eliminate this reduction, which is a limitation of our work. We aim to further explore the impact of the moderate embedding dimension in our future work, which was mentioned in the first point of Limitation (Appendix E).\n\n***\n**Q7:** \"SLT-10\"to \"STL-10\".\n\n**A7:** Thanks. We have corrected this typo.\n\n***\n**Q8:** Domain generalization performance of the model?\n\n**A8:** Please see **A4**.", "claims_and_evidence": "The motivation of this work is clear and reasonable. Applying SVD on the input images could filter out the semantic irrelevant information (background) while maintaining the key semantic information (as shown in Figure 4), which reduces the probability of the event that two inter-class images have overlap parts after augmentation (Figure 1) and thus help improve the downstream classification performance. However, since the semantic irrelevant information is remove, it is harder to obtain the same augmented images from the vanilla images by different augmented approaches, indicating that the connectivity of the augmentation graph is reduced. The above claims and intuitions are supported by the experimental results.", "methods_and_evaluation_criteria": "The proposed method are evaluated on benchmark image datasets, which are also adopted in previous contrastive learning studies. Most of the experimental settings also followed previous ones. Therefore, the experimental results are sound and convinced.", "theoretical_claims": "I have carefully checked all the proofs provided in the appendix. The core idea generally follows that of (Wang et al., 2022), and there is no major issues. However, I still have the following concerns: (1) In line 748-752, a constant factor $2$ appears suddenly in terms $\\mathbb{E} _{p(x,y^{\\neg} _{\\bar{x}})}  [f(x)^{\\top}  \\mu _{\\bar{x}}]$ and $\\sqrt{\\mathbb{E} _{p(x,y^{\\neg} _{\\bar{x}})}  [\\Vert f(x)^{\\top}  \\mu _{\\bar{x}} \\Vert^2] }$. It's not clear to me why this constant factor appears. It seems that maintaining the original factor $1$ does not effect the final bounds. Besides, inequality (4) holds only when $\\mathbb{E} _{p(x,y^{\\neg} _{\\bar{x}})}  [f(x)^{\\top}  \\mu _{\\bar{x}}] \\geq 0$. The authors should provide more detailed explanation on this. (2) In line 902-904, the authors claim that inequality (3) could be obtained by $\\Vert f(x) - \\mu _{y _{\\bar{x}}} \\Vert^2 = \\Vert f(x) \\Vert^2 + \\Vert \\mu _{y _{\\bar{x}}} \\Vert^2 -2 f(x)^\\top \\mu _{y _{\\bar{x}}} \\leq \\epsilon^2_q$ and $\\Vert f(x) \\Vert \\leq 1$. However, I can only conclude that $-2 f(x)^\\top \\mu _{y _{\\bar{x}}} \\leq \\epsilon^2_q - (\\Vert f(x) \\Vert^2 + \\Vert \\mu _{y _{\\bar{x}}} \\Vert^2)$ holds. Moreover, if $\\Vert f(x) \\Vert \\geq 1$ and $\\Vert \\mu _{y _{\\bar{x}}} \\Vert \\geq 1$ hold, I can also conclude that $-2 f(x)^\\top \\mu _{y _{\\bar{x}}} \\leq \\epsilon^2_q - 2$. But, inequality (3) holds only when $-2 f(x)^\\top \\mu _{y _{\\bar{x}}} \\geq \\epsilon^2_q - 2 $ holds. (3) In line 910-915, there is no appearance of the term $\\frac{1}{2} \\epsilon^2_q$, yet it appears in line 923-925. Could the authors clarify this?", "experimental_designs_or_analyses": "From my view, the experimental design is reasonable. For the analysis, the authors mainly focus on the downstream classification performance of the model in this work. Indeed, the domain generalization ability is also important. It would be better if the authors could conduct some experiments to analyze how SVD and the moderate embedding dimension technique affects the domain generalization ability.", "relation_to_broader_scientific_literature": "The key contribution of this work is establishing fine-grained theoretical analysis on the effects of data augmentation on the downstream classification performance in contrastive learning, particularly the effects from inter-class augmentation overlap. This could enhance the understanding to the mechanism of contrastive learning. Also, this work could also provide new insights to the computer vision community on how to learn representation via proper data augmentation. The moderate embedding dimension technique may also be helpful for researchers that apply contrastive learning on other formations of data, such as graphs or texts.", "essential_references_not_discussed": "Most of the key studies on the theoretical analysis for contrastive learning are cited in this paper. However, there is also some work that need to be cited and discuss. In [1], the authors introduce the concept of augmented distance to depict the semantic similarity of two augmented images, which could also used to analyze the labeling error. In [2], the authors also analyze how the noisy-label information affect the down\nstream classification error.\n\n[1] Towards the Generalization of Contrastive Self-Supervised Learning. Huang et al., ICLR 2023.\n\n[2] Rethinking Weak Supervision in Helping Contrastive Representation Learning. Cui et al., ICML 2023.", "other_strengths_and_weaknesses": "The strength of this work is providing detailed theoretic analysis on the effects of intra-class and inter-class augmentation overlap on the downstream classification performance, which goes beyond previous work. As for the weakness, although applying SVD could mitigate the inter-class augmentation overlap, it also reduce the connectivity of the augmentation graph. Therefore, it is unclear whether the combination of applying SVD and using moderate embedding dimension would bring positive or negative effects on the downstream classification performance.", "other_comments_or_suggestions": "Typo: \"SLT-10\" in the caption of Figure 4 should be \"STL-10\".", "questions_for_authors": "1. You only consider downstream classification performance in this work. The domain generalization performance or domain transfer ability is also another important criteria. To what extent applying SVD or using moderate embedding dimension would affect the domain generalization performance of the model?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper discusses the labeling error issues in contrastive learning and proposes a remedy through SVD on the inputs. While some reviewers initially had concerns about the scalability and applicability of the proposed method, the authors provided effective feedback with larger scale experiments on TinyImageNet, enhancing the practical applicability. Thus I recommend this paper for acceptance. Since the reviewers provided important feedback on the paper, I expect the authors to incorporate them into the final version.
Decision: Accept (poster)