How Weakly Supervised Information helps Contrastive Learning
Authors: Jingyi Cui
Weiran Huang
Yisen Wang
Venue: Submitted to ICLR 2023 (2023)

Abstract: Contrastive learning has shown outstanding performances in both supervised and unsupervised learning. However, little is known about when and how weakly supervised information helps improve contrastive learning, especially from the theoretical perspective. The major challenge is that the existing theory of contrastive learning based on supervised learning frameworks failed to distinguish between supervised and unsupervised contrastive learning. Therefore, we turn to the unsupervised learning frameworks, and based on the posterior probability of labels, we translate the weakly supervised information into a similarity graph under the framework of spectral clustering. In this paper, we investigate two typical weakly supervised learning problems, noisy label learning, and semi-supervised learning, and analyze their influence on contrastive learning within a unified framework. Specifically, we analyze the effect of weakly supervised information on the augmentation graph of unsupervised contrastive learning, and consequently on its corresponding error bound. Numerical experiments are carried out to verify the theoretical findings.

Under review as a conference paper at ICLR 2023
How WEAKLY SUPERVISED INFORMATION CONTRASTIVE LEARNING
HELPS
Anonymous authors
Paper under double-blind review
ABSTRACT
Contrastive learning has shown outstanding performances in both supervised and unsupervised learning. However, little is known about when and how weakly su- pervised information helps improve contrastive learning, especially from the the- oretical perspective. The major challenge is that the existing theory of contrastive learning based on supervised learning frameworks failed to distinguish between supervised and unsupervised contrastive learning. Therefore, we turn to the un- supervised learning frameworks, and based on the posterior probability of labels, we translate the weakly supervised information into similarity graph under the framework of spectral clustering. In this paper, we investigate two typical weakly supervised learning problems, noisy label learning, and semi-supervised learning, and analyze their influence on contrastive learning within unified framework. Specifically, we analyze the effect of weakly supervised information on the aug- mentation graph of unsupervised contrastive learning, and consequently on its corresponding error bound. Numerical experiments are carried out to verify the theoretical findings.
INTRODUCTION
Contrastive learning has shown state-of-the-art empirical performances in both supervised and unsu- pervised learning. In unsupervised learning, contrastive learning algorithms (Chen all 2020; He caal 2020: Khen et all, 2021: Chen and Hc, (02(T) learn good representations of nigh-dimeesiioal observations from a large amount of unlabeled data, by pulling together an anchor and its aug- mented views in the embedding space. On the other hand, supervised contrastive learning (Khosla alal 2020) uses same-class examples and their corresponding augmentations as positive labels, and achieves significantly better performance than the state-of-the-ant cross entropy loss, especially on large-scale datasets.
Recently, contrastive learning has been introduced to solve weakly supervised learning problems such as noisy label learning (Wan all 2021; Waan et all 2022) and semi-supervised learning. For noisy label learning, most methodological studies use contrastive learning as tool to select confident samples based on the learned representations (Yao all 2021: Ortego et all, 2021; Liet all 2022), whereas the theoretical studies focus on proving the robustness of downstream classifiers with features learned by self-supervised contrastive learning (Cheng all, 2021; Xue all, 2022). For semi-supervised learning, contrastive loss is often used as regularization to improve the precision of pseudo labeling (Lee 2022: Yang all 2022).
However, none of the existing studies use weakly supervised information to improve contrastive learning. Perhaps the closest attempt is Yan (2022), which leverages the negative correlations from the noisy data to avoid same-class negatives for contrastive learning. Nonetheless, there are purely empirical results presented, without showing when and how the weakly supervised infor- mation helps improve contrastive learning. Moreover, a proper theoretical framework of weakly supervised contrastive learning is especially lacking.
The major challenge lies in the fact that the existing theoretical frameworks compatible with both supervised and unsupervised contrastive learning (Arora all, 2019; Nozawa and Sato, 202T; Ash atall 2022: Bao et all 2022) fail to distinguish between the two settings. To be specific, in order to build relationship with supervised learning losses, such studies assume that the positive pairs for unsupervised contrastive learning are generated from the same latent class, and this is exactly how positive samples for supervised contrastive learning selected. Consequently, such mathemati-

---

Under review as a conference paper at ICLR 2023
cal modeling cannot tell the difference between supervised and unsupervised contrastive learning- Therefore, in this paper, we in turn base our theoretical analysis on an unsupervised learning frame- work. Based on the posterior probability of labeled samples, we translate the weakly supervised information into similarity graph under the framework of spectral clustering. This enables to analyze the effect of the label information on the augmentation graph of the unsupervised spectral clustering (HaoChen all, 2021), and consequently on its corresponding error bound.
The contributions of this paper are summarized as follows.
We for the first time establish a theoretical framework for weakly supervised learning con- trastive learning, which is compatible with both noisy label and semi-supervised learning,
By formulating the label information into a similarity graph based on the posterior prob- ability of labels, we derive the downstream error bound of contrastive learning from both weakly supervised labels and feature information. We show that both noisy labels and semi- supervised labels can improve the error bound of unsupervised contrastive learning under certain constraints on the noise rate and labeled sample size.
We empirically verify our theoretical results.
2 RELATED WORKS
Theoretical Frameworks Contrastive Learning. The theoretical frameworks of unsupervised contrastive learning can be divided into two major categories. The first category is devoted to build- ing the relationship between unsupervised contrastive learning and supervised downstream classifi- cation. Arora et all (2019) first introduces the concept of latent classes, hypothesize that semantically similar points are sampled from the same latent class, and proves that the unsupervised contrastive loss serves an upper bound of downstream supervised learning loss. Nozawa and Sato (2021); Ash at (2022);; Bao et all (2122) further investigate the effect of negative samples, and establish sur- rogate bounds for the downstream classification loss that better match the empirical observations on the negative sample size. However, studies in this category have to assume the existence of super- vised latent classes, and that the positive pairs are conditionally independently drawn from the same latent class. This assumption fails to distinguish between supervised and unsupervised contrastive learning, and cannot be used to analyze the weakly supervised setting.
Another major approach is to analyze contrastive learning by modeling the feature similarity. HaoChen et all (202I) first introduces the concept of the augmentation graph to represent the fea- ture similarity the augmented samples, and analyzes contrastive learning from the perspective of spectral clustering. Shen et all (2022) uses a stochastic block model to analyze spectral con- trastive learning for the problem of unsupervised domain adaption. Similarly, WWang all (2021) proposes the concept of augmentation overlap to formulate how the positive samples are aligned. Moreover, contrastive learning is also understood through other existing theoretical frameworks of unsupervised learning, such as nonlinear independent component analysis (ICA) (Zmmeermmn all 20211,, neighborhood component analysis (NCA) (Ko all 2022). variational autoencoder (VAB) (Atchhssnn 202T)) etc.
In this paper, we follow the second category of contrastive learning approaches, and formulate the weakly supervised information into similarity graph based on both label and feature information.
Contrastive Learning for Noisy Label Learning. Ghosh and Lan (2112T)) first finds that pretrain- ing with contrastive learning improves robustness to label noise through empirical evidences. Many methodological studies are carried out for noisy label learning with the help of contrastive learning. Yao all (202T); Ortego et all (202T); Ll (20222 use representations learned from unsupervised contrastive learning to filter out confident samples from all noisy ones, and in turn use the confident samples to conduct supervised contrastive learning to generate better representations. By contrast, Yan all (2022) follows the idea of negative learning (Kim et all 201% 202T), and leverage the negative correlations from the noisy data avoid same-class negatives in contrastive learning. For theoretical studies, Cheng el all (2122) analyzes the robustness of cross-entropy with SSL features, and Xuee el all (2122) proves the robustness of downstream classifier in contrastive learning.
Contrastive Learning for Semi-supervised Learning. Lee et all (2022); Yang et al. (2022) use contrastive regularization enhance the reliability of pseudo-laheling in semi-supervised learning.
2

---

Under review as a conference paper at ICLR 2023
introduces semi-supervisee learning method that combines self- self-supervised con- trastive pre-training and semi-supervised fine-tuning based on augmentation consistency regular- ization. Zhang et al; (2022) uses contrastive loss to model pairwise similarities among samples, generates pseudo labels from the cross entropy loss, and in turn calibrates the prediction distribution of the two branches.
al (2024T
For both noisy label learning and semi-supervised learning tasks, the existing studies all focus using contrastive learning as a tool to improve the weakly supervised learning performance, whereas to the best of our knowledge, none of the previous works show the effect of weak supervision to contrastive learning itself. To fill in the blank, this paper, we establish a theoretical framework for weakly supervised contrastive learning, which is compatible with both noisy label and semi- supervised learning tasks.
3 PRELIMINARIES
Notations. Suppose that random variables X & Rd, and Y E Y natural data { 9: ]EeN]] be i.i.d. sampled from the joint distribution YY. Given natural data E X, we use A(-(z) to denote the distribution of its augmentations and use X' to denote the set of all augmented data, which is assumed to be finite but exponentially large. Denote n = |
(1,...,r), (t. Let the input
3.1 SPECTRAL CONTRASTIVE LEARNING
In HaoChen (202T)), an augmentation graph G is used to describe the distribution of augmented
samples, where the edge weight Waz'' (<>p(((<<)))) A(x=" [ā)] denotes the marginal probability of generating augmented views and x' from the same natural data. Due to the total probability
mass, Ex,'Ex Waz' 1. The adjacent matrix of the augmentation graph is denoted as A
)2.''e E R and the normalized adjacent matrix is denoted as A = D- 1/2 where D == diag(w.)zex, and wa Ev'ex Was'.
In this paper. we consider the spectral contrastive loss proposed by Han Chen et all (2021), that is, for embedding function X R
[()) = 2 Er,z+ [f(x)] [(x)))((x))]] E., (())))
(1)
Spectral contrastive loss is proved to be equivalent to the matrix factorization loss, i... for F e Baxk (uz) U2 w//² f(x),
Lut F) = | FF 13 -(f) const.
3.2 NOISY LABEL LEARNING
Recall that denote the true label of an given instance E X' is y. One common assumption of the generation procedure of label noise is as follows. Given the true labels, the noisy label randomly flipped to another label ÿ with some probability. In this paper, we take the widely adopted symmetric label noise assumption as an example.
For notational simplicity, we write the the symmetric label noise assumption in matrix form. Denote P(Y j|æ:), as the posterior probability matrix of the clean
Y := (11 (v) ny nz(x)
label distribution, and denote Y == (xx))) je[r]. ()) = P(Ý = j|z), as the noisy label distribution. In Assumption II. we assume that the flipping probability is conditional independent of the input data, and that the flipping probability to all other classes are uniformly at random.
Assumption 1. For symmetric label noise with noise rate 7 E (0, 1), we denote the transition matrix T (Er))etri where
for xfrj 7 for
1 = 1- and lij = r
(3)
Then the noisy label posterior distribution assumed be
YTT.
(4)
Under Assumption M1 T is symmetric. Specifically, when 0. T degenerates to the identity matrix Lxxr- Moreover, to guarantee the PAC-learaability. we usually assume the true label the dominating class, i.e. 7 1=1
3

---

Under review as a conference paper at ICLR 2023
3.3 SEMI-SUPERVISED LEARNING
For j E [r]. let ny be the number of labeled samples of Class j. Let n2 = Eeer)) nL.j be the number of all labeled samples, and ny be the number of unlabeled samples. Obviously, we have nz ny
Usually, the number of labeled samples is much smaller than that of the unlabeled because human annotation costly and
That is, we can naturally assume a & ny-
In the following parts of the paper, we analyze the settings of noisy label learning and semi- supervised learning
unified framework. Without loss of generality, we assume (I1,.... Zng is labeled with noise rate y E [0, ² and denote the corresponding clean and noisy posterior prob- ability matrices as YL and Y,. respectively. Then we have Yz = Y,T. Specifically, when ? = 0. our analyzing framework degenerates to the standard setting of semi-supervised learning, and when n n2, our analyzing framework reduces to the standard noisy label learning.
MATHEMATICAL FORMULATIONS
We mention that our formulation of "similarity graph" is not distributional assumption on the un- derlying similarity among data, but to formulate possible probability of drawing positive samples in contrastive learning that takes both label and feature information into consideration. Specifically, in Sections ] and [.21. only discuss the similarity graph describing the weakly supervised labels and neglected feature similarity. Then in Section we take both label and feature similarity into consideration through convex combination.
4.1 SIMILARITY GRAPH DESCRIBING NOISY LABEL INFORMATION
To leverage the labeled information in the form of similarity graph, we first consider simple ex- ample where noise rate true label y. the posterior probability (x)) naturally assume that in the similarity graph describing label information, the intra-class vertices are fully connected and the inter-class vertices are disconnected. That is, Was' 1 if x and x' has the same label and otherwise Wax' 0.
O and the label distribution is deterministic, i.c. for a sample x with and (z) for j * y. In this case, we can
Then we consider the more general stochastic label scenario. Recall that for unsupervised spec- tral contrastive learning, the edge weight War' in an augmentation graph G describes the marginal probability of generating x and x' from the same natural data. That is, Was' describes the joint probability of a pair of positive samples. Similarly, since the positive samples for supervised con- trastive learning (Khosla et all 2020) are selected as all same-class samples, we can naturally define the edge weight War the probability of two views and x' generating from the same class, i.c. Wzx' ΣSE(K] Mi (x) (z)n, (z'), and therefore A == Y Moreover, we denote as the nor- malized adjacent matrix. For the simplicity of notations, we consider the case where the data is class-balanced, i.e. 71
n, RL/T. Then we have A A.
Next, we add label noise to the our mathematical formulations. To be specific, when performing supervised contrastive learning based on noisy labeled data, we naturally select positive samples as the samples with the same noisy labeled data. According to Assumption II we have Y, Y,T, where T is symmetric. Then the adjacent matrix of the similarity graph induced by noisy labels is formulated as
Å :- Y YIT(YT)) = N.TT YI Y.
(S)
Similarly, when data is class balanced, we have the normalized adjacent matrix À, 22 ÀL.
4.2 SIMILARITY GRAPH DESCRIBING
NOISY LABEL INFORMATION
Under the setting of semi-supervised learning, we have no prior knowledge about the label infor- mation of the unlabeled samples. From the perspective of unsupervised contrastive learning, the unlabeled samples can be viewed as having unique class labels. Therefore, to construct the similar- ity graph, we attach sumple-specifee labels to the unlabeled samples. Thus, the posterior probability matrix of unlabeled samples Yu an identity matrix Ing x Note that here we only discuss the similarity graph describing supervised information, so the feature similarity between samples is not included in the similarity graph.
4

---

Under review as a conference paper at ICLR 2023
Combining both labeled and unlabeled samples, the posterior probability matrix of all semi- supervised samples can be denoted as
Yr 0 YU
Y,T = 0
0
Y
(6)
Ine xnu
Therefore, the similarity graph of samples with ni noisy labels can be denoted as
Y
-=
0
Ing xno
In Lemma 11 present the influence of symmetrie label noise with noise rate 7 the similarity graph Á.
Lemma 1. Under Assumption 01 if the data is class balanced, i.e. ni n, ", then there holds
Á = (r, + B(7) = Ing
IT n2
(8)
0
Ino Xnc
where a<7)) = -1--111) and B(7) = 1-1 (2 ,-,
Note that without label noise, i.e. 7 O, have a<1) and ((7) O. For the sake of simplicity, in the following we write a and / instead of a(?) and (()) when no ambiguity aroused.
In Lemma II we show that the effect of symmetric label noise is to add uniform weight to the edges between all labeled samples. This uniform weight increase the confusion between intra- and inter-class similarities, For example, under the deterministic label scenario, we have A, = In x The original intra-class similarity is uniformly shrinked from a and the inter-class similarity increases from 0 to B. Moreover, as the noise rate y increases, a decreases and / increases, which results in severer confusion between the intra- and inter-class similarities.
4.3 SIMILARITY GRAPH DESCRIBING BOTH LABEL AND FEATURE INFORMATION
In this part, we take both label and feature information into consideration. For the feature informa- tion, we denote Ag as the augmentation graph of arbitrary unlabeled samples. For label information, take the similarity graph describing the semi-supervised noisy labeled (augmented) samples À as denoted in equation M When leveraging the use of both label and feature information in contrastive learning, mix the two similarity graphs by convex combination, i.c. for 0 E (0, 1), the mixed similarity graph of all augmented samples Ag, is denoted as
A = (1 Ag AAA
where Ag and À denote the (by row) normalization Ao and Á, respectively. Recall that edge weights of the similarity graphs represent the probability of two augmented views being drawn as pair of positive samples. The similarity graph equation can be understood as selecting positive pairs based on both weakly supervised labels and feature information.
THEORETICAL RESULTS
In this section, we first compute eigenvalues of the similarity graph describing both label and feature information, which plays key role in deriving the error bound of contrastive learning. Then in Section 5. we show that the weakly supervised information helps reduce the error of the best possible linear classifier on the representations learned by weakly supervised contrastive learning.
5.1 EIGENVALUES OF SIMILARITY GRAPH DESCRIBING WEAKLY SUPERVISED LABEL INFORMATION
We first compute the eigenvalues of the similarity graph describing the semi-supervised noisy labels.
5

---

Under review as a conference paper at ICLR 2023
Proposition 1. For arbitrary Y. assume that the labeled data is class-balanced i.e. Lie(ne) Mh (z()))) = nt./r for E [r]. Assume that the eigenvalues of Ar. are pl Po (in de- scending order). Then under Assumption II the eigenvalues of are
in = 1mg+1 =1,
(10)
Bi = p- (1---)) for jorj== j= my ,,...,
(11)
In Proposition II show that the eigenvalues of A rely the cigenvalues of A and consequently rely on the posterior probabilities of clean labels. Specifically, if the true label has higher posterior probability, i.e. maxxe(r] P(Y j|æ>) is larger, then the eigenvalues of A,, are larger. On the other hand, the existence of label noise uniformly shrink the eigenvalues of A except for the largest ones, and larger noise rate y results smaller a and thus leads to smaller eigenvalues of A. Moreover, the number of largest eigenvalues is decided by the number of unlabeled samples.
Note that rank < rank + < ny +r, and therefore we have Ang Specifically, under the deterministic label scenario, we have Mny+2 ... Mnutr = 1. Then the eigenvalues of A become
An =..... Pny+ +1 = 1,
(12)
#ng+2 Un = =0 (11-1-_1))
(13)
Pay +r+1 = ... ihn =
(14)
5.2 EIGENVALUES SIMILARITY GRAPH DESCRIBING BOTH LABEL AND FEATURE INFORMATION
In the following proposition, we discuss the eigenvalues of the mixed similarity graph Ag, describing both weak labels and feature information.
Proposition 2. Denote An
An as the eigenvalues of A0, Then given the eigenvalues of Au i.e. Vi...., Vn and the eigenvalues of AL, i.e. p1...., Hog (in descending order), when k nu. there holds
AR+1 2 max {0 10+0 - -0)mm th,
(0cph44411) + )) (1-0)=+11
max
(15)
when ny < k < ny + there holds
A4t1 2 max
iinn n. + (80patk41-1+ -0)=:
max
(.::).1 (1 0)14+1
(16)
and when l 2 ny + there holds
AK+1 2 (1 0)
(17)
According to Proposition A the lower bound of Specifically, under the deterministic scenario, we have for k nu.
A 2 max {0 + (1 0) Pa + (1 O)v (1 -0)xk+1)
(18)
for ny < k < ny +r,
A max la (1 0) - 0)
(19)
and for k no +r,
A++1 2 (1-00 (1-0)
(20)
We see that under the deterministic scenario, the lower bound of the k 1-th largest eigenvalue AR+1 of Ag,y,n depends on the eigenvalues tng+k. +ng and Uk++1 of the unsupervised augmentation graph Ag. The value of A is also affected by the weighting parameter 0 and the noise rate y. A perhaps anti-intuitive conclusion is that when k is larger than ny r, the lower bound of A is unaffected by the noise rate. That is, when l is large enough, the weak labels will not affect the l + 1-largest eigenvalue of the mixed similarity graph.

---

Under review as a conference paper at ICLR 2023
5.3 WEAK SUPERVISION HELPS REDUCE ERROR BOUND
Recall that the goal of contrastive representation learning is to learn a embedding function f X' + Rt The quality of the learned embedding is often evaluated through linear evaluation. To be specific, denote B E R as the weights of the downstream linear classifier, and the lin- ear predictor is denoted as (T.
t), where J (z) arg ge (f)) B); In this paper, we focus on analyzing the error bound of the best possible downstream linear classifier B' where fopp arg f:X-===> C(f) the minimizer of the population spectral contrastive loss L(f) as defined in equation M1 and 3 is the optimal weight for the downstream linear classifier.
arg (e() (x)A(1))(9)
Following Hao Chen et all (202II, we assume that the labels are recoverable from augmentations, i.e. we assume there exists classifier g that can predict label y(x) given input æ with error at most ò E (0, 1) with function ý = X + [r].
Assumption 2. Let Px be the probability distribution of original input data T. Denote x as an augmented sample and y is its label. Assume that for some 8 > 0, there holds
(+) * v) S 6, E Ar. (8)
(21)
and
E Uni ()) # y) S 5.
(22)
Compared with Assumption 3.5 in Hag hen et all (2021)) Assumption additionally assume the recoverable of labels taking expectation under the uniform probability distribution. We mention that Assumption is minor revision of the original assumption. The additional assumption equation 22 does not change the nature of the original idea of label recovery, and will be used to bound the error term of learning from weakly supervised labels.
Then in the following theorem we can derive the error bound of downstream linear evaluation learned by weakly supervised contrastive learning.
Theorem 1. For arbitrary Y, assume that the labeled data is class-balanced, i.e. Eie(ns)) ny (1(14)= FL/r for E [r] Denote Vt....,Vn as the eigenvalues of Ao (in descending order). Denote E == PZwP p..P...nee A1-AE))) (Fgggp B- (x(x)= # y(ā)) as the linear evaluation error, where 8 E B with norm |B"|| S 1/)x. Then under the deterministic scenario and Assumptions M and LA for k S nu. there holds
2/25 + 0(1 - =
+ 88, E min - 0)(1 +x), (1 0)(1 - Unn tn.++++1r)) + 0(1 a), (1 (1- 0)(1 - MK41) + 0}
(23)
for ny + 1 S k S ny + 1, there holds
2|25 0(1 - a) r-1
1k+1) 3 min 0)(1 - Unn -n + (1 01 (1++) + +0 + +88,
(24)
for k 2 ny + r, there holds
2(25 0(1 - =1]
E < (1 0)(1 -k-1)) +0 + +8,,
(25)
where a = (1 --
By Theorem M. we show that the form of the linear evaluation error bound depends on the dimension of embedding k. whereas the error bound is larger when the noise rate y and the label recovery error 8 gets larger, regardless of the dimension k. Recall that in HaoChen et all (2(21), the error bound of purely unsupervised contrastive learning is i-ve+ + 85. under the setting of standard semi-supervised classification, i.e. when 7 0, and usually k S no, we have
48
+ 86.
(26)
Vnztk
which improves the error bound of purely unsupervised contrastive learning since 1k+1 2 Vnr.k
Next, we discuss the situation when noisy label exists, i.e. 7 > 0.
7

---

Under review as a conference paper at ICLR 2023
For k 2 no +r,
- if - a >
2nd Me+1
then when = 0, there holds
1-2x:1
26
+ 86;
(27)
Vk+1
2ró Mall if 1 S
then when 1, there holds
ux+11
E r 7
(1 a²) + 106.
(28)
For ny +15k k S ny + k ny and 8 m= 2r
Un tntt++1-r), there holds
if
2ró 1111, then when 0,
I-
26
8 S
+ 80;
(29)
vk+1
if a S
2rá Me+1
then when 1/ M t
May +
1-
28 + (1 a) 2/
Pki May
+ 888.
(30)
Uk 1- [kll Mk+1
show that when < ny + r, when a smaller than a certain threshold, the weakly supervised information can improve the downstream error bound by leveraging both label and feature information and by selecting proper weighting parameter However, when the noise rate is large enough, the introduction of noisy labels cannot directly improve the linear evaluation error, regardless of the dimension of the embedding k. Fortunately, for contrastive learning under severe label noise, we can use multiple empirical techniques such as using spatial relationship to select confident samples and reduce noise rate, use the pre-filtered weakly supervised data help improve contrastive learning, and in turn use the improved feature embedding to further reduce label noise. This philosophy has been empirically proved to be effective in many methodology studies (Yao et al 2021; Ortego et all 2021: Li atl 2022).
M which equivalent to the noise rate
1-1++1
EXPERIMENTS
In this section, we aim to empirically verify the our theoretical results that mixing noisy labels and feature information can improve the performance of contrastive learning.
Loss function. Recall that in the theoretical analysis, we investigate the mixed similarity graph A = (1 (1-0) An + OA. By triangle inequality, we have the matrix factorization loss
Cmt (F) = A T.L - FF 1/ = | (1 0) A. + A - FFT 1
<1 A FF 1/3 + 0 FF 11..
(31)
According to equation 2 the spectral contrastive loss is equivalent to the matrix factorization loss. Therefore, experiments, we use the convex combination of supervised and unsupervised con- trastive losses to leverage the noisy label and feature information, i.e. for 0 E (0, 1). we use
Luix = (1 0) Cussup + +0C Cuup-
(32)
Setup. We conduct numerical comparisons the CIFAR- 10 and Tinylmagenet-200 benchmark image dataset (the results on Tinylmagenet-200 can found Appendix (A.2) and follow setting of SimCLR (Chen et all (020) and SupCon (Khosla et all 2020). We use the SGD optimizer and use ResNet-50 as the encoder and a 2-leyer MLP as the projection head. We run experiments on 4 NVIDIA Tesla v100 32GB GPUs. The data augmentations we use are random crop and resize (with random flip), color distortion and color dropping. The models trained with batch size 1024
8

---

Under review as a conference paper at ICLR 2023
and 1000 epochs for each model. We evaluate the self-supervised learned representation by linear evaluation protocol, where linear classifier is trained on the top of the encoder, and regard test accuracy as the performance of the encoder. The symmetric noisy labels are generated by flipping the labels of given proportion of training samples uniformly to one of the other class labels.
In Table II, we compare the performance of unsupervised contrastive learning (SimCLR), supervised contrastive learning (SupCon), and weakly supervised contrastive learning (Mix) under noise rate = 5% and y = 20%. In SimCLR, we neglect all labels the training procedure, and SupCon, select positive samples those with the same noisy annotations. The parameter grid of 0 for Mix is {0.1, 0.2, 0.4, 0.6, 0.8, 0.9, 0.95). The performance comparisons with more noise rates can be found in Appendix A reported.
The best results are marked in bold. The standard deviation is also
Table I: Performance comparisons on CIFAR-10 dataset.
0%
5%
7 20%
SimCLR
92.30
91.99
0.06
91.59 0.06
SupCon
95.48
93.86 + 0.05
90.56 0.05
Mix 95.48
94.39 + 0.03
92.84 0.09
We show in Table II that if the noise rate is small (y = 5%). SupCon results in better performance than SimCLR, whereas when the noise rate raises to 7 20%, the noisy labels actually harm the performance of contrastive learning. Nonetheless, under y = 5% and y = 20%. the weakly supervised Mix outperforms both unsupervised SimCLR and supervised SupCon. This verifies the result in Theorem II that when the noise rate is smaller than certain threshold, leveraging both weakly supervised and feature information helps improve the performance of contrastive learning.
Accuracy
Accuracy
02.1
(a) Noise rate 7 5%.
(b) Noise rate 20%.
Figure 1: Parameter analysis of A.
In Figure 0 we conduct parameter analysis of for the weakly supervised contrastive learning (Mix). We show that as 0 increases from 0 to 1, the performance of Mix first increases and then decreases. Moreover, larger noise rate requires smaller optimal T.e optimal value of 0 for 7 5% is larger than that for = 20%. That is, under severer label noise, less supervised information should be considered in weakly supervised contrastive learning.
7 CONCLUSION
In this paper, we establish theoretical framework for weakly supervised contrastive learning. which is compatible with the settings of both noisy label learning and semi-supervised learning. By for- mulating mixed similarity graph describing both weakly supervised label information and unsu- pervised feature information, we analyze the weakly supervised spectral contrastive learning based the framework of spectral clustering, and derive the downstream linear evaluation error bound. Our theoretical results show that semi-supervised noisy labels improves the downstream error bound when the noise rate is smaller than certain threshold. Our theoretical framework reveals the effect of weak supervision contrastive learning, and has the potential to explain the existing weakly su- pervised learning algorithms based on contrastive learning approaches and to inspire new algorithms. For future works, we will investigate the effect of more complex weak supervision, such active learning and label-dependent label noise, on contrastive learning.
REFERENCES
Aitchison, I.. (2021). Infonce is variational autoencoder. arXiv preprint arXiv:2

---

Under review as a conference paper at ICLR 2023
Arora, S., H. Khandeparkar, M. Khodak, O. Plevrakis, and N. Saunshi (2019). A theoretical analysis of contrastive unsupervised representation learning. In 36th International Conference on Machine Learning, ICML 2019.
Ash, J., S. Goel, A. Krishnamurthy, and D. Misra (2022). Investigating the role of negatives in contrastive representation learning. International Conference on Artificial Intelligence and Statistics. PMLR.
Bao, H., Y. Nagano, and K. Nozawa (2022). On the surrogate gap between contrastive and super- vised losses. In International Conference on Machine Learning. PMLR.
Chen, T. S. Komblith, M. Norouzi, and G. Hinton (2020). A simple framework for contrastive learning of visual representations. In ICML. PMLR.
Chen, X. and K. He (2021). Exploring simple siamese representation learning. In CVPR.
Chen, X., S. Xie, and K. He (2021). An empirical study of training self-supervised vision transform- ers. arXiv preprint wxXiv2204.00052:
Cheng, H., Z. Zhu, X. Sun, and Y. Liu (2021). Demystifying how self-supervised features improve training from noisy labels. arXiv preprint arXiv:2110.09022
Fulton, W. (2000). Eigenvalues, invariant factors, highest weights, and schubert calculus. Bulletin of the American Mathematical Sociery 37(3), 209-14 -49.
Ghosh, A. and A. Lan (2021). Contrastive learning improves model robustness under label noise. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2703-2708.
HaoChen, J. Z., C. Wei, A. Gaidon, and T. Ma (2021). Provable guarantees for self-superviedd deep learning with spectral contrastive loss. Advances in Neural Information Processing Systems 34, 5000-5011.
He, K., H. Fan, Y. Wu, S. Xie, and R. Girshick (2020). Momentum contrast for unsupervised visual representation learning. In CVPR.
Khosla, P., P. Teterwak, C. Wang, A. Sarna, Tian, P. Isola, A. Maschinot, C. Liu, and D. Krishnan (2020). Supervised contrastive learning. Advances in Neural Information Processing Systems.
Kim, B., J. Choo, Y.D.. Kwon, S. Joe, S. Min, and Y. Gwon (2021). Selfmatch: Combin- ing contrastive self-suppriision and consistency for semi-supervised learning. arXiv preprint arXiv:2101.06480
Kim, Y., J. Yim, J. Yun, and Kim (2019). Ninl: Negative learning for noisy labels. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 101-110.
Kim, Y. J. Yun, H. Shon, and J. Kim (2021), Joint negative and positive learning for noisy labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9442-9451.
Ko, C.-Y.. Mohapatra, S. Liu, P.-Y. Chen, L. Daniel, and L. Weng (2022). Revisiting contrastive learning through the lens of neighborhood component analysis: an integrated framework. In International Conference Machine Learning, pp. 11387-1181 11412. PMLR.
Lee, D., S. Kim, I. Kim, Y. Cheon, M. Cho, and W. Han (2022). Contrastive regularization for semi-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3911-3920.
Li, S., X. Xia, Ge, and T. Liu (2022). Selective-supervised contrastive learning with noisy labels. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 316-325.
Nozawa, K. and I. Sato (2021). Understanding negative samples instance discriminative self- supervised representation learning. Advances in Neural Information Processing Systems 34.
10

---

Under review as a conference paper at ICLR 2023
Ortego, D., E. Arazo, P. Albert, N. E. O'Connor, and K. McGuinness (2021). Multi-objective interpolation training for robustness to label noise. Proceedings of the IEEE/CVE Conference Computer Vision and Puttern Recognition, pp. 6606-6615.
Shen, K., R. M. Jones, A. Kumar, S. M. Xie, J. Z. HaoChen, T. Ma, and P. Liang (2022). Connect, not collapse: Explaining contrastive learning for unsupervised domain adaptation. In International Conference on Machine Learning, pp. 19847-19878. PMLR.
Tan, C., J. Xia, L. Wu, and S. Z. Li (2021). Co-learning: Learning from noisy labels with self- supervision. In Proceedings of the 29th ACM International Conference on Multimedia, pp. 1405- 1413.
Wang, K., X. Peng, S. Yang, J. Yang, Z. Zhu, X. Wang, and Y. You (2022). Reliable label correction is good booster when learning with extremely noisy labels. arXiv preprint arXiv:2205.0011 00166...........
Wang, Y., Q. Zhang, Y. Wang, J. Yang, and Z. Lin (2021). Chaos is ladder: A new theoretical understanding of contrastive learning via augmentation overlap. In International Conference on Learning Representations.
Xue, Y., K. Whitecross, and B. Mirzasoleiman (2022). Investigating why contrastive learning bene- fits robustness against label noise. arXiv preprint vrXiv:2201.12495
Yan, J., Luo, C. Xu, C. Deng, and H. Huang (2022). Noise also useful: Negative correlation- steered latent contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 31-40.
Yang, F., K. Wu, S. Zhang, G. Jiang, Y. Liu, F. Zheng, W. Zhang, C. Wang. and L. Zeng (2022). Class-aware contrastive semi-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14421-14430.
Yao, Y., Z. Sun, C. Zhang, F. Shen, Q. Wu, J. Zhang, and Z. Tang (2021). Jo-src: A contrastive approach for combating noisy labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5192-5201.
Zhang, Y., X. Zhang, J. Li, R. Qiu, H. Xu, and Q. Tian (2022). Semi-supervised contrastive learning with similarity co-calibration. IEEE Transactions on Multimedia.
Zimmermann, R. S., Y. Sharma, S. Schneider, M. Bethge, and W. Brendel (2021). Contrastive learning inverts the data generating process. arXiv preprint arXiv: 102.00800

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 PROOFS
Proof of Lemma W1 Under Assumption we have
(1 (1-)))++
i=
(T' =
- (r- 1)3, i
(1-7)² 1 1)), i=j
r- 2 r
r- - it
(33)
That is, we have
T = (1 - 7) + 72 //(r - 1) r-
2 - - - Ixx
2 r r-1 I,,,
r
1 --
2
Lexr r-
7 -
- I,I
r
= alxxx + 31,11,
(34)
Given 7 E [0, 1), we have
A, YIT = Ya + (I,IT) Yy
=YYY, I.
= =AA, + BI IT..
(35)
where the last equality holds because E, (x,()) for i E [m]]
and the normalized augmentation graph
-=
1/2 1/2 AD 1/2 1D21/2
(36)
where
D Inee xne
D
(37)
Dr diag(d)),
(38)
and
d= Σ A
Σ
Σ ne(xx)ne(Tr)) + +
jE(mn]
jE|mi] (E(r)
= a Σ ne(x)))
Σ
ne(g)) + n1 = a Σ
ne(x:)m4 + n1,3
(39)
EEIr)
SE|mc]
EE(r]
Specifically, when the labeled data is class-balanced, i.c. = =n, nn/r. Then we have
nL d
ne(o.)
n n + nB
(40)
r EE(r]
and thus
2
r Ar,
+
nz n
π
0
(41)
0
I Xnu
12

---

Under review as a conference paper at ICLR 2023
Proof of Proposition 0 We first prove that v1 ynz eigenvalue μ사 1. To be specific,
Inc. is an eigenvector of A =
As with
na
A
Inz
Ing r Y Y
r Ar.
VRL
VnL
nt
VnL a
YL 1 I,-
VnL
n.
Ing. VTT.
(42)
where the second last equality is due to class balance, i.e. Eie(nal (z(2;) n2/r for [r]. and the last equality holds because Eje(r) 7y for i E [mz]. Therefore, we can rewrite AL as
0
Vny RL
Âg =
Vnn. 12 ......... Une
Li
1/2
0
U2
(43)
:
Pmn
Unp
Note that I IT, m no
can be decomposed
1
(G
In nL
Inz
Inz)
VnL
/nL
1T VRL
RL
0
- Ine ... Una
U/
(44)
Vnr.
Then have
Å =
r
Ar + r33
In IT
-nc
1
TLL
a
V 172
0
ajf2
I Ing: U2
Unz
VLL
affin
Un
T
0
07
IT
0
IIne 02 .......,80. Unu
VUL
0
+ r
0
TT
VRL
1 VOL
Imr. Unz
0
ajt2
0
1/2
(45)
Unl
Since + r8 = 1, the eigenvalues of A, are 1, app, .... afpin Thus the eigenvalues of
A
A 0
0
(46)
Ing xny
13

---

Under review as a conference paper at ICLR 2023
An ... inne +1 1, P. = apj, for j = nu + ,,.
(47)
(48)
Proof of Proposition M By equation 13 in Fulton (2000)) for two real symmetric n by n matrix (1 Ao and AA,, the k + 1-th largest eigenvalue of Ag,Am (1 0) + 0AA can take any value in the interval
(1 0)us + Big S d min (1 0)u; 8p- i+j=k+2
(49)
++j=++++1
By Proposition we have
1,
j 1, nu + 1;
By
afji
j ny ................................ , nT T;
(50)
0,
j = nu + r + 1, ,.... n.
Therefore, have
max
(1 0)v, + -pj
(4 stj=n+k--
max
(1 (1-)) + 0fn+++1-1i
1
0 ++ (1 0)4r:
i == k, ..,, Y2;
max
0astntt11 + (1 0)vr, = + k-k-a - + 1,... nT + k - 1;
(1 B)V;
i= 1,... ar + k
0 + (1 0)
= max
0apm+kk1-1st (1 Olu,, i= + - 7 I.... nr + 1;
(51)
(1 0)v++1
where the last equality holds because bel ranked in descending order. Then when k S ny,
Ac+1 2 max {0 + (1 11-0)pp O)Une ths in ny 4h-1 (00pp+4411 + (1 O))s (1 0) }.
max
(52)
when ny < k < ny + r,
AR+1 2 max { (1 0)
max i=ny k Il.... (aapntkti-i (1 0)v;}}
(53)
and when ny + r.
A 2 - 0)
(54)
Proof of Theorem M By Lemma B.3 of HaoChen et al (2021), for any labeling function y X + (r), there exists linear probe B* Brak with norm B || S 1/Ae such that
208
P PEMPR (Dimp R- (x) + u(t)
<
85,
(55)
A6+1
where according to the definition of Ag.
σC 0)
Σ (An).. 1bx + [(x;)]
Σ
A An.1(0(2z)) + (xxx))
(56)
[JE(M]
(JE(M)
We investigate the RHS of equation respectively. The first term
Σ (Ao)., (i(x)) # ((,;))
1.j
14

---

Under review as a conference paper at ICLR 2023
Σ
En Pa
A0n (())]
[z;
+ (T)) + 1(((z)) + (())))
2
Σ
A * 0
i.jiml
25.
(57)
The second term is
1
Σ
A 18(x.)) # (0,))
n
1
Σ
(A (0(()))) + [(,5)] + nh
E (0(a,,) * ((+))]
nL 1.EE(1))
+ 2
1
Σ
(0(a)) + ú(x:)]]
(58)
According to the definition of A, the last two terms are equal to O. Then by Lemma II the second term on the RHS of equation 56 becomes
1
Σ
A 1((();) + [(1;)]
Th. M.E(m]
1
E
(aAi. + +3- Ine ITe ) 11j(xx)) # [(2))]
nL (JE(mL)
n
Σ
Σ me(xs)me(ss,) (10((a)) (1 + $ 17(),) # #0)
L t.jE(m)) EEIr]
(10(x,,) * F yil 10(())) + us + 18 + wil)
+ 8 1
(.jE(mn)
r
2 n
Te(x:) E
1
EE|r
sE|nr.
+
8
r
2
1[i(Iz) # v/] + Σ
1lvi + vull
nT.
kE(mz]
(JE(mZ)
2a
Σ Σ
ne(x))1((x;) + f 23 r
1[7(x;)) + vil 3(r 1).
(59)
n1. iE(me] KE[r]
nr ie(mz]
Under the deterministic scenario, we have
Σ
([(()) + ((x;))
1
1 Σ
2a 1[;(x;; # 0 + 28r
1[Aa,t) # U1l + 3(r 1)
TL sE|na]
n2 vE[mz]
2n5 + 2,3rō + 3(r - 1)
r-1
2ad 211-a)6 a + (1 a)
r
28 + (1 a) r-1
(60)
where the second last equation holds due to a + r3 1. Combining equation 5d equation 57 and equation 50, we have
Φ < 00-6 + (1 a) a)"= 1 )
= =2 + 8(1 a) r
(61)
r
15

---

Under review as a conference paper at ICLR 2023
Therefore, by equation 55 we have
2 + 0(1 aa)
= P=PP (Iemp. n- (x) # y(3) <
86,
(62)
1 Ak+1
Combined with Proposition H. we have for k Σ ny, there holds
2 0(1 (x) 0=
3 min (1 - 0)(1 Von +k); (1 0) Vmn Pni+k+l-r) + -1) - Mk+1) + +0}
+ 88,
(63)
for ny + 1 s k $ ny + 1, there holds
2/25 0(1 - 1=1 ==
83 min (1 (1 0) Un1 +b+1 011 a), (1 0) vk+1) + + 86,
(64)
and for & 2 no there holds
2|25 0(1 - a
3 A (1 0)(1 1%-1) + 86.
(65)
A.2 ADDITIONAL EXPERIMENTS
We run additional experiments on CIFAR-10 dataset with a noise rate varying from 0% to 60% in Table C The parameter grid of for Mix is (0.05, 0.1, 0.2, 0.4, 0.6, 0.8, 0.9, 0.95). Table B that Mix consistently outperforms SimCLR and SupCon.
Table 2: Additional Performance comparisons on CIFAR-10 dataset.
0%
5%
10%
7 20%
30%
7
40%
7 50%
? 60%
SimCLR
92.30
91.99
91.63
91.59
90.95
90.42
90.18
88,92
SupCon
95.48
93.86
92.62
90.56
87.76 76
84.80
80.56
74.90
Mix
95.48
94.39
93.12
92.84
91.89
91.74
90.80
90.66
We run additional experimental comparisons on the Tinylmagenet-200 dataset with noise rate y 0.4. The parameter grid of is (0.1, 0.4, 0.6}. We additionally adopt Gaussian Blur for data augmentation and keep the other experimental setups same as Table MI The results are presented in Table E We show that Mix outperforms both SimCLR and SupCon on the 20n dataset.
Table Performance comparisons on Tinylmagenet-200 dataset.
7 = 0%
7 = 40%
SimCLR
53.78 + 0.13
49.79 + 0.05
SupCon
63.79 + 0.23
50.24 + 0.09
Mix
63.79 0.23
53.40 + 0.05
16


Reviews:
[{"review_id": "jFX3fiERMu", "reviewer": "Reviewer_V7qm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "++++++++ After Rebuttal +++++++++++++\n\nIn the rebuttal and follow up discussion, the author(s) have clarified my questions and provided additional empirical results to support their claims. I am adjusting the score accordingly. \n\n(Author(s) please make sure the MIX loss are clearly defined in the revised manuscript.)\n\n+++++++++++++++++++++++++++++++++\n\nBuilding on the theoretical framework of the spectral contrastive learning, this paper studies the theory for weakly supervised contrastive learning. In particular, the author(s) presented a unified contrastive learning framework for two typical weakly supervised learning setups: noisy label learning and semi-supervised learning, and showed that under mild noise level both settings can improve the error bound of unsupervised contrastive learning with a carefully chosen hyper-parameter. Some empirical evidence is provided to support the claims. ", "summary_of_the_review": "Overall this paper is written with clarity. My major reservation is due to two points: (a) the key conclusions are aligned with intuition and they do not lend additional insights to better guide practice; (b) the comparison reported in the experiments is not aligned with the theory developed here (see my comments in the weakness, basically the author(s) used InfoNCE as baseline when they use have used SpecNCE). \n\nI look forward to hearing the author(s) thoughts during the rebuttal discussion. Should more convincing argument or empirical results surface I will be happy to adjust my score. \n", "strength_and_weaknesses": "### Strength\n* This paper is well-written, relevant concepts and notations are clearly defined. Although much of content is very technically involved, the author(s) have accompanied these sections with remarks and analyses that are more accessible to non-expert audience. \n* Studying the theoretical foundation for semi/weakly-supervised contrastive learning is a timely topic. This provides theoretical guidance most closely related to the popular pre-training + fine-tuning paradigm. \n* Excellent discussion of the relevant literature, the coverage is adequate and up-to-date. That said, I think [A] will make a nice addition to the list, which discussed more generic settings of weak supervision. \n\n\n\n### Weakness\n* Experiments are a bit loose. \n    * It is not clear how the author(s) make the labeled/unlabeled split in the experiments. I am assuming SimCLR and SupCon have used all data, and Mix also used all data both as labeled and unlabeled. If that is the case, then this is different from the setting used in the theoretical analysis, where labeled/unlabeled do not overlap. \n    * Not an apple-to-apple comparison. The chosen baselines (SimCLR and SupCon) used the InfoNCE, while Mix used SpecNCE. It is well known that InfoNCE has some issues which motivates many follow up works [B, C, D, E], including SpecNCE. So the gain in Mix may not come from the theoretical predictions made in this paper, but rather gains from SepcNCE itself.\n    * Only Cifar-10 results are reported, which is insufficient by current standards. The author(s) should at least include miniimagenet or tieredimagenet results. \n    * The author(s) should also characterize the variability of the results, as the reported gains could be well within normal statistical fluctuations. While I understand pre-training can be costly, at least repeat the experiments on smaller datasets such as MNIST and report the standard deviation for the accuracy. \n* To summarize the key take-away: (1) adding labeled data on top of self-supervised learning improves accuracy; (2) if the labels are too noisy, then it will wash away the information resulting no improvement. Although I appreciate the theoretical rigor, these are a bit straightforwardly intuitive and practitioners have long been practicing based on similar hunches. \n\n#### References\n[A] Strength from weakness: Fast learning using weak supervision. ICML, 2020 \n\n[B] Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCE\n\n[C] Provable guarantees for self-supervised deep learning with spectral contrastive loss. NeurIPS 2021\n\n[D] Provable Stochastic Optimization for Global Contrastive Learning: Small Batch Does Not Harm Performance. ICML 2022\n\n[E] EqCo: Equivalent Rules for Self-supervised Contrastive Learning\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "eJYx6Vyux7", "reviewer": "Reviewer_pQcs", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Contrastive Learning (CL) has become popular for both unsupervised and fully supervised settings. Recent empirical studies have indicated that CL can also be beneficial under weakly (ex noisy label) or semi-supervised settings. \n\nThis paper studies CL under weakly supervised setting and proposed an approach Mix where they propose to use a convex combination of unsupervised (infoNCE) and supervised contrastive loss (supcon): \\theta infoNCE + (1 - \\theta) supcon. \n\nThey analyze this algorithm using tools from spectral contrastive learning Haochen et.al. and show some empirical validation comparing with supcon and infoNCE on cifar and imagenet benchmark.", "summary_of_the_review": "The theoretical contributions are somewhat novel , but I feel the paper lacks algorithmic novelty (or rather they do not position the contribution w.r.t relevant literature ex - semi supervised , consistency regularization literature where similar algorithm has been used). \nEmpirical evidence is somewhat lacking - especially does not flesh out the strategy to select theta , make this algorithm work well in reality. \n\nFirtsly, cheers to the authors on tackling this important problem in a systematic way -  Overall, I feel the authors might want to dive deeper and investigate further and submit a stronger paper in next ML venue - will be looking forward to the findings. ", "strength_and_weaknesses": "Strength:\n--------------\nExtending the theoretical framework on Haochen et.al. to weak supervision ~ by leveraging stochastic edges with prob \\theta between samples. \n\nusing a combination of supcon and infoNCE is done before for semi-supervised setting (suNCEt:   https://arxiv.org/pdf/2006.10803.pdf )  and biased weak supervision (Positive Unlabeled Contrastive: https://arxiv.org/pdf/2206.01206.pdf )\n\nWeakness :\n----------------------\n1. The empirical evaluation is limited - only on CIFAR-10 comparing infoNCE, supCon and the proposed approach. \nI feel we need further empirical evidence over multiple datasets and under various noise levels. \n\n2. This loss is very similar to \"consistency regularization\" where one would add a semantic consistency term (eg. infoNCE loss) with supervised loss (CE) see for example: \nhttps://arxiv.org/pdf/2011.01403\nhttps://arxiv.org/pdf/2106.08226\nhttps://arxiv.org/pdf/2010.07835\n\nIsn't this equivalent to https://arxiv.org/pdf/2011.01403 where instead of cvx comb of supcon and infoNCE they use cvx combination of CE (label consistency) and supcon / infoNCE etc (semantic consistency) -- since they tackle finetuning. \n\nIsn't Mix just applying the same idea of enforcing semantic consistency by using label agnostic infoNCE ? \n\n3. The parameter \\theta is critical for the success of the method - it should be dependent on \\gamma the noise rate. as pointed out by authors (and is the obvious intuition) that as noise rate is higher we should rely more on self supervision - which is label agnostic hence robust. what would be the dependence ? For example on Table 2 the authors show performance of Mix to be somewhat robust for different noise levels - it is critical for understanding the merit of the method to show the optimal \\theta used for different values of \\gamma. When does \\theta_opt go to 0 - when all labels are corrupted ? a plot of \\theta_optimal vs \\gamma over multiple datasets is going to be helpful.\n- 3.a. From Figure 1 it seems using Mix has little effect when \\gamma is slightly high - for example at \\gamma = 20% theta_opt = 0.1 whereas \\gamma = 5% theta_opt = 0.9 so on slightly high gamma \\theta -> 0 i.e. using infoNCE. \n\nI am really ** Surprised ** by your Table 2 results especially the gap with infoNCE at large noise levels - which such small theta  \n\n4. Linear Probing: How did you linear probe ? on held out clean data ? or on noisy data. If on noise data - what loss did you use for linear probing ? since standard CE would result in performance degradation even if the encoder is high quality. \n-- 4.a. As seen from Table 2 SimCLR results it seems you linear probed on noisy data since the infoNCE-encoder is identical for different \\gamma. \n-- 4.b How about you linear probe on clean data ? i.e. the encoder is trained on noisy supervision but the linear head is trained with clean \n \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "X8IZHEVuin", "reviewer": "Reviewer_v3Am", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In contrast to existing works on weakly supervised contrastive learning, this paper considers how weakly supervised information such as noisy labels and semi-supervised labels affects the downstream performance of contrastive learning. The theoretical analysis provided in this paper is based on the spectral contrastive loss and augmentation graph, a recent theoretical framework proposed by HaoChen et al. (2021). The authors incorporate weakly supervised information into the augmentation graph by using a convex combination of label and feature similarity adjacent matrices. As a result, it is shown that the downstream linear classification error can be improved unless the noise rate is moderate and the representation dimension is not very large.", "summary_of_the_review": "The authors provide and demonstrate an interesting way to leverage weakly supervised information in contrastive learning by extending the framework of unsupervised contrastive learning. Nevertheless, there are several points that could be improved in the final version as pointed out in the above sections.", "strength_and_weaknesses": "### Strengths\n\nThis paper seems to be the first one to consider how weak supervision helps contrastive learning, not the other way around, how contrastive learning can elicit useful information for weakly supervised learning. In practice, it is a natural scenario to do contrastive learning with a bunch of unreliable data, which aligns with the main goal of this paper.\n\n### Weaknesses\n\nIn contrast to the original augmentation graph discussed in HaoChen et al. (2021), the formulation discussed in this paper looks slightly strange and puzzling.\n\n1. At the beginning of Section 3, $n$ is defined to be the number of training data, but soon after $n$ is assumed to be the domain cardinality $|\\\\mathcal{X}|$. Consequently, the training data covers the entire domain $\\\\mathcal{X}$ exactly, which is already a little bit unrealistic (but this may be fine if we focus only on the analysis of the expected loss).\n2. In Section 3.3, $n_L + n_U = n$ is assumed, but this could also be strange because it is unnatural that the number of labeled and unlabeled data is exactly the same as the domain size $n = |\\\\mathcal{X}|$.\n3. The authors observe the different behaviors of the eigenvalues and error bound depending on the relationship between $n_U$ and $k$, the representation dimension. I suppose that $n = n_L + n_U$ is the source of these behaviors, which would not be intrinsic to contrastive learning itself. Indeed, it could hardly happen that \"when $k$ is larger than $n_U + r$, the lower bound of $\\\\lambda_{k+1}$ is unaffected by the noise rate\" (Section 5.2). If it does happen, demonstrating that the downstream performance improves with larger $k$ empirically would be an important step for supporting the validity.\n4. In Section 4.2, the posterior probability matrix for unlabeled data $\\\\mathbf{Y}_U$ is assumed to be a $(n_U \\\\times n_U)$-matrix, but I believe it should be a $(n_U \\\\times r)$-matrix. Accordingly, $\\\\tilde{\\\\mathbf{Y}}$ in equation (6) should be $(n_L + n_U) \\\\times r$. Moreover, $\\\\mathbf{Y}_U$ must not be an identity matrix because it is a posterior probability matrix and hence satisfies the sum-to-one constraint.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "U5SrmHyDgs7", "reviewer": "Reviewer_kEnG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work aims at using weakly supervised information to improve contrastive learning. Following the work of HaoChen et al. (2021), this work also defines spectral contrastive loss and shows the graph under noisy and semi-supervised noisy conditions. After theoretical analysis, a mixed loss is verified on CIFAR-10.   ", "summary_of_the_review": "The motivation is clear and the topic is interesting, but the work seems incremental. The experiment part is weak. It will be better to see more experiments.", "strength_and_weaknesses": "This is a relatively interesting topic. The whole paper is well-written. The logic flow is very clear. Two citations should be considered in the Introduction together with Yan et al. (2022). \nhttps://arxiv.org/pdf/2205.00186.pdf\nhttps://arxiv.org/pdf/2108.04063.pdf\n\nOverall, the experiments are weak. Only one dataset and two baselines are used. In table 1, the proposed method decreases as the others. So why not showing more results, e.g., step by 5% from 0 to 1? It will be more convincing to see the robustness of the proposed method. Why no result at 0%? SimCLR did you use InfoNCE? If so experiment part should include the eq. 32 by plugging in InfoNCE. Did you run 1000 epochs for all the experiments? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject