A Theory-Driven Self-Labeling Refinement Method for Contrastive Representation Learning
Authors: Pan Zhou
Caiming Xiong
Xiaotong Yuan
Steven Hoi
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: self-supervised learning
contrastive learning

Abstract: For an image  query, unsupervised contrastive learning  labels crops of  the same image as positives,  and other image crops as  negatives. Although intuitive, such a native label assignment strategy cannot reveal the underlying semantic similarity between a  query and  its positives and negatives, and impairs performance,  since some negatives are  semantically similar to  the query or even share the same semantic class as the query.  In this work, we first  prove that for  contrastive learning,  inaccurate label assignment heavily  impairs its generalization for semantic instance discrimination, while accurate labels  benefit its generalization.  Inspired by this theory, we  propose   a novel self-labeling refinement approach for contrastive learning. It improves the label quality via two complementary  modules:  (i)  self-labeling refinery (SLR) to  generate accurate labels and (ii)  momentum mixup (MM)  to enhance similarity between query and its positive. SLR uses a positive of a query to estimate  semantic similarity between  a query and its positive and negatives, and  combines estimated similarity with  vanilla label assignment in contrastive learning to  iteratively generate  more accurate and informative soft labels. We theoretically show that our SLR can exactly recover the true semantic  labels of  label-corrupted  data, and  supervises   networks to achieve zero prediction  error on classification tasks.  MM randomly  combines   queries and  positives to increase  semantic similarity between the generated virtual queries and their positives so as to improves label accuracy.  Experimental results on CIFAR10,  ImageNet, VOC and COCO show the effectiveness of our method.  

A Theory-Driven Self-Labeling Refinement Method for Contrastive Representation Learning
Pan Zhou'
Caiming Xiong'
Xiao-Tong Yuan Salesforce Research
Steven Hoi"
Nanjing University of Information Science & Technology
panzhou3 @ gmm
[exiong, shoi) @ @alesforce.com xtyuan nuist.edu.cn
Abstract
For an image query, unsupervised contrastive learning labels crops of the same image as positives, and other image crops negatives, Although intuitive, such a native label assignment strategy cannot reveal the underlying semantic similarity between query and its positives and negatives, and impairs performance, since some negatives are semantically similar the query even share the same semantic class as the query. In this work, we first prove that for contrastive learning, inaccurate label assignment heavily impairs generalization for semantic instance discrimination, while accurate labels benefit its generalization. Inspired by this theory, we propose a novel self-labeling refinement approach for contrastive learning. It improves the label quality via two complementary modules: (i) self- labeling refinery (SLR) to generate accurate labels and (ii) momentum mixup (MM) enhance similarity between query and its positive. SLR uses positive of query to estimate semantic similarity between a query and its positive and negatives, and combines estimated similarity with vanilla label assignment in contrastive learning to iteratively generate more accurate and informative soft labels. We theoretically show that our SLR can exactly recover the true semantic labels of label-corrupted data, and supervises networks to achieve zero prediction error on classification tasks. MM randomly combines queries and positives to increase semantic similarity between the generated virtual queries and their positives so as to improves label accuracy. Experimental results on CIFARIO, ImageNet, VOC and COCO show the effectiveness of our method.
Introduction
Self-supervised learning (SSL) an effective approach to learn features without manual annotations, with great success witnessed to many downstream tasks, e.g. image classification and object detec- tion [1-7]. The methodology
SSL to construct a pretext task that can obtain data labels via well designing the task itself, and then build network learn from these tasks. For instance, by constructing jigsaw puzzle [8], spatial arrangement identification [9], orientation [10], or chromatic channels [11] as pretext task, SSL learns high-qualified features from the pretext task that can well transfer to downstream tasks. As it gets rid of the manual annotation requirement in supervised deep learning, SSL has been widely attracted increasing researching interests [1, 12].
As leading approach SSL, contrastive learning [1, 4, 13-17] constructs a novel instance discrimi- nation pretext task train network that the representations of different crops (augmentations) of the same instance are close, while representations of different instances are far from each other. Specifically, for an image crop query, it randomly augments the same image to obtain positive, and view other image crops as negatives. Then constructs a one-hot label over the positive and negatives to pull the query together with its positive and push the query away its negatives in the feature space.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Motivation. But the one-hot labels in contrastive learning are indeed inaccurate and uninformative because of the following two reasons. Firstly, for query, it could be semantically similar or even more similar to some negatives than its positives. Indeed, some negatives even belong to the same semantic class as the query [1 8-20]. It holds in practice, as (i) to achieve good performance, one often uses sufficient negatives that are much more than the semantic class number, e.g. tens of thousands of negatives for ImageNet [21] in MoCo [I], unavoidably leading to the issue on negatives; (ii) even for the same image, especially for image containing different objects which occurs in ImageNet, random augmentations, e.g. crop, provide crops with (slightly) different semantic information, and thus some of the huge negatives could be more similar to query. Secondly, samples from different classes also have some similarity which is not characterized in the one-hot labels. For example, given a query cat, it is often more similar to dog than car though both dog and car are negatives. Learning from those similarity among samples is also important and can improve the performance. This point is also supported by knowledge distillation a teacher model or a currently-training model is used to predict semantic similarity of a sample on different classes for further supervising model training, achieving better performance. Therefore, the one-hot label cannot well reveal the semantic similarity between query and its positíves and "negatives", and cannot guarantee the semantically similar samples close each other, leading to performance degradation.
self-training approaches [22, 23], where
Contributions. In this work, we alleviate the above label issue, and derive some new results and alternatives for contrastive learning. Particularly, we theoretically show that inaccurate labels impair the performance of contrastive learning. Then we propose self-labeling refinement method to obtain more accurate labels for contrastive learning. Our main contributions are highlighted below.
Our first contribution is proving that the generalization error of MoCo for instance discrimination linearly depends on the discrepancy between the estimated labels (e.g. one-hot labels) in MoCo and the true labels that really reflect semantical similarity between query and its positives and negatives.
Formally, given n training queries D= { with estimated labels {3r) (e.g. one-hot labels in MoCo) and ground truth labels {y; jm their corresponding positives and negatives, the general- ization error of MoCo for instance discrimination is lower bounded by O |/y y ll2]) where ED Illy y [|]l]= - Ily,
1ly..yy. y ll2, and is upper bounded by O ()(n((F)) + +ED Illy- y y²112)).
where |F] is the covering number of the network hypotheses in MoCo. means that the more accurate of the estimated labels } the better generalization of MoCo for instance discrimination.
Inspired by our theory, we propose a Self-IAbeliNg rEfinement (SANE) method which iteratively employs the network and data themselves to generate more accurate and informative soft labels for contrastive learning. SANE has two complementary modules: (i) Leff-Labeiing Refinery (SLR) to explicitly generate accurate labels, and (ii) Momentum Mixup (MM) to increase similarity between query and its positive and implicitly improve label accuracy. Given query, SLR uses its one positive to estimate semantic similarity between the query and its keys (i.e. its positive and negatives) by computing their feature similarity, since query and its positive come from the same image and should have close similarity on the same keys. Then SLR linearly combines the estimated similarity of query with its vanilla one-hot label in contrastive learning to iteratively generate more accurate and informative soft labels. Our strategy is that at the early training stage, one-hot label has heavy combination weight to provide relatively accurate labels; along with more training, the estimated similarity becomes more accurate and informative, and its combination weight becomes larger as it explores useful underlying semantic information between the query and its keys that is missing in the one labels. Besides, we prove that when the semantic labels in the instance discrimination task are corrupted, our SLR can exactly recover the true semantic labels of training data, and networks trained with our SLR can exactly predict the true semantic labels of test samples.
Moreover, we introduce MM for contrastive learning further reduce the possible label noise and also increase augmentation diversity. Specifically, we randomly combines queries {æ.} _ and their positives (ã. 2=1 as x = 0z, + (1= and estimate their labels as y 0g, + (1-0)g,. where indexes i and k are randomly selected, D, is the label of both D; and w. estimated by our label refinery, and 8 (0, 1) is random variable. In this way, the component T& in the virtual query x directly increases the similarity between the query of and the positive key Tg. So the label weight (1 - #) of label y on positive key H; to bring x and T& together relatively accurate, as x really contains the semantic information of Æx. Meanwhile, the possible noise at the remaining positions of label y/ is scaled by 0 and becomes smaller. In this way, MM also improves the label quality.
2

---

Other Related Work. To estimate similarity between a query and its negatives, Wei et al. [20] approximated the similarity by computing cosine similarity between a positive and its negatives. and directly replaced the one-hot label for instance discrimination. Wang el al. [12] used similar similarity estimated on weak augmentations to supervise the learning of strong augmentations. In contrast, we respectively estimate the similarities of the query on all contrastive keys ( its positive and negatives) and on only negatives, and linearly combines two estimated similarities with vanilla one-hot label to obtain more accurate and informative label with provable performance guarantee. Learning from noisy label, e.g. [24-26] also uses soft labels generalized by a network to denoise crop labels and supervise representation learning, and often focus (semi-)superrisee learning that differs from our self-supervised learning.
Two relevant works [27, 28] performed vanilla mixup on all query instances to increase data diversity. Differently, our momentum mixup mainly aims to reduce label noise, as randomly combines one query with one positive (instead of one query) of other instances to increase the similarity between the query and its its positive. Verma et al. [29] showed that mixup is a better aagnosi noise than Gaussian noise for positive pair construction. But they did not perform mixup on labels, which is contrast to [27, 28] and ours. See more discussion in Sec. 3.2 and empirical comparison in See. 4.3.
2 Inspiration: A Generalization Analysis of MoCo
In this section, we first briefly review the MoCo [1] method popularly studied for contrastive learning, and then analyze the impact of inaccurate label assignment on its generalization ability.
Review of MoCo. The MoCo method contains an online network fw and a target network ge receptively parameterized by w and &. Both fur and g consists of a feature encoder and a projection head (e.g. 3-layered MLP). Given a minibatch {c:);=] at each iteration, it first randomly augments each vanilla image c, into two views (æ,, T. and optimizes the following contrastive loss:
(w( = 1 8
Σ
O 2.,)
log Oxx Ei oex bi)
(I)
where (2;, ")) = exp +w))(=))) g(a))lla) with a temperature T. The dictionary B 3 (b;} y denotes the negative keys of current minibatch queries { and is often of huge size to achieve satisfactory performance. e.g. 65,536 in MoCo. In practice, B in MoCo is updated by the minibatch features {g( ) in a first-in and first-out order. By fixing gE and updating fw in Eqn. (1), MoCo pushes the query æ; away from its negative keys in dictionary B while pulling together its positive key T.. For gg. it is updated via exponential moving average, i.e. E (1 (1-L) & taD with a constant & E (0,
((
From Eqn. (1), one can observe that MoCo views each image as an individual class and uses one-hot label y E 12+1 (its nonzero position the position of its positive key) to train fw. However, as mentioned in Sec. 1, the one-hot labels cannot reveal the semantic similarity between a query I; and its positive and negatives and thus impair representation learning. In the following, we theoretically analyze the effect of inaccurate labels to the generalization of MoCo for instance discrimination.
Generalization Analysis. We focus on analyzing MoCo in the final training stage where the sample (key) distribution in the dictionary B is almost fixed. This simplified setup is reasonable because (i) in the final training stage, the target network 9 almost does not change due the very small momentum updating parameter i in practice and the oncoming convergence of the online network fwi (i) dictionary is sufficient large to cover different patterns in the dataset. This fixed sample distribution simplifies the analysis, and also provides valuable insights.
Let D { 24))%= denote the training positive pairs in MoCo sampled from unknown distribution S. Moreover, the query æ; has ground truth soft label y," E B over the key set B U B). where the t-th entry y measures the semantic similarity between x; and the f-th key by in B; In practice, given query æ; and dictionary B, MoCo estimates an one-hot label of æ; as E R whose first entry is one and remaining entries are zero. So y; ignores the semantic similarity between x; and keys B, and differs from y². Then MoCo minimizes an empirical risk:
((fm) - ((h(f (x), v.). n
B.).
(2)
i=1
3

---

where ff (;)
o (i). b.)] denotes the predicted class probability, and ((.-)) is cross-entropy loss. Ideally, one should sample sufficient pairs (æ,, Ti)) from the distribu- tion S and use the ground truth label y. of a, to minimize the population risk:
((fw) = ((),,.))- ((h(ffim))))
v.))
Here we assume the ground truth label yf is soft which is indeed more reasonable and stricter than the one-hot label setting especially for contrastive learning [22]. It is because soft label requires the networks to capture the semantic similarity between query and the instances in B; and bring semantically similar instances together, greatly helping downstream tasks (c.g. classification) where global semantic information is needed, while one-hot label only needs networks to distinguish each instance from others and does not consider the global semantic structures in the data. Actually, the semantic similarity among samples here is also known "dark knowledge" in knowledge distillation [22] or "Bayes class probabbiitb,","" and is often used to replace the one-hot label for training network with remarkable performance improvement in many tasks, e.g. classification [30]. As both the data distribution S and the ground truth labels are unknown, MoCo optimizes the empirical risk @(ww) in (2) instead of the population risk @(fw) in (3). In this way, the optimal network fw by minimizing @(fw) differs from that via optimizing ((fw). ). It is natural to ask whether fw by minimizing @(fw) can well perform instance discrimination task in contrastive learning, i.e. whether fw can capture the semantic similarity (y,") between any test sample (æ,, Z2,~~ ~ and the keys (samples) in B,. To solve this issue, Theorem analyzes the generalization error of Su for instance discrimination. We are interested in the generalization error defined with the true soft labels to measure the semantic similarity learning performance of f via optimizing ((fw)) on the instance discrimination task which can often better reflect the performance
the downstream tasks.
), y) E a2), {(., y) is LipLippchitzz w.r.t. y. Let F be a finite Theorem 1. Suppose [ (x)) BB))
class of hypotheses E(h(fw (x) Bx), ), y)
X x - R and [F]
be its covering number under || lloc
metric.
(I) Let ED y-y--y"nn
y [12] EDS 1/ Σ y
v vlll For any e (0, 1). it holds
@(fw) - -((mx) S STy ED.. s |1| - 12
2( a1 Vn
+ 7(a2- - ((22-a))21n22)) 1n22111())))
n
3)
with probability at least 1-v, where VDD is the variance of ((h(f(x))
on the data D.
(2) There exists a contrastive classification problem. a class of hypotheses f(h(fw (x), Bz), y)
X x y + R. and a constant cu such that the generalization error of fw is lower bounded
[Q(fw) - -((w) 2 co ED-S l1y-- - y 12]
See its proof in Appendix B. Theorem 1 shows that for the task of learning semantic similarity between a query and its positive and negatives which is important for downstream tasks (e.g.. classification), the generalization error of fw trained with the one-hot labels y is upper bounded In(
by O (ED=s [|| y 2 (l(mn))) It means that large training sample number n gives
small generalization error, as intuitively. model sees sufficient samples and can generalize better. The loss variance Vo on the dataset D measures data diversity: the larger data diversity Vo, the more challenging to learn model with good generalization. Here we are particularly interested in the factor ED-ssll y yy'l, [l,] which reveals an important property: the higher accuracy of the training label y to the ground truth label y". the smaller generalization error. Moreover, Theorem I proves that there exists a contrastive classification problem such that the lower bound of generalization error depends
EDDss sllv - y y2|lz]. So the upper bound of generalization error is tight in terms of Ep~silla Illy y ll2] Thus, to better capture the underlying semantic similarity between query æ; and samples in dictionary B, to bring semantically similar samples together and better solve downstream tasks, one should provide accurate label y, to the soft true label y, In the following, we introduce our solution to estimate more accurate and informative soft labels for contrastive learning.
3 Self-Labeling Refinement for Contrastive Learning
Our Self-IAbeliNg rEfinement (SANE) approach for contrastive learning contains (i) Self-Labeling Refinery (SLR for short) and (ii) Momentum Mixup (MM) which complementally refine noisy labels

---

respectively from label estimation and positive pair construction. SLR uses current training model and data to estimate more accurate and informative soft labels, while MM increases similarity between virtual query and its positive, and thus improves label accuracy.
We begin by slightly modifying the instance discrimination task in MoCo. Specifically, for the query 2; in the current minibatch {<e,, Ty Hiz we maximize its similarity to its positive sample æ; in the key set B= = { J(b,} )--1 and minimize its similarity to the remaining samples in B:
s+b Le (e ( v.)1) - E Yiu log
o b%) Ll=1 o b1)
(4)
k=
where by is the k-th sample in B, and y, the one-hot label of query æ; whose i-th entry yı one. In this way, the labels of current queries {= are defined on a shared set B, and can be linearly combined which is key for SLR & MM. Next, we aim to improve the quality of label yi in (4) below.
3.1 Self-Labeling Refinery
Methodology. As analyzed in Sec. 1 and 2, the one-hot labels in Eqn. (4) could not well reveal the semantic similarity between 2; and the instance keys in the set B, and thus impairs good representation learning. To alleviate this issue, we introduce Self-Labeling Refinery (SLR) which employs network and data themselves to generate more accurate and informative labels, and improves the performance of contrastive learning. Specifically, to refine the one-hot label y; of query æ;, SLR uses its positive instance 2; to estimate the underlying semantic similarity between I; instances in B= (æt; } 1 ) since æ; and 2; come from the same image and should have close semantic similarity with instances in B. Let by be the k-th sample in B. Then at the t-th iteration, SLR first estimates the instance-class probability pt ERR+ of m; on the set B whose k-th entry is defined
0 1/7 (T., bk)
P
(r' E ())
+ 1/T (z., b,)
The constant 7' sharpens pl and removes some possible small noise, since smooth labels cannot well distillate their knowledge to a network [31]. Then SLR uses p to approximate the semantic similarity between æ; and the instances in B and employs it as the soft label of x; for contrastive learning.
However, since 2;; highly similar to itself, ph, could be much larger than others and conceals the similarity of other semantically similar instances in B. To alleviate this artificial effect, SLR removes 2; from the set B and re-estimates the similarity between æ; and the remaining instances B:
11/T' (T;, bb)
dh 0.
Li=1, 1/ bi)
Finally, SLR linearly combines the one-hot label y, and two label estimations, i.c. p, and q. to obtain more accurate, robust and informative label of of æ; at the f-th itcration:
y (1 a B.)y, + axp! + B.q
(5)
where ar and 3e are two constants. In our experiments,
set a2 μ max Pia/2 and Br I max; /z. where z = 1 1+ max; Pts + max; L. the constants 1, max; Pie and max; g respectively denote the largest confidences of labels ys, pf and of on a certain class. Here hyperpa- rameter μ controls the prior confidence of p' and q². So SLR only has two parameters T' and μ to tune.
The Benefit Analysis of Label Refinery. Now we analyze the performance of our SLR on label- corrupted data. We first describe the dataset. Let {ca [c,]Kr C Rd be K vanilla samples belonging to K<K semantic classes, and {x, y:)) Rd x R be the random crops of {c; k. Since in practice, one often cares more the semantic class prediction performance of a model which often directly reflects the performance on the downstream tasks, we assume that the labels {y, 1 denote corrupted semantic-class labels. Accordingly, we will analyze whether SLR can refine the corrupted labels (y, } and whether it helps model learn the essential semantic-class knowledge of (æ; ) Finally, while allowing for multiple classes, we assume the labels are scalars and take values in [-1, 1] interval for simplicity. We formally define our label-corruped dataset below.
5

---

Definition 1
e. ((p.e,8)-come cooorupte dataset), Let (2,, Vi) denote the pairs of crops (augmentations) and ground trutth semantic label, where crop æe generated the t-th sample e, obeys |0, - -cll2 S E with a constant E, and y E JR of x, is the label of ct. Moreover, samples and the crops are normalized, i.e. (cal2==|aax = 1(vi, k).. Each c; has n4 crops, where ci n. S,; Cu n. with two constants a₁ and Cu. Besides, different classes are separated with a label separation 6:
1 7k 2 5,
||-- - 2 28, (Vi + k).
A (p,8,) --corrupee dataset {aa; y obeys the above conditions but with corrupted label {ye 1 Spefically, for each sample C2 al most png augmenlations are assigned wrong labels in (nd
This data model is rich enough model realistic data, since different clusters can be assigned to the same label. This definition allows for a fraction p of corruptions in each cluster, and can well characterize the realistic data.
Then we study a network of one hidden layer as an example to investigate the label refining perfor- mance of our SLR. The network parameterized by WER and u E R& is defined as
ZER' f(W. x) = v ((Wx).
(6)
where é is an activation function. Following [32-34] which analyze convergence of networks or robust learning of network, we fix v to be a unit vector where half the entries are 1/Vk: and other half are -1/Kk & to simplify exposition. So we only optimize over W that contains most network parameters and will be shown to be sufficient for label refinery. Then given a (p, - dataset {(z,, y/) 17 1. at the f-iteration we train the network via minimizing the quadratic loss:
(W) - 1 Σ. ( f(W, a. x))))= Il - f((,, x))
Here the label J of sample æ; is estimated by Eqn. (5) in which p f(W,, 2x)) denotes predicted label by using the positive T; of x,, i.e. ||ū - c:ll2 S E æ; is augmented from vanilla sample (i) performing nonlinear mapping on network output greatly increases analysis difficulty; (ii) our refinery (5) is still provably sufficient to refine labels when 8. =0 0 and r' =1, 1. Then we update W via gradient descent algorithm with learning rate ry:
e. We set B 0 and 7' =1 for simplicity,
W W.- (W.))
Following most works on network convergence analysis [32-40], we use gradient descent and quadratic loss, since (i) gradient descent is expectation version of stochastic one and often reveals similar convergence behaviors; (ii) one can expect similar results for other losses, e.g. cross entropy, but quadratic loss gives simpler gradient computation. For analysis, we impose mild assumptions on network (6) and our SLR, which are widely used in network analysis [41-46].
Assumption 1. For network (6), assume & and its first- and second-order derivatives obey (=)) for Vz and some r' 1, the entries of initialization Wo obey i.i.d. (0, 1). Assumption 2. Define network covariance matrix E(C) (CCT) @ Bulo'(Cu)e (Cu)" where [=[c...ck]] u~ (0, I), @ is the elementwise product. Let X(C) > be the minimum eigen- value E(C). For label refinery, assume 3// Eie la a1+1| t f(W. 0, X) y 1|2 and 3/n Eioo (1- ) to t0 lay act1l S &2 (WW, X) 21 where to m)(() (1-ommx)p)) with three constants Vi, 1/2 and c₁. Here Ommax is defined as amax 1
KjK log I /log
Assumption 1 is mild, as most differential activation functions, e.g. softplus and sigmoid, satisfy it, and the Gaussian initialization is used in practice. We assume Gaussian variance to be one for notation simplicity, but our technique is applicable to any constant variance. Assumption 2 requires that the discrepancy between a₁ and a++1 until some iteration number to are bounded, which holds by setting proper ag Here we assume B = 0 and r' "1= for simplicity, as (i) performing nonlinear mapping on network output greatly increases analysis difficulty; (ii) we will show that even though Be = 0 and r' 1, our refinery (5) is still provably sufficient to refine labels. For X(C). many works [34, 41 41-44] empirically and theoretically show X(C) > Based on these assumptions, we state our results in Theorem 2 with constants C1 Co.

---

Theorem 2. Assume {(x; yi)l is (p,,,8)-corrupte dataset with noiseless labels 1y;1i=1 Let
{= colog( ( ([viloRR). Suppose E and the number of hidden nodes satisfy & I C2 min( X(C)
< comax
K k = 30 Let v/ = -1+ 11 + v422- step size nl S 2 with probability omax A(C)
<4 1 1-3/ KK100K0KK exp(-100d), after 12 mnX(C)
log
((1 (Vvllog
iterations, the gradient descent (7)
satisfies:
(1) By defining C=E Aog K /MC)
.. .... the discrepancy between the label y estimated by our SLR (5) and the true label y" of the augmentation data { ) is
and y² .y.....
bounded:
1
v2122
lly y 112 aa..
Vn
where "y' = -'y T'l- Moreover, if p estimated label g predicts true label yi of any crop xt
32' & S cod min
MCO TT rva). 1- 16 S oz the
Tk = with k* =rargming I -Mkl
(2) By using the refined label g' in (5) to train network and letting f(W X) ".). mn),, the error of network prediction on {a;))- is upper bounded
1 11 X) - - 32
Vn
If assumptions on p and E in (1) hold, for vanilla sample e& (vk =1 ..... K), network f(W,,)) predicts the true semantic label "Ik of any augmentation x that satisfies - callaSe:
7k- = 7k with
aargmin ((W,, 7
See its proof roadmap and proof in Appendix C.2. The first result in Theorem 2 shows that after training iterations lo. the discrepancy between the label y estimated by our label refinery (5), i.e. SLR, and ground truth label y of cropped training data { is upper bounded by Olly y²llz + 5)- Both factors |y y/ |la and p in the factor ( reflect the label error of the provided corrupted label y. Another important factor in Ç is the smallest eigenvalue X(C') of network covariance matrix E(C) in Assumption Typically, the performance of a network heavily relies on the data diversity even without label corruption. For instance, if two samples are nearly the same but have different labels, the learning of a network is difficult. X(C') can quantify this data diversity, as one can think of X(C) as condition number associated with the network which measures the diversity of the vanilla samples { Intuitively, if there are two similar vanilla samples, E(C) is trivially rank deficient and has small minimum eigenvalue, meaning more challenges to distinguish the augmentations generated from c, Moreover, when the label corruption ratio p and the augmentation distance e are small, the label y estimated by our SLR can predict the true semantic label y, for any crop sample H;, and thus can supervises network to learn the essential semantic-class knowledges from {æ; )
The second result in Theorem 2 shows that by using the refined label o' in our SLR (5) to train network f(W,
.). ), the error network prediction on augmentations { can be upper bounded by Ç. Similarly, the factor P and X(C) in respectively reflect the initial label error and the data diversity, which both reflect the learning difficulty for model on the augmentation data {(x,, y: ) More importantly,
results also guarantee the test performance of the trained network -WW Specifically, when the label corruption ratio and sample augmentation distance e are small, for any vanilla sample ea (vk=1- the true semantic label 7/ of its any augmentation æ (i.e. || - Cx S E). These results accord with Theorem 1 that shows the more accurate of training labels, the better generalization of the trained network. These results show the effectiveness of the refined labels by our method.
K), the network f f(W.,.) trained by our SLR can exactly prediet
3.2 Momentum Mixup
Now we propose momentum mixup (MM) to further reduce the possible label noise in realistic data and increase the data diversity well. Similar to vanilla mixup [47], we construct virtual instance
x 0x, (1 0) y! 0ÿ. + 0) 0 ~ Beta(k, (s, K) [0, 1],
(8)
7

---

where 2y is randomly sampled from the key set ( y, denotes the refined label by Eqn. (5), Beta(K, k) is beta distribution. Here x; and T, share the same label y; on the set B == [bi as they come from the same instance. We call the mixup (8) 'momentum mixup", since the sample Et is fed into the momentum-updatee network gg. and plays contrastive key for instance discrimination. So MM differs from the vanilla mixup used in [27, 28] where T is replaced with 2y and both are fed into online network fw. and enjoys the following advantages. Firstly, MM can improve the accuracy of the label y/ compared with vanilla mixup. For explanation, assume D, in (8) is one-hot hot label. Then o. has two positive keys I, and T in B decided by its label v/- Accordingly, the component 3y in a: = 0zx; (1-0)at 0)a; directly increases the similarity between the query af and its positive key 3g in B. So the label weight (1 (1-0) of label y on the key 7y. to bring æ² and arg together is relatively accurate, as x really contains the semantic information of atg- Mcanwhile, the sum of label weights in y remaining instance in B 2 is scaled by 0, which also scales the possible label noise on instances in B .G smaller due to 0 << 1. By comparison, for vanilla mixup, the label weight (1-0) 0) of label y on the key T; does not improve label accuracy. It is because the positive pair Iy and 2 are obtained via random augmentation, e.g. crop, and may not be semantically similar, meaning that the component 2& in x could not increase similarity with Ty. So its label weight (1 - -)) to push x close to the key Ty is not as accurate as the one in MM. Another advantage of MM is that it allows to use strong augmentation. As observed in [12], directly using strong augmentation in contrastive learning, e.g. MoCo, leads to performance degradation, since the instance obtained by strong augmentation often heavily differs from the one with weak augmentation. As aforementioned, the component T& in x; = Azz + (1 - 0) increases the similarity between the query xf and the key 3y in B, even though (zx, .x4) obtained via strong augmentation. So MM could reduce the matching difficulty between positive instances.
With all the components in place, we are ready to define our proposed SANE model as follows:
((w) (1 Re (w, (( 9.))) ALe (ur, ((a' v/))),
where Le (w. {(,, y:}}) defined in Eqn. (4) denotes the vanilla contrastive loss with one-hot label Ys. (ww, ((rl )())) denotes the momentum mixup loss with label y estimated by our self-labeling refinery (5), and À is a constant. Experimental results in Sec. 4 show the effectiveness of both loss terms. See algorithm details in Algorithm I of Appendix A.
Limitation Discussion. SANE follows MoCo-alike framework and hopes to obtain a more accurate soft label of a query over its positive and negatives for instance discrimination. So one limitation of SANE is that it does not apply to BYOL-alike methods [6] that only pulls positive pair together and does not require any labels. However, momentum mixup in SANE which increases the similarity of positive pair may also benefit B BOL, which is left as our future work to thoroughly test.
Societal Impact Discussion. As an unsupervised learning method, SANE could benefit many applications of societal interest where only low-resource labeled data are available, e.g. medical data [48], as SANE can use a mass of unlabeled data for pretraining and few labeled data for fine- tuning. But same to standard contrastive learning. SANE may suffer from learning bias caused by the potential data bias, and may provide bias or worse performance on smaller classes or groups.
4 Experiments
Our Pytorch code is available at http:////ppaneeeewww.ee///rrrrtp Reffereer//Ssattoor//ooooooos//
4.1 Evaluation Results on CIFARIO and ImageNet
Table 1: Classification accuracy (%).
Settings. We use ResNet50 [49] with 3-layered MLP head for CIFAR10 [50] and ImageNet [21]. We first pretrain SANE, and then train a linear classifier on top of -04s-dimensional frozen features in ResNet50. With dictionary size 4, 096, we pretrain 2, 000 epochs on CIFARIO instead of 4, 000 epochs of MoCo, BYOL, and i-Mix in [28]. Dictionary size on Im- ageNet is 65, 536. For linear classifier, we train 200/100
CIFAR 10 dataset
KNNN lincar evaluation
MoCo v2 [2]
92.5
93.9
SimCLR [3 4J
94.0
BYOL. [
92.4
93.9
DACL [29]
94.4
CLSA (strong) [12]
93.4
94.9
i-Mix (+MoCoo [2
95.9
SANE
96.1
epochs on CIFAR 10/ImageNet. See all optimizer settings in
SANE (strong)
95.5
96.5
Supervised [28]
95.5
8

---

Table 2: Top-1 accuracy (%) under linear evaluation on ImageNet.
augmentation
method (200 epochs)
Top
method (2800 epochs)
Top
MoCo [1]
60.8
PIRL-800epuchs [51]
63.6
SimCLR 4
61.9
CMC [52]
66.2
CPC v2 [53]
63.8
70.0
PCL [X]
65.9
MoCo v2- 2000phhss
2
71.1
weak
MoCo v2 [2]
67.5
BYOL-1000epon [
74.3
CO2 [20]
68.0
SinSiam-800epon [54]
71.3
MixCo [27]
68.4
i-Miix-00lepochs [28]
71.3
SWAV-Multi [5]
72.7
SWAV-Multi-S00ppen [S]
75.3
SANE-Single
70.6
73.0
SANE-Multi
73.5
SANE-Multi- Sollii-000pochs
75.7
CLSA-Single [12]
69.4
[12]
72.2
CLSA-Multi
73.3
CLSA-Multi-8 -0-pppll--800ppchh [12]
76.2
strong
JANE-Single
70.1
JANE-Single 800epos
73.5
SANE-Multi
73.7
SANE Multi-M00e 800epochsp
76.4
strong JigSaw
InfoMin Aug [55]
70.1
II
InfoMin Aug- tug-800epochs [55]
73.0
others
InstDise [56]
54.0
BigBiGAN [57]
56.6
localAgg [58]
58.8
Sel.a-400lepochs [59]
61.5
Supervised [4]
76.5
II
Supervised [4]
76.5
Appendix A.
We use standard data augmentations in [1] for pretraining and test unless otherwise stated. E.g., for test, perform normalization on CIFARIO, and use center crop and normalization on ImageNet. For SANE, set 0.2, r' = 0.8, = in Beta(k, K. x) CIFARIO, 10, and = 0.2, r'= 1=0.1 ImageNet. with current iteration m2 on CIFARIO, and mi 0.5, m2 10
For confidence μM, we increase as f4 m2 (m2 mı (ttt))((TtT/))) I and total training iteration T. We set mh ImageNet. For KNN on CIFAR10, its neighborhood number is 50 and its temperature is 0.05.
For CIFAR10, to fairly compare with [28], we crop each image into two views to construct the loss (9). For ImageNet, we follow CLSA [12] and train SANE in two settings. SANE-Single Single uses a single crop in momentum mixup loss Le (w. Kaa/ v(}} in (9) that crops each image to smaller size of 96 x 96, without much extra computational cost to process these small images. SANE-Mult crop each image into five sizes 224 x 224, 192 x 192, 160 160, 128 x 128, and 96 x 96 and averages their momentum mixup losses. This ensures a fair comparison with CLSA and SwAV. Moreover, we use strong augmentation strategy in CLSA. Spefically, for the above small image, operation from 14 augmentations used in CLSA, and apply it to the image with a probability of 0.5, which is repeated 5 times. We "(strong)" mark whether small images in momentum mixup loss. Thus, SANE has almost the same training cost with CLSA,, i.e. about 75 (198) hours with 8 GPUs, 200 epochs, batch size of 256 for SANE-Single (-Multi). For vanilla contrastive loss on ImageNet, we always use weak augmentations. See more details of the augmentation, loss construction, and pretraining cost on CIFARIO and ImageNet in Appendix A.
randomly select an
strong augmentations
the
Results. Table shows that with weak or strong augmentations, SANE always surpasses the baseline on CIFAR1O. Moreover, SANE with strong (weak) augmentation improves supervised baseline by 1.0% (0.6%).
Table 2 also shows that for ImageNet under weak augmentation setting, for 200 (800) epochs SANE-Muilt respectively brings 0.8% (0.6%) improvements over SwAV; with 200 (800) epochs, S SANE-Single also beats the runner-up MixCo (i-Mix and SimSiam). Note, BYOL outperforms SANE-Single but was trained 1, 000 epochs. With strong augmentation, SANE-Single and SANE- Multi also respectively outperform CLSA-Single and CLSA-Multi. Moreover, self-supervised accuracy 76.4% is very close to the accuracy 76.5% of supervised baseline, and still improves 0.2% over CLEAN-Multi even for this challenging case. These results show the superiority and robustness of SANE, thanks to its self-labeling refinery and momentum mixup which both improve label quality and thus bring semantically similar samples together.
4.2 Transfer Results on Downstream Tasks
Settings. We evaluate the pretrained SANE model on VOC [61] and COCO [62]. For classification, we train a linear classifier upon ResNet50 100 epochs by SGD. For object detection, we use the same protocol in [1] to fine-tune the pretrained ResNet50 based on detectron2 [63] for faimess. On VOC,

---

Table 3: Transfer learning results.
Table Effects of the components in SANE with
classification VOCO7
object detection VOC07+12 COCO
strong augmentation on CIFARIO.
method
Accuracy
AP50
AP
lahel in (5) in (5) momentum mixup accuracy (%)
NPID++ + [56]
76.6
79.1
93.7
MoCo []
79.8
81.5
04.6
PIRL [S]]
80.7
BNW [60]
SimCLR [4]
CO2 [20]
85.2
86.4
79.2
81.3
82.7
95.
94.9
94.8
95.
MoCo
i-Xix
[2]
87.1
82.5
82.7
42.0
95.9
SWAV-Mult [S]
88.9
82.6
42.1
Table 5: Effects of parameter À in SANE with
CLSA-Multic (ulli((troon))1)
93.6
83.2
42.3
strong augmentation CIFARIO.
SANE-Multi
92.9
82.9
42.2
SANE-Mull (strong)
94.0
83.4
42.4
regularization X
0.25
0.5
0.75
Supervised [12]
87.5
81.3
40.8
accuracy (%) 94.3
95.8
95.9
95.5
94.5
Table 6: Effects of various mixups on ImageNet.
Accuracy (%)
MoCo-mixup
MuCo+momentum mixup
CIFARIO 10 (weak) CIFARIO (strong)
93.7
94.2
93.3
94.8
ImageNet (weak)
68.4
69.0
train detection head with VOCO7+12 trainval data and tested VOCO7 test data. On COCO, train the head on train2017 set and evaluate on the val2017. See optimization settings in Appendix A.
Results. Table shows that SANE consistently outperforms the compared state-of-the-art approaches on both classification and object detection tasks, and enjoys better performance than supervised method pretrained on ImageNet. These results show the superior transferability of SANE behind which the potential reasons have been discussed in Sec. 4.1.
4.3 Ablation Study
We train SANE 1,000 epochs on CIFARIO to investigate the effects of each component in SANE using strong augmentation. Table 4 shows the benefits of each component, i.e. the label estimations p and q in self-labeling refinery, and momentum mixup.
To investigate the robustness of our SANE to the regularization parameter À in (9), we run 2,000 epochs on CIFAR 10, and report the performance in Table 5. From the results, one observe that the stable performance (robustness) of SANE on CIAFR 10 when regularization parameter À in (9) varies in large range, thus testifying the robustness of SANE.
Then we compare our momentum mixup (8) with vanilla mixup in the works 28]. Specifically, we use one-hot label in MoCo and replace T, in (8) with the query æ; to obtain "MoCo+ mixup", and ours with one-hot label can be viewed as "MoCo+momentum mixup". Then we train them 1,000 epochs CIFAR10 10 with weak/strong augmentation, and 200 epochs ImageNet with weak augmentations. The results in Table 6 show that with weak augmentation, momentum mixup respectively makes about 1.1% and 0.6% improvements over vanilla mixup in [27, 28] CIFAR10 and ImageNet. Moreover, momentum mixup using strong augmentation has accuracy 94.8% and improves its weak augmentation version, while vanilla mixup with strong augmentation suffers from performance degradation. is because as discussed in Sec. 3.2, momentum mixup well reduces the possible label noise, especially for strong augmentations, and can enhance the performance more.
Conclusion
In this work, we prove the benefits of accurate labels to the generalization of contrastive learning. Inspired by this theory, we propose SANE to improve label quality in contrastive learning via self-labeling refinery and momentum mixup. The former uses the positive of a query to generate informative soft labels and combines with vanilla one-hot label to improve label quality. The latter randomly combines queries and positives to make virtual queries more similar to their corresponding positives, improving label accuracy. Experimental results testified the advantages of SANE.
10

---

Acknowledgements
The authors sincerely thank the anonymous reviewers for their constructive comments on this work.
1. Funding. Pan Zhou, Caiming Xiong and Steven HOI are supported by Salesforce, mainly for their GPU resource support. Xiao-Tong Yuan is supported in part by the National Key Research and Development Program of China under Grant No. 201 A018AA00000000 and in part by the Natural Science Foundation of China (NSFC) under Grant No.61 876090 and No.61936005.
Competing Interests. Pan Zhou, Caiming Xiong and Steven HOI are staffs in Salesforce. Xiao- Tong Yuan works a professor in Nanjing University of Information Science & Technology, Nanjing, China.
References
[1] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, pages 9729 9738,, 2020.
[2] X. Chen, H. Fan, R. Girshick, and K. He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04220 04297, 2020.
[3] T. Chen, S. Kornblith, K. Swersky, M. Norouzi, and G. Hinton. Big self-supervised models are strong semi sumi-super learners. In Proc. Conf. Neural Information Processing Systems, 2020.
[4] T. Chen, S. Korablith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In Proc. Int'l Conf. Machine Learning, 2020.
[5] M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Proc. Conf. Neural Information Processing Systems, 2020.
[6] J. Grill, Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Pires, Z. Guo, and M. Azar. Bootstrap your own latent: A new approach to self-supervised learning. In Proc. Conf. Neural Information Processing Systems, 2020.
[7] J. Li. P. Zhou, C. Xiong, R. Socher, and S. CH Hoi. Prototypical contrastive learning of unsupervised representations. In Int'l Conf. Learning Reprexentations, 2021.
[8] M. Noroozi and P. Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In Proc. European Conf. Computer Vision, pages 69-84. Springer, 2016.
[9] C. Doersch and A. Zisserman. Multi-task self-supervised visual learning. In IEEE International Conference Computer Vision, pages 2051-2060, 2017.
[10] N. Komodakis and S. Gidaris. Unsupervised representation learning by predicting image rotations. 2018.
[11] R. Zhang, P. Isola, and A. Efros. Colorful image colorization. In Proc. European Conf. Computer Vision, pages 649-666. Springer, 2016.
[12] X. Wang and G. Qi. Contrastive lcarning with stronger augmentations. 2021.
[13] R. Hadsell, S. Chopra, and Y. LeCun. Dimensionality reduction by learning an invariant mapping. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, volume 2, pages 1735-1742. IEEF, 2006.
[14] R. Hjelm, A. Fedorov, S. deep representations by mutual information estimation and maximization. arXiv preprint arXiv: 1808. 06670, 2018.
K. Grewal, Bachman, A. Trischler, and Y. Bengio. Learning
[15] A. Oord, Li, and O. Vinyals. Representation learning with contrastive predictive cnding, arXiv preprint arXiv: 1807.03748, 2018.
[16] P. Bachman, R. Hjelm, and W. Buchwalter. Learning representations by maximizing mutual information across views. In Proc. Conf. Neural Information Processing Systems, pages
2019.
[17] S. Lin, P. Zhou, Z. Hu, S. Wang, R. Zhao, Y. Zheng, L. Lin, E. Xing, and X. Liang. Prototypical graph contrastive learning, arXiv preprint ar arXiv: 1tt::2106.09666 09645, 2021.
[18] S. Arora, H. Khandeparkar, M. Khodak, O. Plevrakis, and N. Saunshi. A theoretical analysis of contrastive unsupervised representation learning. In Proc. Int'l Conf. Machine Learning, pages 6628-5637, 2019.
11

---

[19] C. Chuang, J. Robinson, Y. Lin, A. Torralba, and S. Jegelka. Debiased contrastive learning. In Proc. Conf. Neural Information Processing Systems, volume 33, 2020.
[20] C. Wei, H. Wang. W. Shen, and A. Yuille. Co2: Consistent contrast for unsupervised visual representation learning. arXiv preprint xrXiv:2010.02217, 2020.
[21] J. Deng, W. Dong, R. Socher, L. Li, K. Li, and F. Li. Imagenet: A large-scale hicrarchical image database. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, pages 248-255. IEEE, 2009.
[22] G. Hinton. O. Vinyals, and J. Dean. Distilling the knowledge in neural network. In Proc. Conf. Neural Information Processing Systems, 2015.
[23] H. Zhang, S. Lin, W. Liu, P. Zhou, J. Tang, X. Liang, and E. Xing. Iterative graph ciff-disiilatton. arXiv preprint arXiv:2010. 12609, 2020.
[24] S. Reed, H. Lee. D. Anguelov, C. Szegedy, D. Erhan, and A. Rabinovich. Training deep neural networks on noisy labels with bootstrapping. arXiv preprint arXiv:1412.6528 2014.
[25] H. Bagherinezhad, M. Horton, M. Rastegari, and A. Farhadi. Label refinery: Improving imagenet classification through label progression. arXiv preprint arXiv: 1805.88054.4) 2018.
[26] D. Berthelot, N. Carlini, I. Goodfellow, N. Papernot, Oliver, and Raffel. Mixmatch: A holistic approach to semi-supervised learning. In Proc. Conf. Neural Information Processing Systems, pages 5049-5059, 2019.
[27] S. Kim, G. Lee, S. Bac, and S. Yun. Mixco: Mix-up contrastive learning for visual representation. arXiv preprint arXiv:0010006300, 2020.
[28] K. Lee, Y. Zhu, K. Sohn, C. Li, J. Shin, and H. Lee. i-mix: A strategy for regularizing contrastive representation learning. arXiv preprint arXiv:2010.08887, 2020.
[29] V. Verma, M. Luong. K. Kawaguchi, H. Pham, and Q Le. Towards domain-agnostic contrastive learning. arXiv preprint arXiv:2011 0.4499, 2020.
[30] G. Chen, W. Choi, X. Yu, T. Han, and M. Chandraker. Learning efficient object detection models with knowledge distillation. In Proc. Conf. Neural Information Processing Systems, pages 742-751, 2017.
[31] R. Müller, S. Komblith, and G. Hinton. When does label smoothing help? In Proc. Conf. Neural Information Processing Systems, pages 4694-4703, 2019.
[32] Y. Li and Y. Liang. Learning overparameterized neural networks via stochastic gradient descent on structured data. In Proc. Conf. Neural Information Processing Systems, volume 31, pages 8157-8166, 2018.
[33] S. Oymak and M. Soltanolkotabi. Overparameterized nonlinear Icarning: Gradient descent takes the shortest path? Proc. Int'l Conf. Machine Learning, pages 4951 49660, 2019.
[34] M. Li, M. Soltanolkotabi, and S. Oymak. Gradient descent with early stopping is provably robust to label noise for overparameterina neural networks. Proc. Int'l Conf. Arrificial Intelligence and Statistics, pages
2020.
[35] P. Zhou, J. Feng. C. Ma. C. Xiong, S. Hoi, and W. E. Towards theoretically understanding why sgd generalizes better than adam in deep learning. In Proc. Conf. Neural Information Processing Systems, 2020.
[36] P. Zhou, X. Yuan, H. Xu, S. Yan, and J. Feng. Efficient meta learning via minibatch proximal update. In Proc. Conf. Neural Information Processing Systems, 2019.
[37] P. Zhou, C. Xiong. R. Socher, and S. Hoi. Theory-inspicet path-regularized differential network architecture search. In Proc. Conf. Neural Information Processing Sysiems, 2020.
[38] P. Zhou, Y. Zou, X. Yuan, J. Feng, C. Xiong, and S Hoi. Task similarity aware meta learning: Theory- inspired improvement maml. In Conf. Uncertainty in Artificial Intelligence, 2021.
[39] P. Zhou, H. Yan, X. Yuan, J. Feng, and S. Yan. Towards understanding why lookahead generalizes better than sgd and beyond. In Proc. Conf. Neural Information Processing Systems, 2021.
[40] Y. Bai, M. Chen, P. Zhou, T. Zhao, J. Lee, S. Kakade, H. Wang. and C. Xiong. How important is the train- validatiio split in meta-learning? In Proc. Int'l Conf. Machine Learning, 2021.
12

---

[41] Z. Allen-Zhu, Y. Li, and Z. Song. A convergence theory for deep learning via over- paraaettreeriziionnn In Proc. Int'l Conf. Machine Learning, pages 242-252, 2019.
[42] B. Xic, Y. Liang, and L. Song. Diverse neural network learns true target functions. Proc. Int'l Conf. Artificial Intelligence and Statistics, pages 1216-1224, 2017.
[43] S. Du, J. Lee, H. Li, L. Wang, and X. Zhai. Gradient descent finds global minima of deep neural networks. In Proc. Int'l Conf. Machine Learning, pages (675-1685, 2019.
[44] S. Du, X. Zhai, B. Poczos, and A. Singh. Gradient descent provably optimizes over-parameterized neural networks. arXiv preprini arXiv:1870.02055 2018.
[45] P. Zhou and Feng. Empirical risk landscape analysis for understanding deep neural networks. In Int't Conf. Learning Representations, 2018.
[46] P. Zhou and J. Feng. Understanding generalization and optimization performance of deep cnns. In Proc. Int'l Conf. Machine Learning, 2018.
[47] H. Zhang, M. Cisse, Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint 17X1v 2017.
[48] R. Miotto, F. Wang. S. Wang, X. Jiang, and J. Dudley, Deep learning for healthcare: review, opportunities and challenges. Briefings in bioinformatics, 123-)1133, 2018.
[49] K. Hc, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. Proc. IEEE Conf. Computer Vision and Pattern Recognition, pages 770-778, 2016.
[50] A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. 2009.
[51] Misra and I., Maaten. Self-supervised learning of pretext-invariant representations. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, pages 6707-6717, 2020.
[52] Y. Tian, D. Krishnan, and P. Isola. Contrastive multiview coding, arXiv preprint arXiv:1906.05888 2019.
[53] O. Hénaff, A. Srinivas, J. Fauw. A. Razavi, C. Doersch, S. Eslami, and A. Oord. Data-efficient image recognition with contrastive predictive coding, arXiv preprint 19502799,. 2019.
[54] X. Chen and K. He. Exploring simple siamese representation learning. arXiv preprint arXiv:2011.10566 2020.
[55] Y. Tian, C. Sun, B. Poole, D. Krishnan, C, Schmid, and P. Isola. What makes for good views for contrastive learning. arXiv preprint arXiv:2005. 2020.
[56] Z. Wu. Y. Xiong, S. Yu, and D. Lin. Unsupervised feature learning via non-parametric instance discrimina- tion. Proc. IEEE Conf. Computer Vision and Puttern Recognition, pages 3733-3742, 2018.
[57] J. Donahue and K. Simonyan. Large scale adversarial representation lcarning. In Proc. Conf. Neural Information Processing Systems, pages 10542-10552, 2019.
[58] C. Zhuang, A. Lin, and Yamins. Local aggregation for unsupervised learning of visual embeddings. In IEEE International Conference on Computer Vision, pages 6002-6012, 2019.
[59] Y. Asano, C. Rupprecht, and A. Vedaldi. Sell-labelling via simultaneous clustering and representation learning. arXiv preprint vrXiv:1911.053711 2019.
[60] S. Gidaris, A. Bursuc, N. Komodakis, P. Pérez, and M. Cord. Leariing representations by predicting bags of visual words. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, pages 6928-6938, 2020.
[61] M. Everingham, G. Van, C. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. Int'L 1. J. Computer Vision, 88(2)3033 -38, 2010.
[62] T. Lin, M. Maire, S. Belongie, J. Hays, Perona, D. Ramanan, P. Dollár, and C. Zitnick. Microsoft coco: Common objects in context. In Proc. European Conf. Computer Vision, pages 740 -500, Springer, 2014.
[63] Y. Wu, A. Kirillov, Massa, W. Lo, and R. Girshick. Detectron2, 2019.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] We have briefly discussed the
main limitations of our work at the end of Sec. 3.2.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We have briefly discussed the potential negative societal impact of our work the end of Sec. 3.2.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] Please see the assumptions made in each theorem and corollary.
(b) Did you include complete proofs of all theoretical results? [Yes] We provide the complete proof' of all theories in Appendix.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed reproduce the main exper- imental results (either in the supplemental material or URLY? [Yes] Please see the attached code. We will further clean up models and other code, and release them online.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Please see the detailed experimental settings Sec. and Appendix A.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No] Since contrastive learning methods are computationally expensive, e.g. days on single GPU for 2000 training epochs on CIFAR 10 and 198 hours on 8 GPUs for 200 epochs on Imagenet, it is not affordable for us to train several times. But from Tables and 5 which show the ablation study of SANE, one can observe very consistent results.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We use one single V100 GPU for training CIFARIO, 10, and 32 GPUs for 800 training epochs on ImageNet. We mentioned this in Sec. 4.1 and provide more details in Appendix A.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Our codes are implemented based on MoCo, CLSA and detectron2. We have also mentioned this at line 375 in the manuscript and Appendix A.
(b) Did you mention the license of the assets? [Yes] The code of MoCo and CLSA satisfies "Creative Commons Attribution-None NonCommercial 4.0 International Public License",
which is also mentioned in Appendix A.
(c) Did you include any assets either in the supplemental material In Appendix, we provide more experimental results, more experimental settings, and proofs of all theories.
URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] We use standard public datasets, including CIFAR10. 10, ImageNet, VOC and COCO which allow researchers to use.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A] We use standard public datasets, including C1, ImageNet, VOC and COCO which consist of objectives/views.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
14

---

(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "borQO4QiyX7", "reviewer": "Reviewer_uTpF", "summary": "The paper argues that the one-hot label for instance discriminative contrastive loss will impair the performance in downstream tasks. By assuming ground-truth soft labels, the authors prove that the generalization of the network depends on the discrepancy between ground-truth soft labels and refined labels. Therefore, two approaches, SLR and MM are proposed to address the issue. Empirically, the proposed approach outperforms the baselines by non-trivial margins.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Over all, the paper is well written and easy to follow. The motivation is clear and supported theoretical analysis. The proposed approach seems to be simple yet effective. It is good to see paper include theoretical analysis beyond intuitive motivation, Although I found theorem 2 a bit hard to follow. The experiment results are impressive, achieving state-of-the-art in various benchmarks including transfer learning.\n\nHere are a few questions and suggestions:\n\n1. In Theorem 1, shall we add an assumption that the loss function $\\ell$ is smooth? As for non-smooth function, the norm of Jacobian might be ill-defined or not be bounded by the Lipschitz constant (the proof in Appendix).\n\n2. In equation (5), what is the main reason to keep both $p$ and $q$? As line 190 states, $p$ could be very concentrated, which makes itself similar to the role of one-hot label $y$. It is actually interesting to see including both $p$ and $q$ improves the performance in Table 4. I try to go over the proofs of Theorem 2 in appendix to investigate the role of $p$, would you mind briefly explain the role of $p$ in theoretical analysis?\n\n3. Theorem 1 bounds the difference between risks with different label distributions. Nevertheless, it is a bit hard to tell which label distribution benefits the downstream tasks. For instance, [1] shows that supervised loss is upper bounded by the contrastive loss under specific assumptions. If one can show the risk with ground-truth smooth labels is a tighter bound or a better estimator of the unknown supervised risk, it would make the big picture more clear.\n\n4. Other contrastive frameworks such as SimCLR also leverage negative samples to learn representations. Can SANE be extended to SimCLR? It seems that the only difference is that there could be more dynamics for the ket set $B$ which makes it hard to be theoretically analyzed. But I hypothesize that SANE can also improve the performance of other contrastive learning frameworks that base on info-NCE loss.\n\nOverall, I think this is a good paper with both theoretical and empirical analysis. I did not observe obvious flaw in the paper.\n\n\n\n[1] Arora et al., A Theoretical Analysis of Contrastive Unsupervised Representation Learning, 2019.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed the limitations and potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "ZAkEDdFF-bM", "reviewer": "Reviewer_wb6d", "summary": "This paper discusses why inaccurate one-hot label in contrastive learning cannot reveal well semantic similarities between samples, and how these inaccurate label assignment would impair its generalization for semantic instance discrimination. \n\nTo address this issue, a novel self-labeling refinement method is proposed to obtain more accurate labels for contrastive learning. This is done via two complementary modules that respectively generates accurate soft-labels and enhances semantic similarity between queries and their positives during training.\n\nBoth theoretical analysis and empirical results are provided to support the claims made in this paper.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Originality\nThe theoretical analysis shows that one should provide accurate labels close to the soft true labels to better capture the underlying semantic similarity between queries and different samples. This is an interesting discovery and it motivates the authors to develop a new self-labeling refinement method that includes soft labels and mixup. Although both of these two techniques are not new, the authors modify each of them in the framework of contrastive learning and make them complementary to each other.\n\n### Quality\n* **Theorem-1** shows that the generalization error of MoCo depends on the discrepancy between the assigned labels and the true labels that really reflect the semantical similarity. While this theorem is of vital importance for this paper, the demonstration of it is completely moved to the supplementary materials, which makes the paper not self-content.\n* **Theorem-2** leads to several important findings of the method, however, it is difficult to understand the intuitions and mathematical proofs behind it due to the complicated  equations and abrupt transitions between them.\n* **Robustness Analysis** is sort of missing for the five hyper-parameters of the proposed method. Regarding the choice for each dataset described in L334-337, more ablation studies should be included for a better understanding of how the proposed method is sensitive to these hyper-parameters.\n* **Baseline Performance** is a bit confusing in Table-4 and Table-5. As shown in Eq-9, the SANE model combines the vanilla contrastive loss with weight (1-$\\lambda$) and the the momentum mixup loss with weight $\\lambda$, the first entry of Table-5 should be the same as the first entry of Table-4, why they are different?\n* **Semi-Supervised** setting comparison could be added. Although the authors claim that they focus on self-supervised learning in this paper, a comparison in the semi-supervised setting, e.g., use 1% or 10% labels as done in SwAV, could be interesting and might help to understand better the proposed methods that leverage soft labels.\n\n### Clarity\nThe motivation of this paper is clear, but the theoretical analysis provided in the main paper is not self-content. Key assumptions and theorems come out suddenly without a smooth roadmap, and the mathematical notations are over-complex (see L232-256). I'd recommend to reshape the Section-3 to make it easier to follow and more understandable.\n\nDiscussion of related works seem also a bit rushed, and extend it with more self-supervised learning methods considering label noises could be welcome.\n\n### Significance\n\nWhile the contrastive loss such as infoNCE become a popular choice in many self-supervised learning methods, the authors discuss why the one-hot label assignment commonly used in these methods could not reflect the true semantic similarity between different samples. This is an important problem and resolving it could benefit many contrastive-based methods as well as the semantic-related downstream tasks.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations and societal impact has been well discussed in the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "ExRiHXmj9h", "reviewer": "Reviewer_xAFA", "summary": "This paper proves inaccurate label assignment heavily  impairs the generalization of contrastive learning, and then proposes self-labeling refinery (SLR) to  generate accurate labels and momentum mixup (MM) to enhance similarity between query and its positive. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proves inaccurate label assignment heavily impairs the generalization of contrastive learning, and then proposes self-labeling refinery (SLR) to  generate accurate labels and momentum mixup (MM) to enhance similarity between query and its positive. I didn't check the proof, but the idea and the methodology looks good for me.\nThe major limitations in my opinion:\n1) The issue of inaccurate label assignment exists but the severity of this issue is arguable. The proportion of negative samples with the same semantic label as query is very small, given the large batch size.\n2) Experimentally the proposed method outperforms CLSA (Moco with strong augmentation) marginally (0.2 - 0.4 %) in Table 2.\n3) I don't understand why p and q are sum-to-one vectors. It is possible that there're multiple samples with the same label as query.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "N.A.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers are unanimously positive about the paper. Relevant question, good analysis and well-justified remedy.
Decision: Accept (Spotlight)