A Theory-Driven Self-Labeling Refinement Method for Contrastive Representation Learning

1 Introduction . Self-supervised learning ( SSL ) is an effective approach to learn features without manual annotations , with great success witnessed to many downstream tasks , e.g . image classification and object detection [ 1–7 ] . The methodology of SSL is to construct a pretext task that can obtain data labels via well designing the task itself , and then build a network to learn from these tasks . For instance , by constructing jigsaw puzzle [ 8 ] , spatial arrangement identification [ 9 ] , orientation [ 10 ] , or chromatic channels [ 11 ] as a pretext task , SSL learns high-qualified features from the pretext task that can well transfer to downstream tasks . As it gets rid of the manual annotation requirement in supervised deep learning , SSL has been widely attracted increasing researching interests [ 1 , 12 ] . As a leading approach in SSL , contrastive learning [ 1 , 4 , 13–17 ] constructs a novel instance discrimination pretext task to train a network so that the representations of different crops ( augmentations ) of the same instance are close , while representations of different instances are far from each other . Specifically , for an image crop query , it randomly augments the same image to obtain a positive , and view other image crops as negatives . Then it constructs a one-hot label over the positive and negatives to pull the query together with its positive and push the query away its negatives in the feature space . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Motivation . But the one-hot labels in contrastive learning are indeed inaccurate and uninformative because of the following two reasons . Firstly , for a query , it could be semantically similar or even more similar to some negatives than its positives . Indeed , some negatives even belong to the same semantic class as the query [ 18–20 ] . It holds in practice , as ( i ) to achieve good performance , one often uses sufficient negatives that are much more than the semantic class number , e.g . tens of thousands of negatives for ImageNet [ 21 ] in MoCo [ 1 ] , unavoidably leading to the issue on negatives ; ( ii ) even for the same image , especially for an image containing different objects which occurs in ImageNet , random augmentations , e.g . crop , provide crops with ( slightly ) different semantic information , and thus some of the huge negatives could be more similar to query . Secondly , samples from different classes also have some similarity which is not characterized in the one-hot labels . For example , given a query cat , it is often more similar to a dog than a car though both dog and car are negatives . Learning from those similarity among samples is also important and can improve the performance . This point is also supported by knowledge distillation or self-training approaches [ 22 , 23 ] , where a teacher model or a currently-training model is used to predict semantic similarity of a sample on different classes for further supervising model training , achieving better performance . Therefore , the one-hot label can not well reveal the semantic similarity between query and its positives and “ negatives '' , and can not guarantee the semantically similar samples to close each other , leading to performance degradation . Contributions . In this work , we alleviate the above label issue , and derive some new results and alternatives for contrastive learning . Particularly , we theoretically show that inaccurate labels impair the performance of contrastive learning . Then we propose a self-labeling refinement method to obtain more accurate labels for contrastive learning . Our main contributions are highlighted below . Our first contribution is proving that the generalization error of MoCo for instance discrimination linearly depends on the discrepancy between the estimated labels ( e.g . one-hot labels ) in MoCo and the true labels that really reflect semantical similarity between a query and its positives and negatives . Formally , given n training queries D= { xi } ni=1 with estimated labels { yi } ni=1 ( e.g . one-hot labels in MoCo ) and ground truth labels { y∗i } ni=1 on their corresponding positives and negatives , the generalization error of MoCo for instance discrimination is lower bounded by O ( ED [ ‖y−y∗‖2 ] ) where ED [ ‖y−y∗‖2 ] = 1 n ∑n i=1‖yi−y∗i ‖2 , and is upper bounded by O ( √ ln ( |F| ) /n +ED [ ‖y−y∗‖2 ] ) , where |F| is the covering number of the network hypotheses in MoCo . It means that the more accurate of the estimated labels { yi } ni=1 , the better generalization of MoCo for instance discrimination . Inspired by our theory , we propose a Self-lAbeliNg rEfinement ( SANE ) method which iteratively employs the network and data themselves to generate more accurate and informative soft labels for contrastive learning . SANE has two complementary modules : ( i ) Self-Labeling Refinery ( SLR ) to explicitly generate accurate labels , and ( ii ) Momentum Mixup ( MM ) to increase similarity between query and its positive and implicitly improve label accuracy . Given a query , SLR uses its one positive to estimate semantic similarity between the query and its keys ( i.e . its positive and negatives ) by computing their feature similarity , since a query and its positive come from the same image and should have close similarity on the same keys . Then SLR linearly combines the estimated similarity of a query with its vanilla one-hot label in contrastive learning to iteratively generate more accurate and informative soft labels . Our strategy is that at the early training stage , one-hot label has heavy combination weight to provide relatively accurate labels ; along with more training , the estimated similarity becomes more accurate and informative , and its combination weight becomes larger as it explores useful underlying semantic information between the query and its keys that is missing in the one-hot labels . Besides , we prove that when the semantic labels in the instance discrimination task are corrupted , our SLR can exactly recover the true semantic labels of training data , and networks trained with our SLR can exactly predict the true semantic labels of test samples . Moreover , we introduce MM for contrastive learning to further reduce the possible label noise and also increase augmentation diversity . Specifically , we randomly combines queries { xi } ni=1 and their positives { x̃i } ni=1 as x′i=θxi + ( 1−θ ) x̃k and estimate their labels as y′i= θȳi + ( 1−θ ) ȳk , where indexes i and k are randomly selected , ȳi is the label of both xi and x̃i estimated by our label refinery , and θ∈ ( 0 , 1 ) is a random variable . In this way , the component x̃k in the virtual query x′i directly increases the similarity between the query x′i and the positive key x̃k . So the label weight ( 1− θ ) of label y′i on positive key x̃i to bring x ′ i and x̃k together is relatively accurate , as x ′ i really contains the semantic information of x̃k . Meanwhile , the possible noise at the remaining positions of label y′i is scaled by θ and becomes smaller . In this way , MM also improves the label quality . Other Related Work . To estimate similarity between a query and its negatives , Wei et al . [ 20 ] approximated the similarity by computing cosine similarity between a positive and its negatives , and directly replaced the one-hot label for instance discrimination . Wang et al . [ 12 ] used similar similarity estimated on weak augmentations to supervise the learning of strong augmentations . In contrast , we respectively estimate the similarities of the query on all contrastive keys ( its positive and negatives ) and on only negatives , and linearly combines two estimated similarities with vanilla one-hot label to obtain more accurate and informative label with provable performance guarantee . Learning from noisy label , e.g . [ 24–26 ] also uses soft labels generalized by a network to denoise crop labels and supervise representation learning , and often focus on ( semi- ) supervised learning that differs from our self-supervised learning . Two relevant works [ 27 , 28 ] performed vanilla mixup on all query instances to increase data diversity . Differently , our momentum mixup mainly aims to reduce label noise , as it randomly combines one query with one positive ( instead of one query ) of other instances to increase the similarity between the query and its its positive . Verma et al . [ 29 ] showed that mixup is a better domain-agnostic noise than Gaussian noise for positive pair construction . But they did not perform mixup on labels , which is contrast to [ 27 , 28 ] and ours . See more discussion in Sec . 3.2 and empirical comparison in Sec . 4.3 . 2 Inspiration : A Generalization Analysis of MoCo . In this section , we first briefly review the MoCo [ 1 ] method popularly studied for contrastive learning , and then analyze the impact of inaccurate label assignment on its generalization ability . Review of MoCo . The MoCo method contains an online network fw and a target network gξ receptively parameterized byw and ξ . Both fw and gξ consists of a feature encoder and a projection head ( e.g . 3-layered MLP ) . Given a minibatch { ci } si=1 at each iteration , it first randomly augments each vanilla image ci into two views ( xi , x̃i ) and optimizes the following contrastive loss : Ln ( w ) = − 1 s ∑s i=1 log ( σ ( xi , x̃i ) σ ( xi , x̃i ) + ∑b l=1 σ ( xi , bl ) ) , ( 1 ) where σ ( xi , x̃i ) = exp ( − 〈f ( xi ) , g ( x̃i ) 〉τ‖f ( xi ) ‖2·‖g ( x̃i ) ‖2 ) with a temperature τ . The dictionary B= { bi } bi=1 denotes the negative keys of current minibatch queries { xi } si=1 , and is often of huge size to achieve satisfactory performance , e.g . 65,536 in MoCo . In practice , B in MoCo is updated by the minibatch features { g ( x̃i ) } si=1 in a first-in and first-out order . By fixing gξ and updating fw in Eqn . ( 1 ) , MoCo pushes the query xi away from its negative keys in dictionaryB while pulling together its positive key x̃i . For gξ , it is updated via exponential moving average , i.e . ξ= ( 1−ι ) ξ+ιw with a constant ι ∈ ( 0 , 1 ) . From Eqn . ( 1 ) , one can observe that MoCo views each image as an individual class and uses one-hot label y∈Rb+1 ( its nonzero position is at the position of its positive key ) to train fw . However , as mentioned in Sec . 1 , the one-hot labels can not reveal the semantic similarity between a query xi and its positive and negatives and thus impair representation learning . In the following , we theoretically analyze the effect of inaccurate labels to the generalization of MoCo for instance discrimination . Generalization Analysis . We focus on analyzing MoCo in the final training stage where the sample ( key ) distribution in the dictionary B is almost fixed . This simplified setup is reasonable because ( i ) in the final training stage , the target network gξ almost does not change due to the very small momentum updating parameter ι in practice and the oncoming convergence of the online network fw ; ( ii ) dictionary is sufficient large to cover different patterns in the dataset . This fixed sample distribution simplifies the analysis , and also provides valuable insights . Let D = { ( xi , x̃i ) } ni=1 denote the training positive pairs in MoCo sampled from an unknown distribution S. Moreover , the query xi has ground truth soft label y∗i ∈ Rb+1 over the key set Bi= { x̃i ∪B } , where the t-th entry y∗it measures the semantic similarity between xi and the t-th key b′t inBi . In practice , given query xi and dictionaryBi , MoCo estimates an one-hot label of xi as yi ∈ Rb+1 whose first entry is one and remaining entries are zero . So yi ignores the semantic similarity between xi and keys inBi , and differs from y∗i . Then MoCo minimizes an empirical risk : Q̃ ( fw ) = 1 n ∑n i=1 ` ( h ( fw ( xi ) , Bi ) , yi ) , ( 2 ) where h ( fw ( xi ) , Bi ) = [ σ ( xi , x̃i ) , σ ( xi , b1 ) , · · · , σ ( xi , bb ) ] denotes the predicted class probability , and ` ( · , · ) is cross-entropy loss . Ideally , one should sample sufficient pairs ( xi , x̃i ) from the distribution S and use the ground truth label y∗i of xi to minimize the population risk : Q ( fw ) = E ( xi , x̃i ) ∼S [ ` ( h ( fw ( xi ) , Bi ) , y ∗ i ) ] . ( 3 ) Here we assume the ground truth label y∗i is soft which is indeed more reasonable and stricter than the one-hot label setting especially for contrastive learning [ 22 ] . It is because soft label requires the networks to capture the semantic similarity between query and the instances in Bi and bring semantically similar instances together , greatly helping downstream tasks ( e.g . classification ) where global semantic information is needed , while one-hot label only needs networks to distinguish each instance from others and does not consider the global semantic structures in the data . Actually , the semantic similarity among samples here is also known as “ dark knowledge '' in knowledge distillation [ 22 ] or “ Bayes class-probability '' , and is often used to replace the one-hot label for training network with remarkable performance improvement in many tasks , e.g . classification [ 30 ] . As both the data distribution S and the ground truth labels are unknown , MoCo optimizes the empirical risk Q̃ ( fw ) in ( 2 ) instead of the population risk Q ( fw ) in ( 3 ) . In this way , the optimal network fw by minimizing Q̃ ( fw ) differs from that via optimizing Q ( fw ) . It is natural to ask whether fw by minimizing Q̃ ( fw ) can well perform instance discrimination task in contrastive learning , i.e . whether fw can capture the semantic similarity ( y∗i ) between any test sample ( xi , x̃i ) ∼S and the keys ( samples ) in Bi . To solve this issue , Theorem 1 analyzes the generalization error of fw for instance discrimination . We are interested in the generalization error defined with the true soft labels to measure the semantic similarity learning performance of fw via optimizing Q ( fw ) on the instance discrimination task which can often better reflect the performance on the downstream tasks . Theorem 1 . Suppose ` ( h ( fw ( x ) , Bx ) , y ) ∈ [ a1 , a2 ] , ` ( · , y ) is Ly-Lipschitz w.r.t . y . Let F be a finite class of hypotheses ` ( h ( fw ( x ) , Bx ) , y ) : X×Y→R and |F| be its covering number under ‖·‖∞ metric . ( 1 ) Let ED∼S [ ‖y−y∗‖2 ] = ED∼S [ 1 n ∑n i=1‖yi−y∗i ‖2 ] . For any ν∈ ( 0 , 1 ) , it holds∣∣∣Q ( fw ) − Q̃ ( fw ) ∣∣∣ ≤LyED∼S [ ‖y − y∗‖2 ] + √ 2 ( a2 − a1 ) 2VD ln ( 2|F|/ν ) n + 7 ( a2 − a1 ) 2 ln ( 2|F|/ν ) 3 ( n− 1 ) , with probability at least 1−ν , where VD is the variance of ` ( h ( f ( x ) , Bx ) , y∗ ) on the data D. ( 2 ) There exists a contrastive classification problem , a class of hypotheses ` ( h ( fw ( x ) , Bx ) , y ) : X ×Y → R and a constant c0 such that the generalization error of fw is lower bounded∣∣Q ( fw ) − Q̃ ( fw ) ∣∣ ≥ c0 · ED∼S [ ‖y − y∗‖2 ] . See its proof in Appendix B. Theorem 1 shows that for the task of learning semantic similarity between a query and its positive and negatives which is important for downstream tasks ( e.g. , classification ) , the generalization error of fw trained with the one-hot labels y is upper bounded by O ( ED∼S [ ‖y − y∗‖2 ] + √ VD ln ( |F| ) /n ) . It means that large training sample number n gives small generalization error , as intuitively , model sees sufficient samples and can generalize better . The loss variance VD on the dataset D measures data diversity : the larger data diversity VD , the more challenging to learn a model with good generalization . Here we are particularly interested in the factor ED∼S [ ‖y − y∗‖2 ] which reveals an important property : the higher accuracy of the training label y to the ground truth label y∗ , the smaller generalization error . Moreover , Theorem 1 proves that there exists a contrastive classification problem such that the lower bound of generalization error depends on ED∼S [ ‖y − y∗‖2 ] . So the upper bound of generalization error is tight in terms of ED∼S [ ‖y − y∗‖2 ] . Thus , to better capture the underlying semantic similarity between query xi and samples in dictionaryBi to bring semantically similar samples together and better solve downstream tasks , one should provide accurate label yi to the soft true label y∗i . In the following , we introduce our solution to estimate more accurate and informative soft labels for contrastive learning .

Summary: The paper argues that the one-hot label for instance discriminative contrastive loss will impair the performance in downstream tasks. By assuming ground-truth soft labels, the authors prove that the generalization of the network depends on the discrepancy between ground-truth soft labels and refined labels. Therefore, two approaches, SLR and MM are proposed to address the issue. Empirically, the proposed approach outperforms the baselines by non-trivial margins.