Supervision Accelerates Pre-training in Contrastive Semi-Supervised Learning of Visual Representations

We investigate a strategy for improving the efficiency of contrastive learning of visual representations by leveraging a small amount of supervised information during pre-training . We propose a semi-supervised loss , SuNCEt , based on noise-contrastive estimation and neighbourhood component analysis , that aims to distinguish examples of different classes in addition to the self-supervised instancewise pretext tasks . On ImageNet , we find that SuNCEt can be used to match the semi-supervised learning accuracy of previous contrastive approaches while using less than half the amount of pre-training and compute . Our main insight is that leveraging even a small amount of labeled data during pre-training , and not only during fine-tuning , provides an important signal that can significantly accelerate contrastive learning of visual representations . 1 INTRODUCTION . Learning visual representations that are semantically meaningful with limited semantic annotations is a longstanding challenge with the potential to drastically improve the data-efficiency of learning agents . Semi-supervised learning algorithms based on contrastive instance-wise pretext tasks learn representations with limited label information and have shown great promise ( Hadsell et al. , 2006 ; Wu et al. , 2018b ; Bachman et al. , 2019 ; Misra & van der Maaten , 2020 ; Chen et al. , 2020a ) . Unfortunately , despite achieving state-of-the-art performance , these semi-supervised contrastive approaches typically require at least an order of magnitude more compute than standard supervised training with a cross-entropy loss ( albeit without requiring access to the same amount of labeled data ) . Burdensome computational requirements not only make training laborious and particularly timeand energy-consuming ; they also exacerbate other issues , making it more difficult to scale to more complex models and problems , and potentially inducing significant carbon footprints depending on the infrastructure used for training ( Henderson et al. , 2020 ) . In this work , we investigate a strategy for improving the computational efficiency of contrastive learning of visual representations by leveraging a small amount of supervised information during pre-training . We propose a semi-supervised loss , SuNCEt , based on noise-contrastive estimation ( Gutmann & Hyvärinen , 2010 ) and neighbourhood component analysis ( Goldberger et al. , 2005 ) , that aims at distinguishing examples of different classes in addition to the self-supervised instance-wise pretext tasks . We conduct a case-study with respect to the approach of Chen et al . ( 2020a ) on the ImageNet ( Russakovsky et al. , 2015 ) and CIFAR10 ( Krizhevsky & Hinton , 2009 ) benchmarks . We find that using any available labels during pre-training ( either in the form of a cross-entropy loss or SuNCEt ) can be used to reduce the amount of pre-training required . Our most notable results on ImageNet are obtained with SuNCEt , where we can match the semi-supervised learning accuracy of previous contrastive approaches while using less than half the amount of pre-training and compute , and require no hyper-parameter tuning . : : : By : : : : : : : : : combining SuNCEt : : : with : : : the : : : : : : : : : contrastive : : : : : : SwAV : : : : : : method : : of : : : : : : : : : : : : : : : Caron et al . ( 2020 ) : , : : : we : : : also : : : : : : : achieve : : : : : : : : : : : : state-of-the-art : : : : : top-5 : : : : : : : accuracy : : : on : : : : : : : : ImageNet : : : : with : : : : 10 % : : : : : : labels , : : : : while : : : : : : cutting : : : the : : : : : : : : : : pre-training : : : : : : epochs : : in : : : : : half . 2 BACKGROUND . The goal of contrastive learning is to learn representations by comparison . Recently , this class of approaches has fueled rapid progress in unsupervised representation learning of images through selfsupervision ( Chopra et al. , 2005 ; Hadsell et al. , 2006 ; Bachman et al. , 2019 ; Oord et al. , 2018 ; Hénaff et al. , 2019 ; Tian et al. , 2019 ; Misra & van der Maaten , 2020 ; He et al. , 2019 ; Arora et al. , 2019 ; Chen et al. , 2020a ; Caron et al. , 2020 ; Grill et al. , 2020 ; Chen et al. , 2020b ) . In that context , contrastive approaches usually learn by maximizing the agreement between representations of different views of the same image , either directly , via instance discrimination , or indirectly through , cluster prototypes . Instance-wise approaches perform pairwise comparison of input data to push representations of similar inputs close to one another while pushing apart representations of dissimilar inputs , akin to a form of distance-metric learning . Self-supervised contrastive approaches typically rely on a data-augmentation module , an encoder network , and a contrastive loss . The data augmentation module stochastically maps an image xi ∈ R3×H×W to a different view . Denote by x̂i,1 , x̂i,2 two possible views of an image xi , and denote by fθ the parameterized encoder , which maps an input image x̂i,1 to a representation vector zi,1 = fθ ( x̂i,1 ) ∈ Rd . The encoder fθ is usually parameterized as a deep neural network with learnable parameters θ . Given a representation zi,1 , referred to as an anchor embedding , and the representation of an alternative view of the same input zi,2 , referred to as a positive sample , the goal is to optimize the encoder fθ to output representations that enable one to easily discriminate between the positive sample and noise using multinomial logistic regression . This learning by picking out the positive sample from a pool of negatives is in the spirit of noise-contrastive estimation ( Gutmann & Hyvärinen , 2010 ) . The noise samples in this context are often taken to be the representations of other images . For example , suppose we have a set of images ( xi ) i∈ [ n ] and apply the stochastic data-augmentation to construct a new set with two views of each image , ( x̂i,1 , x̂i,2 ) i∈ [ n ] . Denote by Z = ( zi,1 , zi,2 ) i∈ [ n ] the set of representations corresponding to these augmented images . Then the noise samples with respect to the anchor embedding zi,1 ∈ Z are given by Z\ { zi,1 , zi,2 } . In this work , we minimize the normalized temperature-scaled cross entropy loss ( Chen et al. , 2020a ) for instance-wise discrimination ` inst ( zi,1 ) = − log exp ( sim ( zi,1 , zi,2 ) /τ ) ∑ z∈Z\ { zi,1 } exp ( sim ( zi,1 , z ) /τ ) , ( 1 ) where sim ( a , b ) = a T b ‖a‖‖b‖ denotes the cosine similarity and τ > 0 is a temperature parameter . In typical semi-supervised contrastive learning setups , the encoder fθ is learned in a fully unsupervised pre-training phase . The goal of this pre-training is to learn a representation invariant to common data augmentations ( cf . Hadsell et al . ( 2006 ) ; Misra & van der Maaten ( 2020 ) ) such as random crop/flip , resizing , color distortions , and Gaussian blur . After pre-training on unlabeled data , labeled training instances are leveraged to fine-tune fθ , e.g. , using the canonical cross-entropy loss . 3 METHODOLOGY . Our goal is to investigate a strategy for improving the computational efficiency of contrastive learning of visual representations by leveraging the available supervised information during pre-training . Here we explore a contrastive approach for utilizing available labels , but we also include additional numerical evaluations with a cross-entropy loss and a parametric classifier in Section 4 . Contrastive approach . Consider a set S of labeled samples operated upon by the stochastic dataaugmentation module . The associated set of parameterized embeddings are given by ZS ( θ ) = ( fθ ( x̂ ) ) x̂∈S . Let x̂ ∈ S denote an anchor image view with representation z = fθ ( x̂ ) and class label y . By slight overload of notation , denote by Zy ( θ ) the set of embeddings for images in S with class label y ( same class as the anchor z ) . We define the Supervised Noise Contrastive Estimation ( SuNCEt ) loss as ` ( z ) = − log ∑ zj∈Zy ( θ ) exp ( sim ( z , zj ) /τ ) ∑ zk∈ZS ( θ ) \ { z } exp ( sim ( z , zk ) /τ ) , ( 2 ) which is then averaged over all anchors 1|S| ∑ z∈ZS ( θ ) ` ( z ) . In each iteration of training we sample a few unlabeled images to compute the self-supervised instance-discrimination loss equation 1 , and sample a few labeled images to construct the set S and compute the SuNCEt loss equation 2 . We sum these two losses together and backpropagate through the encoder network . By convention , when “ sampling unlabeled images , ” we actually sample images from the entire training set ( labeled and unlabeled ) . This simple procedure bears some similarity to unsupervised data augmentation ( Xie et al. , 2019 ) , where a supervised cross-entropy loss and a parametric consistency loss are calculated at each iteration . Motivation . We motivate the form of the SuNCEt loss by leveraging the relationship between contrastive representation learning and distance-metric learning . Specifically , the SuNCEt loss can be seen as a form of neighborhood component analysis ( Goldberger et al. , 2005 ) with an alternative similarity metric . Consider a classifier that predicts an image ’ s class based on the similarity of the image ’ s embedding z to those of other labeled images zj using a temperature-scaled cosine similarity metric d ( z , zj ) = z T zj/ ( ‖z‖‖zj‖τ ) . Specifically , let the classifier randomly choose one point as its neighbour , with distribution as described below , and adopt the neighbour ’ s class . Given the query embedding z , denote the probability that the classifier selects point zj ∈ ZS ( θ ) \ { z } as its neighbour by p ( zj |z ) = exp ( d ( z , zj ) ) ∑ zk∈ZS ( θ ) \ { z } exp ( d ( z , zk ) ) . Under mutual exclusivity ( since the classifier only chooses one neighbour ) and a uniform prior , the probability that the classifier predicts the class label ŷ equal to some class c , given a query image x with embedding z , is p ( ŷ = c|z ) = ∑ zj∈Zc ( θ ) p ( zj |z ) = ∑ zj∈Zc ( θ ) exp ( d ( z , zj ) ) ∑ zk∈ZS ( θ ) \ { z } exp ( d ( z , zk ) ) , ( 3 ) where Zc ( θ ) ⊂ ZS ( θ ) is the set of embeddings of labeled images from class c. Minimizing the KL divergence between p ( ŷ|z ) and the true class distribution ( one-hot vector on the true class y ) , one arrives at the SuNCEt loss in equation 2 . Assuming independence between labeled samples , the aggregate loss with respect to all labeled samples S decomposes into the simple sum ∑ z∈ZS ( θ ) ` ( z ) . Numerical experiments in Appendix G show that using SuNCEt during pre-training optimizes this aforementioned non-parametric stochastic nearest-neighbours classifier and significantly out-performs inference with the more common K-Nearest Neighbours strategy . Practical considerations . Rather than directly using the outputs of encoder fθ to contrast samples , we feed the representations into a small multi-layer perceptron ( MLP ) , hθproj , to project the representations into a lower dimensional subspace before evaluating the contrastive loss , following Chen et al . ( 2020a ) . That is , instead of using z = fθ ( x̂ ) directly in equation 1 and equation 2 , we use hθproj ( z ) = hθproj ( fθ ( x̂ ) ) . The projection network hθproj is only used for optimizing the contrastive loss , and is discarded at the fine-tuning phase . In general , adding SuNCEt to a pre-training script only takes a few lines of code . See Listing 2 in Appendix A for the pseudo-code used to compute SuNCEt loss on a mini-batch of labeled images .

Summary: This paper designs a new loss, called SuNCTt, to speed up the convergence of semi-supervised training. Specifically, the loss involves the computation of similarity between anchor and other images with the same class, and the similarity between anchor and other labeled images. It is claimed to be considered as the form of neighborhood component analysis. Together with the standard contrastive learning loss, it only uses less than half the amount of pre-training and computes to match the accuracy of the previous approaches.