Data-Efficient Image Recognition with Contrastive Predictive Coding

1 INTRODUCTION Deep neural networks excel at perceptual tasks when labeled data are abundant , yet their performance degrades substantially when provided with limited supervision ( Fig . 1 , red ) . In contrast , humans and animals can quickly learn about new classes of objects from few examples ( Landau et al. , 1988 ; Markman , 1989 ) . What accounts for this monumental difference in data-efficiency between biological and machine vision ? While highly-structured representations ( e.g . as proposed by Lake et al. , 2015 ) may improve data-efficiency , it remains unclear how to program explicit structures that capture the enormous complexity of real visual scenes like those in ImageNet ( Russakovsky et al. , 2015 ) . An alternative hypothesis has proposed that intelligent systems need not be structured a priori , but can instead learn about the structure of the world in an unsupervised manner ( Barlow , 1989 ; Hinton et al. , 1999 ; LeCun et al. , 2015 ) . Choosing an appropriate training objective is an open problem , but a promising guiding principle has emerged recently : good representations should make the spatio-temporal variability in natural signals more predictable . Indeed , human perceptual representations have been shown to linearize ( or ‘ straighten ’ ) the temporal transformations found in natural videos , a property lacking from current supervised image recognition models ( Hénaff et al. , 2019 ) , and theories of both spatial and temporal predictability have succeeded in describing properties of early visual areas ( Rao & Ballard , 1999 ; Palmer et al. , 2015 ) . In this work , we hypothesize that spatially predictable representations may allow artificial systems to benefit from human-like data-efficiency . Contrastive Predictive Coding ( CPC , van den Oord et al. , 2018 ) is an unsupervised objective which learns such predictable representations . CPC is a general technique that only requires in its definition that observations be ordered along e.g . temporal or spatial dimensions , and as such has been applied to a variety of different modalities including speech , natural language and images . This generality , combined with the strong performance of its representations in downstream linear classification tasks , makes CPC a promising candidate for investigating the efficacy of predictable representations for data-efficient image recognition . Our work makes the following contributions : • We revisit CPC in terms of its architecture and training methodology , and arrive at a new implementation of CPC with dramatically-improved ability to linearly separate image classes ( +17 % Top-1 ImageNet classification accuracy ) . • We then train deep networks on top of the resulting CPC representations using very few labeled images ( e.g . 1 % of the ImageNet dataset ) , and demonstrate test-time classification accuracy far above networks trained on raw pixels ( 73 % Top-5 accuracy , a 28 % absolute improvement ) , outperforming all other unsupervised representation learning methods ( +15 % Top-5 accuracy over the previous state-of-the-art ( Zhai et al. , 2019 ) ) . Surprisingly , this representation also surpasses supervised methods when given the entire ImageNet dataset ( +1 % Top-5 accuracy ) . • We isolate the contributions of different components of the final model to such downstream tasks . Interestingly , we find that linear classification accuracy is not always predictive of low-data classification accuracy , emphasizing the importance of this metric as a stand-alone benchmark for unsupervised learning . • Finally , we assess the generality of CPC representations by transferring them to a new task and dataset : object detection on PASCAL-VOC 2007 . Consistent with the results from the previous section , we find CPC to give state-of-the-art performance in this setting . 2 EXPERIMENTAL SETUP . We first review the CPC architecture and learning objective in section 2.1 , before detailing how we use its resulting representations for image recognition tasks in section 2.2 . 2.1 CONTRASTIVE PREDICTIVE CODING . Contrastive Predictive Coding as formulated in ( van den Oord et al. , 2018 ) learns representations by training neural networks to predict the representations of future observations from those of past ones . When applied to images , the original formulation of CPC operates by predicting the representations of patches below a certain position from those above it ( Fig . 2 , left ) . These predictions are evaluated using a contrastive loss , in which the network must correctly classify the ‘ future ’ representation amongst a set of unrelated ‘ negative ’ representations . This avoids trivial solutions such as representing all patches with a constant vector , as would be the case with a mean squared error loss . In the CPC architecture , each input image is first divided into a set of overlapping patches xi , j , each of which is encoded with a neural network fθ into a single vector zi , j = fθ ( xi , j ) . To make predictions , a masked convolutional network gφ is then applied to the grid of feature vectors . The masks are such that the receptive field of each resulting context vector ci , j only includes feature vectors that lie above it in the image ( i.e . { zu , v } u≤i , v ) . The prediction task then consists o predicting ‘ future ’ feature vectors zi+k , j from current context vectors ci , j , where k > 0 . The predictions are made linearly : given a context vector ci , j , a prediction length k > 0 , and a prediction matrix Wk , the predicted feature vector is ẑi+k , j = Wkci , j . The quality of this prediction is then evaluated using a contrastive loss . Specifically , the goal is to correctly recognize the target zi+k , j among a set of randomly sampled feature vectors { zl } from the dataset . We compute the probability assigned to the target using a softmax , and evaluate this probability using the usual cross-entropy loss . Summing this loss over locations and prediction offsets , we arrive at the CPC objective as defined in ( van den Oord et al. , 2018 ) : LCPC = − ∑ i , j , k log p ( zi+k , j |ẑi+k , j , { zl } ) = − ∑ i , j , k log exp ( ẑTi+k , jzi+k , j ) exp ( ẑTi+k , jzi+k , j ) + ∑ l exp ( ẑ T i+k , jzl ) The negative samples { zl } are taken from other locations in the image and other images in the minibatch . This loss is called InfoNCE ( van den Oord et al. , 2018 ) as it is inspired by Noise-Contrastive Estimation ( Gutmann & Hyvärinen , 2010 ; Mnih & Kavukcuoglu , 2013 ) and has been shown to maximize the mutual information between ci , j and zi+k , j ( van den Oord et al. , 2018 ) . 2.2 EVALUATION PROTOCOL . Having trained an encoder network fθ , a context network gφ , and a set of linear predictors { Wk } using the CPC objective , we use the latents z = fθ ( x ) as a representation of new observations x for downstream tasks , and discard the rest . We then train a model hψ to classify these representations given a dataset of labeled images . More formally , given a dataset of N unlabeled images Du = { xn } , and a ( potentially much smaller ) dataset of M labeled images Dl = { xm , ym } : θ∗ = argmin θ 1 N N∑ n=1 LCPC [ fθ ( xn ) ] , ψ∗ = argmin ψ 1 M M∑ m=1 LSup [ hψ ◦ fθ∗ ( xm ) , ym ] In all cases , the dataset of unlabeled images Du we pre-train on is the full ImageNet ILSVRC 2012 training set ( Russakovsky et al. , 2015 ) . We consider three labeled datasets Dl for evaluation , each with an associated classifier hψ and supervised losse LSup ( see Fig . 2 , right ) . This protocol is sufficiently generic to allow us to later compare the CPC representation to other methods which have their own means of learning a feature extractor fθ . Linear classification is the standard benchmark for evaluating the quality of unsupervised image representations . In this regime , the classification network hψ is restricted to mean pooling followed by a single linear layer , and the parameters of fθ are kept fixed . The labeled dataset Dl is the entire ImageNet dataset , and the supervised loss LSup is standard cross-entropy . We use the same dataaugmentation as in the unsupervised learning phase for training , and none at test time and evaluate with a single crop . Efficient classification directly tests whether the CPC representation enables visual learning from few labels . For this task , the classifier hψ is an arbitrary deep neural network ( we use an 11-block ResNet architecture with 4096-dimensional feature maps and 1024-dimensional bottleneck layers ) . The labeled dataset Dl is a subset of the ImageNet dataset : we investigated using 1 % , 2 % , 5 % , 10 % , 20 % , 50 % and 100 % of the ImageNet dataset . The supervised loss LSup is again cross-entropy . In addition to random color-dropping we use the Inception data-augmentation scheme ( Szegedy et al. , 2014 ) for training , no augmentation at test-time and evaluate with a single crop . Transfer learning tests the generality of the representation by applying it to a new task and dataset . For this we chose image detection on the PASCAL-2007 dataset , a standard benchmark in computer vision ( Everingham et al. , 2007 ) . As such Dl is the entire PASCAL-2007 dataset ( comprised of 5011 labeled images ) ; hψ and LSup are the Faster-RCNN architecture and loss ( Ren et al. , 2015 ) . In addition to color-dropping , we use scale-augmentation ( Doersch et al. , 2015 ) for training . For linear classification , we keep the feature extractor fθ fixed to assess the representation in absolute terms . For efficient classification and transfer learning , we additionally explore fine-tuning the feature extractor for the supervised objective . In this regime , we initialize the feature extractor and classifier with the solutions θ∗ , ψ∗ found in the previous learning phase , and train them both for the supervised objective . To ensure that the feature extractor does not deviate too much from the solution dictated by the CPC objective , we use a smaller learning rate and early-stopping . 3 RELATED WORK . Data-efficient learning has typically been approached by two complementary methods , both of which seek to make use of more plentiful unlabeled data : representation learning and semisupervised learning . The former formulates an objective to learn a feature extractor fθ in an unsupervised manner , whereas the latter directly constrains the classifier hψ using the unlabeled data . Representation learning saw early success using generative modeling ( Kingma et al. , 2014 ) , but likelihood-based models have yet to generalize to more complex stimulus classes . Generative adversarial models have also been harnessed for representation learning ( Donahue et al. , 2016 ) , and large-scale implementations have recently achieved corresponding gains in linear classification accuracy ( Donahue & Simonyan , 2019 ) . In contrast to generative models which require the reconstruction of observations , self-supervised techniques directly formulate tasks involving the learned representation . For example , simply asking a network to recognize the spatial layout of an image led to representations that transferred to popular vision tasks such as classification and detection ( Doersch et al. , 2015 ; Noroozi & Favaro , 2016 ) . Other works showed that prediction of color ( Zhang et al. , 2016 ; Larsson et al. , 2017 ) and image orientation ( Gidaris et al. , 2018 ) , and invariance to data augmentation ( Dosovitskiy et al. , 2014 ) can provide useful self-supervised tasks . Beyond single images , works have leveraged video cues such as object tracking ( Wang & Gupta , 2015 ) , frame ordering ( Misra et al. , 2016 ) , and object boundary cues ( Li et al. , 2016 ; Pathak et al. , 2016 ) . Non-visual information can be equally powerful ; information about camera motion ( Agrawal et al. , 2015 ; Jayaraman & Grauman , 2015 ) , scene geometry ( Zamir et al. , 2016 ) , or sound ( Arandjelovic & Zisserman , 2017 ; 2018 ) can all serve as natural sources of supervision . While many of these tasks require predicting fixed quantities computed from the data , another class of contrastive methods formulate their objectives in the learned representations themselves . CPC is a contrastive representation learning method that maximizes the mutual information between spatially removed latent representations with InfoNCE ( van den Oord et al. , 2018 ) , a loss function based on Noise-Contrastive Estimation ( Gutmann & Hyvärinen , 2010 ; Mnih & Kavukcuoglu , 2013 ) . Two other methods have recently been proposed using the same loss function , but with different associated prediction tasks . Contrastive Multiview Coding ( Tian et al. , 2019 ) maximizes the mutual information between representations of different views of the same observation . Augmented Multiscale Deep InfoMax ( AMDIM , Bachman et al. , 2019 ) is most similar to CPC in that it makes predictions across space , but differs in that it also predicts representations across layers in the model . In addition , AMDIM limits the receptive field of its representation , but does this by constraining the number of spatial convolutions in the network architecture rather than using image patches . A common alternative approach for improving data efficiency is label-propagation ( Zhu & Ghahramani , 2002 ) , where a classifier is trained on a subset of labeled data , then used to label parts of the unlabeled dataset , after which the process is repeated . This label-propagation can either be discrete ( as in pseudo-labeling , Lee , 2013 ) or continuous ( as in entropy minimization , Grandvalet & Bengio , 2005 ) . The predictions of this classifier are often constrained to be smooth with respect to certain deformations , such as data-augmentation ( Xie et al. , 2019 ) or adversarial perturbation ( Miyato et al. , 2018 ) . Representation learning and semi-supervised learning have been shown to be complementary and can be combined to great effect ( Zhai et al. , 2019 ) , which is why we focus solely on representation learning in this paper .

Summary: The authors augment contrastive predictive coding (CPC), a recent representation learning technique organized around making local representations maximally useful for predicting other nearby representations, and evaluates their augmented architecture in several image classification problems. Although the modifications to CPC aren't particularly original, the authors show first that these yield a significant improvement in linear classification accuracy. They then use this improved model to obtain impressive performance in classification within semi-supervised and transfer learning settings, giving strong support for the use of such methods within image processing applications.