Data-Efficient Image Recognition with Contrastive Predictive Coding
Authors: Olivier J Henaff
Aravind Srinivas
Jeffrey De Fauw
Ali Razavi
Carl Doersch
S. M. Ali Eslami
Aaron van den Oord
Venue:  (2020)
Keywords: Deep learning
representation learning
contrastive methods
unsupervised learning
self-supervised learning
vision
data-efficiency

Abstract: Human observers can learn to recognize new categories of objects from a handful of examples, yet doing so with machine perception remains an open challenge. We hypothesize that data-efficient recognition is enabled by representations which make the variability in natural signals more predictable, as suggested by recent perceptual evidence. We therefore revisit and improve Contrastive Predictive Coding, a recently-proposed unsupervised learning framework, and arrive at a representation which enables generalization from small amounts of labeled data. When provided with only 1% of ImageNet labels (i.e. 13 per class), this model retains a strong classification performance, 73% Top-5 accuracy, outperforming supervised networks by 28% (a 65% relative improvement) and state-of-the-art semi-supervised methods by 14%. We also find this representation to serve as a useful substrate for object detection on the PASCAL-VOC 2007 dataset, approaching the performance of representations trained with a fully annotated ImageNet dataset.

Under review as a conference paper at ICLR 2020
DATA- IMAGE RECOGNITION WITH CONTRASTIVE PREDICTIVE CODING
Anonymous authors Paper under double-blind review
ABSTRACT
Human observers can learn to recognize new categories of objects from hand- ful of examples, yet doing with machine perception remains an open challenge. We hypothesize that data-efficient recognition is enabled by representations which make the variability natural signals more predictable, as suggested by recent perceptual evidence. We therefore revisit and improve Contrastive Predictive Cod- ing, recently-proposee sentation which enables generalization from small amounts of labeled data. When provided with only 1% of ImageNet labels (i.e. 13 per class), this model retains strong classification performance, 73% Top-5 accuracy, outperforming super- vised networks by 28% (a 65% relative improvement) and state-of-the-art semi- supervised methods by 14%. We also find this representation serve useful substrate for object detection on the PASCAL-VOC 2007 dataset, approaching the performance representations trained with fully annotated ImageNet dataset.
unsupervised learning framework, and arrive at repre-
INTRODUCTION
Deep neural networks excel at perceptual tasks when labeled data abundant, yet their per- formance degrades substantially when provided
with limited supervision (Fig. [ red). In con-
50% Tewer labels
Top-5 classification accuracy
trast, humans and animals can quickly learn about classes of objects from few exam- ples (Landau et al., 1988) Markman 1989).
0.8
What accounts for this monumental difference in data -ata-efficiency between biological and ma- chine vision? While highly-structured repre- sentations (e.g. as proposed by [ake et al. 2015) may improve data-efficiency, it remains unclear how to program explicit structures that capture the enormous complexity of real visual scenes like those in ImageNet (Russakovsky jtt al. 2013). An alternative hypothesis has proposed that intelligent systems need not be structured a priori, but can instead learn about the structure of the world in unsupervised manner (Barlow 1989 Hinton et al. 19999 Le- Cun al.] 2015). Choosing an appropriate traiiig objective is an open problem. but emerged re- cently: good representations should make the spatio-temporal variability natural signals more predictable. Indeed, human perceptual representations have been shown to linearize (or 'straighten') the temporal transformations found in natural videos, property lacking from cur- rent supervised image recognition models (Hénaff et al.) 2219),, and theories both spatial and tem- poral predictability have succeeded in describing properties of early visual areas (Rao & Ballard
0.7
80% fewer labels
0.6
0.5
ResNet trained CPC
ResNet trained on pixels
0.4
2
20
50
100
Percentage of labeled data
Figure Data-efficient image recognition with Contrastive Predictive Coding. With decreas- ing amounts of labeled data, supervised networks trained on pixels fail generalize (red). When trained on unsupervised representations learned with CPC, these networks retain much higher accuracy in this low-data regime (blue). Equiva- lently, the accuracy of supervised networks can be matched with significantly fewer labels.
promising guiding principle

---

Under review as a conference paper at ICLR 2020
1999 Palmer et al. 2015). In this work, hypothesize that spatially predictable representations may allow artificial systems to benefit from human-like like data-efficiency.
Contrastive Predictive Coding (CPC. van den Oord al. 2018 8) is unsupervised objective which learns such predictable representations. CPC'is a general technique that only requires its definition that observations be ordered along e.g. temporal spatial dimensions, and such has been applied to a variety of different modalities including speech, natural language and images. This generality, combined with the strong performance of its representations downstream linear classification tasks, makes CPC a promising candidate for investigating the efficacy of predictable representations for data-efficient image recognition.
Our work makes the following contributions:
We revisit CPC in terms of its architecture and training methodology. and arrive at a new implementation of CPC with dramatically-improoee ability to linearly separate im- age classes (+17% Top-1 ImageNet classification accuracy).
We then train deep networks on top the resulting CPC representations using very few la- beled images (e.g. 1% of the ImageNet dataset), and demonstrate test-time classification ac- curacy far above networks trained on raw pixels (73% Top-5 accuracy, a 28% absolute im- provement), outperforming all other unsupervised representation learning methods (+15% Top-5 accuracy over the previous state-of-the-art (Zhai et al.] 2019)). Surprisingly, this representation also surpasses supervised methods when given the entire ImageNet dataset (+1% % Top-5 accuracy).
We isolate the contributions of different components of the final model to such downstream tasks. Interestingly, find that linear classification accuracy is not always predictive of low-data classification accuracy, emphasizing the importance of this metric as a stand-alone benchmark for unsupervised learning.
Finally, assess the generality of CPC representations by transferring them to task and dataset: object detection on PASCAL-VOC 2007. Consistent with the results from the previous section, find CPC to give state-of-the-ar performance this setting.
EXPERIMENTAL SETUP
We first review the CPC architecture and learning objective in section 2.1 before detailing how use its resulting representations for image recognition tasks in section 2.2
2.1 CONTRASTIVE PREDICTIVE CODING
Contrastive Predictive Coding as formulated in (van den Oord et al., 2018) learns representations by training neural networks predict the representations of future observations from those of past ones. When applied to images, the original formulation of CPC operates by predicting the represen- tations of patches below certain position from those above it (Fig- left). These predictions evaluated using a contrastive loss, in which the network must correctly classify the 'future' repre- sentation amongst set of unrelated 'negative' representations. This avoids trivial solutions such as representing all patches with a constant vector, as would be the case with mean squared error loss.
In the CPC architecture, each input image is first divided into each of which is encoded with a neural network fe into a single vector Zi,j = fo(z.,)) To make predictions, masked convolutional network Is is then applied to the grid of feature vectors. The masks are such that the receptive field of each resulting context vector G.jj only includes feature vectors that lie above it the image (i.e. (Zu.v)uSi.w) The prediction task then consists o predicting 'future" feature vectors Zitkii from current context vectors Ci,, where k > O. The predictions are made linearly: given a context vector Cj a prediction length > 0, and a prediction matrix WE. the predicted feature vector is 2itk.j W.C.5-
of overlapping patches æi
The quality of this prediction then evaluated using a contrastive loss. Specifically, the goal is to correctly recognize the target 2itkij among set of randomly sampled feature vectors {2] from the dataset. We compute the probability assigned to the target using a softmax, and evaluate this probability using the usual cross-entropy loss. Summing this loss locations and prediction
2

---

Under review as a conference paper at ICLR 2020
Self Self-supervised pre-training images Mobel
pre-training
Fackue Fansuter by Sedtae-111
Irage
Pho Inutad Feed
Corkel Nehvork to Maskac Condist
Lineer classification 9038 imagee and brbwls
Eri
Pre-tained
Evaluation
Efficient classification 196 to 1003 maggee and bovil
herve
Pro tuatta
Trensfer learning 903% innaao lituss
Modal yark aree
hale PCAM
Mat
Baseline
Supervised fraining 196 to 100% magee and bovil
(1100) 1)
Coocs
Overview of the framework for semi-supervised learning with Contrastive Predictive Coding. Left: unsupervised pre-training with the spatial prediction task (See Section 2.1). First, an image is divided into grid of overlapping patches. Each patch is encoded independently from the rest with a feature extractor (blue) which terminates with a mean-pooling operation, yielding a single feature vector for that patch. Doing for all patches yields a field of such feature vectors (wireframe vectors). Feature vectors above a certain level (in this case, the center of the image) then aggregated with a context network (red), yielding to linearly predict features vectors below. Right: using the CPC representation for a classification task. Having trained the encoder network, the context network (red) discarded and replaced by classifier network (green) which can be trained in supervised manner. For some experiments, we also tine the encoder network (blue) for the classification task. When applying the encoder to cropped patches (as opposed to the full image) we refer to it as a patched ResNet in the figure.
Figure 2:
of context vectors which are used
offsets, we arrive the CPC objective as defined in (van den Oord et 2018):
exp( 27 2.tki))
Lcrc =
Σ log p(2++t ( {21}) = JjKk
log
exp( itp(2 (. ((tko) + E, exp( ( {.,
i.j.h
The negative samples ) are taken from other locations in the image and other images in the mini- batch. This loss called InfoNCE (van den Oord et al. 2018) as is inspired by Noise-Contrastive Estimation (Gutmann & Hyvärinen 2010 Mnih & Kavukcuoglu (2013) and has been shown to maximize the mutual information between Ci and zi4k.5 (van den Oord et al. 2018).
2.2 EVALUATION PROTOCOL.
Having trained encoder network fe. a context network Is. and a set of linear predictors {W;} using the CPC objective, we use the latents x = f=()) as a representation of new observations x for downstream tasks, and discard the rest. We then train model hy classify these representations given a dataset of labeled images. More formally, given a dataset of N unlabeled images D,, } and a (potentially much smaller) dataset of M labeled images ID; (I )m):
0 arg min
1 N
Σ (c))(f 4/" arg min
Php fo- (Fm), ymm] M Σ
1
In all cases, the dataset of unlabeled images Da we pre-train on is the full ImageNet ILSVRC 2012 training set (Russakovsky et al.] 2015). We consider three labeled datasets D2 for evaluation, each with an associated classifier he and supervised losse Lsup (see Fig. 2 right). This protocol sufficiently generic to allow us to later compare the CPC representation to other methods which have their own means of learning a feature extractor fe.
Linear classification is the standard benchmark for evaluating the quality of unsupervised image representations. In this regime, the classification network h, is restricted to mean pooling followed by a single linear layer, and the parameters of fe are kept fixed. The labeled dataset Dy is the entire ImageNet dataset, and the supervised loss Lsup is standard tross-entropy. We use the same data- augmentation as in the unsupervised learning phase for training, and none at test time and evaluate with single crop.
3

---

Under review as a conference paper at ICLR 2020
Efficient classification directly tests whether the CPC representation enables visual learning from few labels. For this task, the classifier he is an arbitrary deep neural network (we use an 11-block ResNet architecture with 4096-dimensiona feature maps and 1024-dimensional bottleneck layers). The labeled dataset IB, is subset of the ImageNet dataset: we investigated using I 1%, 2%, 5%, 10%, 20%, 50% and 100% of the ImageNet dataset. The supervised loss Lup is again cross-entropy. In addition to random color-dropping we use the Inception lata-augmentation scheme (Szegedy et al., 2014) for training, no augmentation test-time and evaluate with single crop.
Transfer learning tests the generality of the representation by applying it to a new task and dataset. For this we chose image detection on the PASCAL-2007 dataset, a standard benchmark in computer vision (Everingham et al. 2007). As such D2 is the entire PASCAL-2007 dataset (comprised of 5011 1 labeled images); his and Suup are the Faster-RCNN architecture and loss (Ren et al. 2015). In addition to color-dropping, we use scale-augmentation (Doersch et al. 2015) for training.
For linear classification, we keep the feature extractor fo fixed to assess the representation in ab- solute terms. For efficient classification and transfer learning, we additionally explore fine-tuning the feature extractor for the supervised objective. In this regime, we initialize the feature extractor and classifier with the solutions 0*, x found in the previous learning phase, and train them both for the supervised objective. To ensure that the feature extractor does not deviate too much from the solution dictated by the CPC objective, use a smaller learning rate and carly-stopping.
3 RELATED WORK
Data-efficient learning has typically been approached by two complementary methods, both of which seek to make use of more plentiful unlabeled data: representation learning and semi- supervised learning. The former formulates an objective learn a feature extractor fo in un- supervised manner, whereas the latter directly constrains the classifier hys using the unlabeled data.
Representation learning saw early success using generative modeling (Kingma et al. 2014), but likelihood-based models have yet to generalize to more complex stimulus classes. Generative ad- versarial models have also been harnessed for representation learning (Donahue et al. 2016), and large-scale implementations have recently achieved corresponding gains in linear classification ac- curacy (Donahue & Simonyan 2019).
In contrast to generative models which require the reconstruction of observations. self-supervised techniques directly formulate tasks involving the learned representation. For example, simply ask- ing network to recognize the spatial layout of an image led to representations that transferred to popular vision tasks such as classification and detection (Doersch et al. 2015; Noroozi & Favaro) 2016). Other works showed that prediction of color (Zhang et al. 2016 Larsson et al. 2017)) and image orientation (Gidaris et al. 2018), and invariance to data augmentation (Dosovitskiy et al. 2014) can provide useful self-supervised tasks. Beyond single images, works have leveraged video cues such as object tracking (Wang & Guptal 20015), frame ordering (Misra et al., 2016), and object boundary cues (Li et al.) 2016, Pathak et al. 2016). Non-visual information can be equally power- ful; information about camera motion (Agrawal et al. [2015] Jayaraman & Grauman 2015). scene geometry (Zamir et al., 2016), or sound (Arandjelovic & Zisserman 2017 2018) can all serve as natural sources of supervision.
While many of these tasks require predicting fixed quantities computed from the data, another class of contrastive methods formulate their objectives in the learned representations themselves. CPC is a contrastive representation learning method that maximizes the mutual information between spa- tially removed latent representations with InfoNCE (van den Oord et al. 2018)), a loss function based on Ciss--Contrattive Estimation (Gutmann & Hyvärinen| [010] Mnih Kavukcuoglu 2013). Two other methods have recently been proposed using the same loss function, but with different associ- ated prediction tasks. Contrastive Multiview Coding (Tian et al. 2019 maximizes the mutual infor- mation between representations of different views of the same observation. Augmented Multiscale Deep InfoMax (AMDIM, Bachman et al., 2019) is most similar to CPC in that it makes predic- tions across space, but differs in that it also predicts representations across layers in the model. In addition, AMDIM limits the receptive field of its representation, but does this by constraining the number of spatial convolutions in the network architecture rather than using image patches.
4

---

either be discrete Under review as a conference paper at ICLR 2020
A common alternative approach for improving data efficiency is labee-propagation (Zhu & Ghahra- mani 2002), where a classifier is trained on a subset of labeled data, then used to label parts of the unlabeled dataset, after which the process is repeated. This label-propagation
(as in pseudo-labeling, Lee, 2013) or continuous (as in entropy minimization, Grandvalet & Bengio, 2005). The predictions of this classifier are often constrained to be smooth with respect to cer- lain deformations, such as data-augmentation (Xie et al. (011)) or adversarial perturbation (Miyato) al. 2018). Representation learning and semi-supervised learning have been shown to comple- mentary and can be combined to great effect (Zhai et al. 2019), which is why we focus solely on representation learning in this paper.
4 RESULTS
When asking whether enables data-efficient learning, wish to use the best possible represen- tative of this model class. Unfortunately, purely unsupervised metrics tell us little about downstream performance, and implementation details have been shown to matter enormously (Doersch & Zisser- man 2017 Kolesnikov et al., 2019). Since many design choices (e.g. network architecture and data- preprocessing) have been previously evaluated using linear classification, we use this benchmark in section 4.1 to align the CPC model with best practices in representation learning and compare to published results. section 4.2 we select the best performing model from the previous section and assess whether it enables efficient classification. We also investigate to what extent the first, more common metric (linear classification accuracy) is predictive efficient classification. Finally, in section 4.3 we investigate the generality of our results through transfer learning to PASCAL-2007.
4.1
FROM CPC VI TO CPC V2
The overarching principle behind our new model design is to increase the scale and efficiency of the encoder architecture while also maximizing the supervisory signal we obtain from each image. At the same time, important not to allow the network to solve the problem trivially, i.e., without learning semantics. To this end, seek to remove low-level cues common across patches by augmenting individual patches independently, using standard stochastic drocesssn techniques from supervised and self-supervised learning.
We identify four axes for model capacity and task setup that could impact the model's 's perfor- mance. The first axis increases model capacity by increasing depth and width, while the second improves training efficiency capacity by intro- ducing layer normalization. The third axis in- creases task complexity by making predictions in all four directions, and the fourth does so by performing more extensive patch-based aug- mentation.
0.67
ResNet-92
Linear classiiicttion accuracy
-0655
ResNet-161
0.63
0.61
0.59
0.57
0.55
Model capacity. Recent work has shown that networks and more effective training improves self-supervised learning (Doersch & Zisser- man 2017 Kolesnikov et al.) 2019), but the original CPC model used only the first 3 stacks of ResNet-101 (He al., (2166a) architecture (i.c. a ResNNet-92. Therefore, we converted the third residual stack of R0sNet-101 (originally containing 23 blocks, 0124sdimessiona feature maps, and 256-dimensional bottleneck layers), to use 46 blocks, with 4096-dimensional fea- ture maps and 512-dimensional bottleneck lay- ers. We call the resulting network ResNet- 161. Consistent with prior results, this new architec- ture delivers better performance regardless of
0.53
CPC
CPC v2
+BU
+HF
+LN
+RC
+TL
+HP
Figure 3: Linear classification performance of new variants of CPC, which incrementally add series of modifications. BU: bottum up spatial predictions. HF: randomly flipping patches hor- izontally. LN: layer normalization. RC: random color-dropping. TL.: tuned prediction lengths. HP: horizontal spatial predictions. We use color to in- dicate the number of spatial predictions used (or- ange, green, blue for 1, 2 and 4 directions).
5

---

Under review as a conference paper at ICLR 2020
Table 1: Linear classifier hy trained with 100% of labels. Comparison linear separability of other self-supervised methods. In all cases feature extractor is optimized in an unsupervised manner, and a linear classifier is trained on top using all labels in the ImageNet dataset.
Method
Top-1
Top-5
Motion Segmentation (MS) Pathak et al.
2016)
27.6
48.3
Exemplar (Ex) (Dosovitskiy al. 2014
31.5
53.1
Relative Position (RP) (Doersch et al.
2015
36.2
59.2
Colorization (Col) (Zhang et al. 2016
39.6
62.5
Combination of MS + Ex + RP
Col
Doersch & Zisserman
2017
69.3
CPC v1 (van den Oord
2018
48.7
73.6
Rotation (ooeemikov et al. 2019)
55.4
CMC (Tian et al.; 2019)
60.1
82.8
Local Aggregation (Zhuang et al.
2019)
60.2
BigBiGAN (Donahue & Simonyan
2019
61.3
81.9
AMDIM (Bachman et al. 2019)
68.1
CPC (ours)
65.9
86.6
other design choices. Interestingly, a larger architecture delivers larger improvements with more efficient training, more self-supervised losses, and more patch- patch-based augmentations (Fig. 3 +5% Top-1 accuracy with original training scheme, +10% % accuracy with new one).
Layer normalization. Large architectures are more difficult to train efficiently. Early works on context prediction with patches used batch normalization (loffe & Szegedy, 2003; Doersch et al.) 2013) to speed training. However, with CPC we find that batch normalization actually harms down- stream performance of large models. We hypothesize that batch normalization allows large models to find a trivial solution CPC: it introduces dependency between patches (through the batch statistics) that can be exploited to bypass the constraints on the receptive field. We find that we can reclaim much of batch normalization's 's training efficiency using layer normalization (Ba et al., 2016), which leads to small gain for the smaller architecture (+1 % accuracy over equivalent architectures that use neither normalization) and a larger gain for the larger architecture (+2.5% % accuracy).
Prediction lengths and directions. Larger architectures also run greater risk of overfitting. We address this by asking more from the network: specifically, whereas van den Oord et al., (2018) pre- dicted each patch using only context from spatially beneath it, repeatedly predict the patch using context from above, to the right, and to the left, resulting in up to four times as many prediction tasks. Combining top-to-bottom with bottom-to-top helps both model architectures (+2% accuracy for both), but using all spatial directions only benefits the larger model (an additional +1.5% for the larger model, -1 % for the smaller), consistent with the idea that model capacity and amount of supervision must go hand-in-hand. We also hypothesized that prediction "length"-i.e offset be- tween the predicted patch and the aggregated context-might affect performance, distant patches might lic on distinct objects, encouraging the network to memorize images. Indeed, limiting the range of the prediction length & {2, 3) performed better than (...... 5) was used originally (+1% % for the larger model).
Patch-based augmentation. If the network can solve CPC using low-level patterns (e-g. straight lines continuing between patches, chromatic aberration), it need not learn semantically meaning- ful content. Augmenting the low-level variability across patches can remove such low level cues. The original CPC model spatially jitters individual patches independently. We further this logic by adopting the 'color dropping' method of Doersch et al. (2015). which randomly drops two of the three color channels in each patch, and find it to delivers systematic gains (+1 % for the small model, +3% % for the larger one). We also randomly flip patches horizontally, but find it only benefits the smaller model (+1 1.).

---

Under review as a conference paper at ICLR 2020
Combined. Cumulatively, these fairly straightforward implementation changes lead to substan- tial improvement to the original CPC model (65.9% Top-1 accuracy, a 17% improvement), making it competitive with recent approaches and outperforming prior methods (see table I Interestingly, we train the same patch-based architecture from scratch in fully supervised manner, we ob- tain 66.4% Top-1 accuracy (with batch normalization; 62.5% without), suggesting that CPC is now nearly saturating the architecture's s representational power despite not using labels. These results illustrate how architecture and data have an outsized impact on the linear classification performance of self- supervised representations, and are interesting to compare with with previous results. For ex- ample, in AMDIM, different settings of data augmentation alone can result in a nearly 10% absolute increase in performance on ImageNet linear classification.
4.2 EFFICIENT IMAGE CLASSIHICATION
We now turn to our original question of whether CPC can enable data-efficient image recognition. We start by evaluating the performance of purely-supervised networks as the size of the labeled dataset Ty varies from 1% to 100% of ImageNet, training separate classifiers on each subset. We found that ResNet-152 to works best across all data-regimes (see Appendix). Despite our efforts to tune the supervised model for low-data classification (including network depth, regularization, and optimization parameters), the accuracy of the best model only reaches 44.1% Top-5 accuracy when trained on 1% the dataset (compared to 93.9% when trained on the entire dataset, see Fig. red).
Contrastive Predictive Coding. We now ad- dress our central question of whether CPC en- ables data-efficient learning. We follow the same paradigm as for the supervised base- line (training and evaluating a separate clas- sifier for each size subset), stacking a neu- ral network classifier on top of the CPC la- tents z = fo( rather than the raw im- age pixels (see section 2.2 efficient clas- sification, and Appendix). This representa- tion, which we selected for its improved linear classification performance (CPC v2 in Fig. }
0.63
Efficient classification (1% labeled)
0.62
0.61
0.6
leads to a sigmificant increase in data-efficiency compared to purely supervised networks (Fig. blue curve). This classifier yields 72.9% Top-5 accuracy with only 1% of the labels, a 29% absolute improvement (65% relative) over purely-supervised methods. Surprisingly, when given the entire dataset, this classifier reaches 80.6%/95.25 5.2% Top1/Top5 accuracy, passing our supervised baseline (ResNet-152 78.0% accuracy) and published results (ResNet-2200 93.9%/955.% He et al. (2016)). We find similar results in all other data-regimes we considered (see Fig.
spatial prediction spatial predictions spatial predictions
0.59
0.58
0.6
0.62
0.64
0.66
Linear classification (100% labeled)
Figure 4: Relationship between linear classifica- tion accuracy and low-data classification. for dif- ferent variants of the CPC model. Left: CPC vari- ants with the same architecture but different train- ing protocols. Orange, green, and blue dots cor- respond to CPC models making predictions in 1, 2, and spatial directions respectively. Within a color group. different models correspond to other implementation details (e.g. layer norm and patch augmentation, and combinations thereof).
How important the model specifications de- scribed in Section [4.]] for low-data classifica- tion? We hypothesized that predictable repre- sentations might enable data-efficient classifi- cation, and therefore expect that increasing the amount of 'predictability' in the representation should also increase its ability to learn from small amounts of data. Fig. 4 shows evidence for this by ablating model parameters and comparing linear classification performance against low-data classification. Consistent with our hypothesis, increasing the number of spatial directions in the CPC prediction task (which increased linear classification performance) systematically increases low-data classification performance (Fig. left, different color groups). As a control, asked if all mod- ifications that improve linear classification also improve low-data classification. We did not find evidence in favor of this: improvements in linear classification result of changing other model
7

---

Under review as a conference paper at ICLR 2020
Table 2: ResNet classifier by trained with 1% or 10% of labels. Comparison other methods for semi-supervised learning using 1% or 10% of labeled data. Representation learning methods use a classifier discriminate an unsupervised representation, and optimize it solely for the super- vised objective on labeled data. Label-propagation methods on the other hand further constrain the classifier with smoothness and entropy criteria on unlabeled data, making the additional assumption that all training images fit into single (unknown) testing category. f denotes methods implemented this work, fixed and fine-tuned denote whether the feature extractor is allowed to accommodate the supervised objective.
Labeled data Method
1%
10%
100%
Top-5 accuracy
Supervised baseline
44.1
82.1
93.9
Methods using label-propagation:
Pseudolabeling (Zhai et al. 2019)
51.6
82.4
VAT Entropy Minimization (Zhai et al. 2019
47.0
83.4
Unsup. Data Augmentation (Xre et al. 2019)
88.5
Rotation + VAT + Ent. Min. (Zhar et 2019
91.2
95.0
Methods using representation learning only:
Instance Discrimination (Wu et al. 2018)
39.2
77.4
Rotation (Zhai et al. 2019)
57.5
86.4
ReeNNet trained on BigBiGAN (fixed)
55.2
78.8
87.0
TReNNet trained on AMDIM (fixed)
67.4
85.8
92.2
RReNNet trained on CPC v2 (fixed)
72.3
89.1
94.4
RResNet trained on CPC v2 (fine-tuned)
72.9
89.5
95.2
parameters (patch-based data-augmentation, layer normalization, and combinations thereof) seem uncorrelated to performance in other tasks (Fig. 4 left, within green group: R = 0.17, 0.36). Different architectural specifications also produced different changes in both tasks (Fig. right). Whereas increasing the depth of the encoding network greatly improves both metrics, increasing the network width (and therefore the number of features used for linear classification) only improves linear classification accuracy.
Other unsupervised representations. How well does the CPC representation compare to other representations that have been learned in an unsupervised manner? If predictable representations are uniquely suited for efficient classification, we would expect other methods within this fam- ily to perform similarly, and other model classes less so. Table [ compares our best model with other works on efficient recognition. We consider three objectives from different model classes: self-supervised learning with rotation prediction (Zhai et al. 2019)) large-scale adversarial feature learning (BigBiGAN, Donahue & Simonyan, 2019). and another contrastive prediction objective (AMDIM, Bachman et al.) 2019). ZZhai et al. (2019) evaluate the low-data classification perfor- mance of representations learned with rotation prediction using similar paradigm and architecture (ResNett1152), hence we report their results directly. Given 1 % of ImageNet, their method achieves 57.5% Top-5 accuracy, consistently with the reduced accuracy of linear classifier (55.4% vs 65.9% for [
Because BigBiGAN and AMDIM achieve stronger linear classification accuracy than rotation pre- diction (61.3% and 68. 1% Top-1 accuracy, respectively), we might expect hetter performance on efficient classification as well. Since their authors do not report results on efficient classification we evaluated these representations using the same paradigm we used for evaluating CPC, stacking ResNet classifier on top of the 7x7x8199 latents of the BigBiGAN and the 7x7x2560 grid of fea- ture vectors of AMDIM. We found fine-tuned representations to yield only marginal gains over fixed
Ithough note that linear classification accuracy is reported using separate model that was designed for linear classification performance (Kolesnikov et al. 2019
8

---

Under review as a conference paper at ICLR 2020
ones (72.9% compared to 72.3% Top-5 accuracy given 1% of labels), hence for simplicity we evalu- ate BigBiGAN and AMDIM on this task while keeping them fixed. We re-tune the hyper-parameter of the classifier (including optimization, regularization, etc.) for each of these representations sep- arately. Although these methods achieve similar performance in terms of linear classification, we find them to achieve very different results in efficient classification. Given 1% of ImageNet, clas- sifiers trained on top of BigBiGAN achieve 55.2% Top-5 accuracy, similarly to rotation prediction (57.5%), despite its increased linear classification accuracy (+6% relative to rotation prediction). In contrast, AMDIM (which also belongs to the family of contrastive prediction methods) achieves 67.4% on this same task. Again, its increased linear classification accuracy did not entail an increase in data-efficiency. Nevertheless, in line with our initial hypothesis, we find that contrastive predic- tion methods such as CPC surpass other approaches in our efficient classífication experiments, and that linear classification performance is not perfectly correlated with these results.
Other semi-supervised techniques A separate class of methods for low-data classification at- tempts propagate the knowledge extracted from the subset of labeled examples to unlabeled examples while being invariant to augmentation or other perturbations. These methods generally depend on the quality of the classifier's predictions, and as such tend to fare well when given in- termediate amounts of data. Although not sufficient in themselves (Unsupervised Data Augmenta- tion (Xie et al. 2019), Virtual Adversarial Training (Miyato et al., 2018) and entropy minimization (Grnnvvalet & Bengio 2005), and pseudo-labeling (Lee 2013) achieve 85.8%, 83.4%, and 82.4% Top-5 accuracy with 10% of labels, compared our 89.4%) when combined with representation learning (e.g. rotation prediction /Zhai et al. 2019) they can provide considerable gains (91.2% Top- 5 accuracy). It is therefore surprising that CPC representations alone can enable accuracy that is comparable to that of these methods, and investigating to what extent they can be combined would be an interesting topic of future work.
4.3 TRANSFER LEARNING: IMAGE DETECTION ON PASCAL VOC 2007
We next investigate transfer performance on object detection on the PASCAL-22007 dataset, which reflects the practical scenario where representation must be trained on dataset with different statistics than the dataset of interest. This dataset also tests the efficiency of the representation as only contains 5011 labeled images train from. In this setting, we replaced the neural network classifier his used previously with Gaster-RCNN (Ren et al. 2015) image detection architecture, and use the pre-trained feature extractor on ImageNet. As before, we first trained the Paster-RRNI model while keeping the feature extractor fixed, then fine-tuned the entire model end-to-end. Table 3 displays our results compared to other methods. Most competing methods, which optimize a single unsupervised objective on ImageNet before fine-tuning on PASCAL detection, attain around 65% mean average precision. Leveraging larger unlabeled datasets increases their performance up to 67.8% (Caron et al./ 2019). Combining multiple forms of self-supervision enables them to reach 70.5% (Doersch & Zisserman 2017). The proposed method, which learns only from ImageNet data using single unsupervised objective, reaches 70.6% when equipped with a 101 feature extractor fa (as for most competing methods (Doersch & Zisserman (017)) but not all (Caron et al.) 2018 2019). Equipped with the more powerful ResNet-161 feature extractor fe, our method reaches 72.7%. Importantly, this result is only 2% short of the performance attained by purely supervised transfer learning, which we obtain by using all ImageNet labels before transferring to PASCAL.
DISCUSSION
We asked whether CPC could enable data-efficient image recognition, and found that it indeed greatly improves the accuracy of classifiers and object detectors when given small amounts of la- beled data. Surprisingly, CPC even improves results given ImageNet-scale labels. Our results show that there is still room for improvement using relatively straightforward changes such as augmenta- tion, optimization, and network architecture. Furthermore, we found that the standard method for evaluating unsupervised representations linear classification is only partially predictive of effi- cient recognition performance, suggesting that further research should focus on efficient recognition as a standalone benchmark. Overall, these results open the door toward research on problems where data is naturally limited, e.g. medical imaging or robotics.

---

Under review as a conference paper at ICLR 2020
Table Faster-RCNN by trained with 100% of PASCAL labels. Comparison of PASCAL 2007 image detection accuracy to other transfer methods. The supervised baseline Iearns from the entire labeled ImageNet dataset and fine-tunes for PASCAL detection. The second class of methods learns from the same unlabeled images before transferring. All of these methods pre-train on the ImageNet dataset, except for DeeperCluster which learns from the larger, but uncurated, YFCC100M dataset (Thomee et al. 2015). All results are reported in terms of mean average precision (mAP). i denotes methods implemented in this work.
Method
MAP
Transfer from labeled data: Supervised ResNett-152
74.7
Transfer from unlabeled data:
Exemplar (Ex) (Dosovitskiy et al. 2014)
60.9
Motion Segmentation (MS) (Pathak et al.
2016
61.1
Colorization (Col) (Zhang et 2016
65.5
Relative Position (RP) (Doersch et al.
2015)
66.8
Combination of Ex MS + Col + RP DDerschh & Zisserman 2017
70.5
Instance Discrimination (Wu et al. 2018)
65.4
Deep Cluster (Caron et al. 2018)
65.9
Deeper Cluster (Caron et al. 2019
67.8
Local Aggregation (Zhuagg et al. 2019
69.1
| trained CPC v2 (ResNet-101, fine-tuned)
70.6
"Faster-RCNN trained CPC v2 (ResNet-161, fine-tuned)
72.7
Furthermore, images are far from the only domain where unsupervised representation learning is important: for example, unsupervised learning is already a critical step in language (Mikolov et al, 2013 Devlin et al. 2018). and shows promise in domains like audio (van den Oord et al. [018]] Arandjelovic & Zisserman 2018 2017). video (Jing & Tian 2018 Misra et al. 2016), and robotic manipulation (Pinto & Gupta 2016 Pinto et al., 2016 Sermanet al., 2018). Currently much self-supervised work builds upon tasks tailored for a specific domain (often images), which may not be easily adapted other domains. Contrastive prediction methods, including the techniques suggested in this paper, are task agnostic and could therefore serve as unifying framework for integrating these tasks and modalities. This generality particularly useful given that many real- world environments are inherently multimodal, e.g. robotic environments which can have vision, audio, touch, proprioception, action, and more over long temporal sequences. Given the importance of increasing the amounts of self-supervision (via additional directions of prediction), integrating these modalities and tasks could lead to unsupervised representations which rival the efficiency and effectiveness of biological ones.
REFERENCES
Pulkit Agrawal, Joao Carreira, and Jitendra Malik. Learning to see by moving. In ICCV, 2015.
Relja Arandjelovic and Andrew Zisserman. Look, listen and learn. In Proceedings of the IEEE International Conference on Computer Vision, pp. 609-617, 2017.
Relja Arandjelovic and Andrew Zisserman. Objects that sound. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 435-451, 2018.
Lei Jimmy Ba, Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. CoRR, 160.00500 2016.
Philip Bachman, Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. arXiv preprint arXiv:1906.00910 2019.
H.B. Barlow. Unsupervised learning. Neural Computation, 1(3):295-311 -111, 1989. doi: 16. 1989.1.3.3 1.3.295.
10

---

Under review as a conference paper at ICLR 2020
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In The European Conference on Computer Vision (ECCV), September 2018.
Mathilde Caron, Piotr Bojanowski, Julien Mairal, and Armand Joulin. Leveraging large-scale uncu- rated data for unsupervised pre-training of visual features. 2019.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04800. 2018.
Carl Doersch and Andrew Zisserman. Multi-task self-supervised visual learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2051-2060, 2017.
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE International Conference on Computer Vision, pp- 1422-1430, 2015.
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. arXiv preprint arXiv:1907.02544, 2019.
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. arXiv preprint 17559:1600.09782, 2016.
Alexey Dosovitskiy, Jost Tobias Springenherg, Martin Riedmiller, and Thomas Brox. Discrimina- tive unsupervised feature learning with convolutional neural networks. In Advances in neural information processing systems, pp. 766-774, 2014.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge 2007 (voc2007) results. 2007.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint
2018.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In Ad- vances in neural information processing systems, pp. 529-536, 2005.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 297-304, 2010.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 20.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Earopean conference on computer vision, pp. 630-645. Springer, 2016b.
G.E. Hinton, TJJ. Sejnowski, H.H.M.C....JJ. Sejnowski, and T.A. Poggio. Unsupervised Learning: Foundations of Neural Computation. A Bradford Book. MIT Press, 1999. ISBN 9780262581684.
Olivier Hénaff, Robbe L. T. Goris, and Eero Simoncelli. Perceptual straightening of natural videos. Nature Neuroxcience, Apr 2019.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03110 2015.
Dinesh Jayaraman and Kristen Grauman. Learning image representations tied to ego-motion. In ICCV, 2015.
Longlong Jing and Yingli Tian. Self-supervised spatiotemporal feature learning by video geometric transformations. arXiv preprint arXiv:/811 1387, 2018

---

Under review as a conference paper at ICLR 2020
Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi Semi-supervised learning with deep generative models. In Advances in neural information processing systems, pp. 3581-3589, 2014.
Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. seessstinnneeffsspprrrvissssssss visuaaal representa tion learning. CoRR, abs/1901.09005,, 2019.
09000
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350(6266):1332-115 1338, 2015.
Barbara Landau, Linda B Smith, and Susan S Jones. The importance of shape in early lexical learning. Cognitive development, 3(3):299-321, 1988.
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Colorization as a proxy task for visual understanding. In CVPR, pp. 6874-6883, 2017.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521 (7553))333,,, 2015.
Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop Challenges in Representation Learning, ICML, volume 3, pp. 2, 2013.
Yin Li, Manohar Paluri, James M Rehg, and Piotr Dollár. Unsupervised learning of edges. In CVPR, 2016.
Ellen Markman. Categorization and naming in children: Problems of induction. mit Press, 1989.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representa- tions of words and phrases and their compositionality. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 26, pp. 3111-3119. Curran Associates, Inc., 2013.
Ishan Misra, C Lawrence Zitnick, and Martial Hebert. Shuffle and learn: unsupervised learning using temporal order verification. In ECCV, 2016.
Takeru Miyato, Shin-ichi Maeda, Shin Ishii, and Masanori Koyama. Virtual adversarial training: regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 2018.
Andriy Mnih and Koray Kavukcuoglu. Learning word embeddings efficiently with noise-contrastive estimation. In Advances in neural information processing systems, pp. 2265-2273, 2013.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations solving jigsaw puzzles. In European Conference on Computer Vision, pp. 69-84. Springer, 2016.
Stephanie E Palmer, Olivier Marre, Michael J Berry, and William Bialek. Predictive information in a sensory population. Proceedings of the National Acudemy of Sciences, 112(22):6908-6111 2015.
Deepak Pathak, Ross Girshick, Piotr Dollár, Trevor Darrell, and Bharath Hariharan. Learning fea- tures by watching objects move. arXiv preprint arXiv: 2016.
Lerrel Pinto and Abhinav Gupta. Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. In ICRA, 2016.
Lerrel Pinto, James Davidson, and Abhinay Gupta. Supervision via competition: Robot adversaries for learning tasks. arXiv preprint
2016.
Rajesh PN Rao and Dana H Ballard. Predictive coding the visual cortex: functional interpreta- tion of some extra-classical receptive-field effects. Nature neuroscience, 2(1):79, 1999.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pp. 91-999999999 2015.
12

---

Under review as a conference paper at ICLR 2020
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):11-252, 2015.
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 1134-1141. IEEE, 2018.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, (5(1):1929-1955) 2014.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. CoRR, abs/1409.4844 2014. Uttp://://arxx org/
4842
Bart Thomee, David Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: The new data in multimedia research. arXiv preprint arXiv:1503.018/7, 2015.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint arXiv: 1906:09900.05880 2019.
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.0374 2018.
Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using videos. In ICCV, 2015.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 7733-3742, 2018.
Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V. Le, Unsupervised Data Augmentation. arXiv e-prints, art. arXiv:1904. 12848. Apr 2019.
Amir Zamir, Tilman Wekel, Pulkit Agrawal, Colin Wei, Jitendra Malik, and Silvio Savarese. Generic 3D representation via pose estimation and matching. In ECCV, 2016.
Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer. S*L: Self-supervised semi- supervised learning, arXiv preprint UrXiv:1905.03670 2019.
Richard Zhang, Phillip Isola, and Alexei A Efros Colorful image colorization. In European confer- ence on computer vision, pp. 649-666. Springer, 2016.
Xiaojin Zhu and Zoubin Ghahramani. Learning from labeled and unlabeled data with label propa- gation. In Technical Report CMU-CALD-02-11 Carnegie Mellon University, 2002.
Chengxu Zhuang, Alex Lin Zhai, and Daniel Yamins. Local aggregation for unsupervised learning of visual embeddings. arXiv preprint arXiv: 1903. 12355, 2019.
13

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1
ADDITIONAL RESULTS
Table 4: Data efficient classification results with Top-I accuracy.
Labeled data Method
I %
2%
5%
10%
20%
50%
100%
Top-1 accuracy
Supervised trained pixels ReseNet trained on CPC v2 (fine-tuned)
23.1
34.8
50.2
60.0
67.6
74.4
78.0
46.3
54.7
64.1
69.7
73.0
77.3
80.6
Table 5: Data efficient classification results with Top-5 accuracy (data Fig. D
Labeled data Method
I %
2%
5%
10%
20%
50%
100%
Top-5 accuracy
Supervised trained on pixels ReseNet trained on CPC v2 (fine-tuned)
44.1
59.9
75.2
82.1
87.9
91.8
93.8
72.9
79.6
86.1
89.5
91.4
93.6
95.2
A.2 INFONCE IMPLEMENTATION
For completeness, provide pseudo-code for the main calculations involved in the InfoNCE ob- jective, loosely modeled after Tensorflow operations. We suppose we have just calculated set of latents žι (xra)) for i, (1,.... 7), each being e.g. 4096-dimensional vector. As- suming we do so for a batch B images (æ), the set of latents is a tensor of size x 7 x 7 x 4096.
def CPC (latents, target_dim=fe emb_scale=0,
steps_to_igmore=2, steps_to_predictte latents: (B, H, W, D]
loss = 0.0
context pixelCNN (larents) targets Conv2D (output muutputt_hhaneelstarrre dim,
kernel_shape= 1)) (latents)
batch_dii col_dim, row_dim targets. sargets..hape[=-11 targets reshape (target, [-1, target_dim) for i in range (steps_to_ignore, steps_to_predice
col vol_dim_ii col_dim - total_elemens batch_dim col col_dim_i dim
preds_i Conv2D (owtppptt_caameeeesstttteetdddde
kernel_shape= (1, 1)) (context)
preds_i preds_il;, :- (i+1), ;, :] emb_scale
preds reshape (preds_ir i, [-1, target_dim])
logits matmul (preds_,,, targets, transpose_b-True)
b range (totallelements) / (col_dim_i row_dim)
col = range (total_elemente
labels col dim im_row_din (i+1) row dim col
loss += oeftmaxn_rross.tttroyy...t... (logits, labels)
return loss
14

---

Under review as a conference paper at ICLR 2020
def pixelcM CNN (latents):
# latents: [B, H, cres = latents
cres dim cres, seessshae[[1 [-1]
for in range (5)
c Conv2D (otttouutttccceeeelss222
kernel_shape= (1, 1)) (cres)
c ReLU (c)
= Conv2D (output__chanrels2255,
hernel_shape-de 3)) (c)
(a, [[O, 0], [1, 0], [0, 01, [0, 011)
c Conv2D (output__haanels-2555
kernel_shape= (2, 1),
type='VALID') (c)
c ReLU (c)
c = Conv2D (outpptt_chammmelsssrrs_.dd
kernel_shape=(1 1) (c)
cres cres +
cres ReLU (cres)
return cres
A.3 LINEAR CLASSIFICATION
Model architecture: Having extracted 80x80 patches with a stride of 32x32 from a 240x240 shaped input image, we end up with a grid of 6x6 features (each of which is obtained from our ResNet-161 architecture). This gives us a [6,6,4096] tensor for the image. We then use Batch-Normalization layer to normalize the features (without scale parameter) followed by a 1x1 convolution mapping each feature in the grid to the 1000 logits for ImageNet classification. We then spatially-mcan-por these logits to end up with the final log probabilities for the linear classification.
We use the Inception preprocessing (Szegedy et al., 2014) to extract 240x240 crops from the raw image. The image divided into subcrops as per CPC prepppppppoocssingggg used for CPC pre-training.
Optimization details: We use Adam Optimizer with learning rate of 5e-4. We train the model batch size of 512 images with 32 images per core spread over 16 workers.
A.4 EFFICIENT CLASSIFICATION PURELY SUPERVISED
In order to find the best model within this class, vary the following hyperparameters:
Model architecture: We investigate using ResNet-50, ResNet-101, and ResNet-152 model architectures, all of them using the 'v2' variant (He al. 2016b), and find larger archi- tecture to perform better, even when given smaller amounts of data. We insert DropOut layer before the final linear classification layer (Srivastava al., 2014).
Data pre-processing: We use the Inception pre-processing pipeline (Szegedy al. 2014).
Optimization details: We vary the learning rate in {0.05, 0. 0.2), the weight decay log- arithmically from 10-5 to 10-2, the DropOut linearly from to 1, and the batch size per worker in { (16, 32).
We chose the best performing model for each training subset Dy of labeled ImageNet (using a sepa- rate validation set), and report accuracy on the test set (i.e. the publicly available ILSVRC-2012 validation set).
15


Reviews:
[{"review_id": "ryxqDXApcr", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to use Contrastive Predictive Coding (CPC), an unsupervised learning approach, to learn representations for further image classification. The authors show that using CPC for representation learning allows to achieve better results than other self-supervised methods. Moreover, CPC is shown to be useful for semi-supervised learning (on par with SOTA method), and transfer learning. All results are very impressive and is in-line with current trends of using a linear classifier on top of a deep feature extractor (e.g., Nalisnick et al., \"Hybrid Models with Deep and Invertible Features\"). The paper is rather well written and the results are convincing. However, The whole idea of the paper is based on the original paper:\n* Oord, Aaron van den, Yazhe Li, and Oriol Vinyals. \"Representation learning with contrastive predictive coding.\" arXiv preprint arXiv:1807.03748 (2018).\nTechnically speaking, the paper is outstanding, but it lacks novelty in terms of new ideas. I highly appreciate new results and new architectures, but it is not enough for a full conference paper.\n\nRemarks\n- In Section 2.1, the problem statement for Contrastive Predictive Coding (CPC) is unclear. For instance, the authors explain CPC by mentioning about masked convolutional layers that is unnecessary at this point. I understand that from engineering perspective it is crucial information, but it does not help to understand CPC. As a result, without knowing the original paper on CPC, Section 2.1 is hard to follow.\n\n- The paper can be treated as an uptaded version of the original CPC paper. I really appreciate all new results and implementation of the idea. The paper is well written and it is technically correct. However, I do not find much novelty compared to the original paper. This would be a perfect workshop contribution, but I am afraid that it is not enough for a full paper.\n\n==== AFTER REBUTTAL ====\nI would like to thank the reviewers for their rebuttal. It was not my intention to dismiss your effort in providing new technical results. Please forgive me if you read it in this way. My point is that the paper presents exactly the same idea as the original paper of CPC, but with new, very interesting results. However, I doubt if this is enough for a full conference paper. This point is debatable and I would be happy to further discuss it with other reviewers and the AC. At this point, I keep my original score, but of I am open for a discussion.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Sye7g-moFH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper improves Contrastive Predictive Coding method and reaches a good performance in several downstream tasks. However, the novelty and technical contributions are limited.\n\nStrengths: \n+ The experimental results seem good. The reimplemented CPC v2 performs much better than the original version. And the performance of down-stream tasks is comparable or better than the state-of-the-art methods.\n+ The paper is well written. The paper structure is clear and figures are well illustrated.\n+ Figure 3 shows clearly the performance improvements of a series of incremental modifications to the original CPC methods.\n\nWeaknesses:\n- The novelty and technical contributions are limited. This paper only proposes some minor improvements based on the original CPC method and use a deeper network to get better performance. The proposed method lacks of important insights for the research community.\n- The capacity of network architecture is crucial for self-supervised learning. But in Table 1,2,3, the network architecture of the proposed method is deeper than that in the comparison methods, which is unfair for the comparison methods. Meanwhile, the network architectures of many compared methods are not listed in the tables, which may be misleading. For example, Unsupervised Data Augmentation (Xie et al., 2019) in table 2 and Instance Discrimination (Wu et al., 2018) in table 3 use ResNet50, which is much more shallow than ResNet-161 in this paper.\n- In section 2.1, the paper doesn't describe clearly what's the input of masked convolutional network $g_{\\phi}$ and how to calculate $c_{i, j}$.\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "HJxj6VX0tr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors augment contrastive predictive coding (CPC), a recent representation learning technique organized around making local representations maximally useful for predicting other nearby representations, and evaluates their augmented architecture in several image classification problems. Although the modifications to CPC aren't particularly original, the authors show first that these yield a significant improvement in linear classification accuracy. They then use this improved model to obtain impressive performance in classification within semi-supervised and transfer learning settings, giving strong support for the use of such methods within image processing applications.\n\nPros:\nOwing to its generality (CPC assumes only a weak spatial prior in the input data), and cheap computational cost relative to earlier generative approaches, CPC is already a promising unsupervised representation learning technique. The paper gives more evidence of this usefulness for image data, yielding leading performance on several different image classification benchmarks.\n\nThe authors also make the observation that linear separability, the standard benchmark for evaluating unsupervised representations, correlates poorly with efficient prediction in the presence of limited labeled data. This observation should be of interest in the broader community, and points to the need for more diverse metrics for unsupervised representations.\n\nCons:\nThe improvements given in the paper are quite useful within their stated domain (image data), but aren't directly applicable to other types of input data. Although the authors make a point of emphasizing the relevance of CPC for other problem domains, they don't currently provide any suggestions for how this current work could be generalized to handle these other cases. In this sense, I think it is a bit deceptive to refer to their model as \"CPC v2\", as the majority of their changes have no bearing on the intrinsic CPC algorithm itself.\n\nI am sure that some of the methods used here could lead to improvements in the use of CPC for other types of data, but the authors currently don't provide any insight on this issue. In line with that, I think their work would be improved by some commentary on this, in particular by any concrete suggestions they have about how similar augmentations to CPC could be carried out in text, audio, and/or video data.\n\nVerdict:\nOwing to the reasons given above, I recommend acceptance.\n\nMinor suggestions:\nPlease use a different color scheme for your figures that is still meaningful if the paper is printed in greyscale.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BJg06Ig69B", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Title: DATA-EFFICIENT IMAGE RECOGNITION\n[Summary]\n-This paper introduces Contrastive Predictive Coding (CPC) image recognition in the data-efficient regime. Concretely, the authors improve CPC in terms of its architecture and training strategy. The extensive experiments show that CPC enables data-efficient image classification and surpassed other unsupervised approaches. \n\n[Pros]\n- Although the CPC was proposed and evaluated in vision task in [1], a new implementation of CPC with dramatically-improved ability is presented in this paper.\n- The CPC is utilized to enhance spatially predictable representations which benefits a lot data-efficient image recognition.\n\n[Cons]\n- In Sec. 4.1, four axes are identified to upgrade CPC v1 to CPC v2. But they are not well motivated. More discussions about why this four axes are investigated in image recognition.\n\n-The core idea is motivated by a critical hypothesis that good representations should make spatio-temporal variability in natural signals more predictable. However, this hypothesis is not well verified. The concept of amount of ‘predictability’ in page 7 is not clear. It would be great if you provide more evidence that the improvement in low-data classification results from the increased ‘predictability’.\n\n- The comparison in Sec. 4.3 seems unfair. The pretrain model trained with different methods should be the same. For example, the Faster RCNN trained on CPC v2 uses ResNet-101 as backbone but Local Aggregation method uses ResNet-50.\n\n[Summary]\n- This work extends CPP to data-efficient image recognition by simply adjusting network architectures and training strategies, which makes it less interesting. Besides, the major hypothesis is not well validated. \n- The experimental results are convincing and encouraging. Some minor flaws such as unfair comparison should be fixed.\n- I want to see how the four axes in Sec. 4.1 are related to core motivation (more predictable) since they are major adjustments from CPP v1 to CPP v2. If the author provides a profound explanation of the problem, I would consider changing the rating. \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper tackles the key question of achieving high prediction performances with few labels. The proposed approach builds upon Contrastive Predictive Coding (van den Oord et al. 2018). The contribution lies in i) refining CPC along several axes including model capacity, directional predictions, patch-based augmentation; ii) showing that the refined representation learned by the called CPC.v2 supports an efficient classification in a few-label regime, and can be transferred to another dataset; iii) showing that the auxiliary losses involved in the CPC are not necessarily predictive of the eventual performance of the network.

This paper generated a hot discussion. Reviewers were not convinced that the paper contributions are sufficiently innovative to deserve being published at ICLR. Authors argued that novelty does not have to lie in equations, and that the new ideas and evidence presented are worth. 

The area chair thinks that the paper raises profound questions (e.g., what auxiliary losses are most conducive to learning a good representation; how to divide the computational efforts among the preliminary phase of representation learning and the later phase of classifier learning), but given the number of options and details involved, these results may support several interpretations besides the authors'. 

The authors might also want to leave the claim about the generality of the CPC++ principles (e.g., regarding audio) for further work - or to bring additional evidence backing up this claim. 

In conclusion, this paper contains brilliant ideas and I hope to see them published with a strengthened analysis of its components. 
Decision: Reject