A critical analysis of self-supervision, or what we can learn from a single image
Authors: Asano YM.
Rupprecht C.
Vedaldi A.
Venue:  (2020)
Keywords: self-supervision
feature representation learning
CNN

Abstract: We look critically at popular self-supervision techniques for learning deep convolutional neural networks without manual labels. We show that three different and representative methods, BiGAN, RotNet and DeepCluster, can learn the first few layers of a convolutional network from a single image as well as using millions of images and manual labels, provided that strong data augmentation is used. However, for deeper layers the gap with manual supervision cannot be closed even if millions of unlabelled images are used for training.
We conclude that:
(1) the weights of the early layers of deep networks contain limited information about the statistics of natural images, that
(2) such low-level statistics can be learned through self-supervision just as well as through strong supervision, and that
(3) the low-level statistics can be captured via synthetic transformations instead of using a large image dataset.

Published as a conference paper at ICLR 2020
A CRITICAL ANALYSIS OF SELF-SUPERVISION, OR WHAT WE CAN LEARN FROM A SINGLE IMAGE
Yuki M. Asano
Christian Rupprecht
Andrea Vedaldi
Visual Geometry Group University of Oxford
{yuki, chrisr, vedaldi sroboos ox, ac.
ABSTRACT
We look critically at popular self-supervision techniques for learning deep convo- lutional neural networks without manual labels. We show that three different and representative methods, BiGAN. RotNet and DeepCluster, can learn the first few layers of a convolutional network from single image as well as using millions of images and manual labels, provided that strong data augmentation used. How- ever, for deeper layers the gap with manual supervision cannot be closed even if millions of unlabelled images are used for training. We conclude that: (1) the weights of the early layers of deep networks contain limited information about the statistics of natural images, that (2) such low-level statistics can be learned through self-supervision just well as through strong supervision, and that (3) the low-level statistics can be captured via synthetic transformations instead using large image dataset.
INTRODUCTION
Despite tremendous progress in supervised learning, learning without external supervision remains difficult. Self-supervision has recently emerged as one of the most promising approaches to address this limitation. Self-supervision builds on the fact that convolutional neural networks (CNNs) trans- fer well between tasks (Shin et al., 2016; Oquab et al., 2014; Girshick, 2015; Huh et al., 2016). The idea then is to pre-train networks via pretext tasks that do not require expensive manual annotations and can be automatically generated from the data itself. Once pre- pre-trained, networks can be applied to target task by using only a modest amount of labelled data.
Early successes in self-supervision have encouraged authors to develop large variety of pretext tasks, from colorization to rotation estimation and image autoencoding. Recent papers have shown performance competitive with supervised learning by learning complex neural networks on very large image datasets. Nevertheless, for a given model complexity, pre-training by using an off-the- shelf annotated image datasets such ImageNet remains much more efficient.
In this paper, we aim to investigate the effectiveness of current self-supervised approaches by char- acterizing how much information they can extract from a given dataset of images. Since deep net- works learn a hierarchy of representations, we further break down this investigation on a per-layer basis. We are motivated by the fact that the first few layers of most networks extract low-level in- formation (Yosinski et al., 2014), and thus learning them may not require the high-level semantic information captured by manual labels.
Concretely, in this paper we answer the following simple question: "is self-supervision able to exploit the information contained in a large number of images in order learn different parts of a neural network?"
We contribute two key findings. First, we show that as little as a single image sufficient, when combined with self-supervision and data augmentation, to learn the first few layers of standard deep networks as well as using millions images and full supervision (Figure 1). Hence, while self- supervised learning works well for these layers, this may be due more the limited complexity of such features than the strength of the supervisory technique. This also confirms the intuition that early layers in convolutional network amounts to low-level feature extractors, analogous to early

---

Published as a conference paper at ICLR 2020
Linear Classifier on ImageNet
100
% suparvited partomance
40
Rottet
Single-image self-supervision. We show that several superviipn methods can be used to train the first few layers of deep neural networks using a single training image, such as this Image A, B or even C (above), provided that sufficient data augmentation is used.
Figure 1:
RottNet
20-
1-BIGAN
DeepCluster
Dluustr
conv1 canc2
000v3
cocld
conc5
learned and hand-crafted features for visual recognition (Olshausen & Field, 1997; Lowe, 2004; Dalal & Triggs, 2005). Finally, it demonstrates the importance of image transformations in learning such low-level features as opposed to image diversity."
Our second finding about the deeper layers of the network. For these, self-supervision remains inferior to strong supervision even if millions of images are used for training. Our finding is that this is unlikely to change with the addition of more data. In particular, we show that training these layers with self-supervision and single image already achieves much as two thirds of the performance that can be achieved by using a million different images.
We show that these conclusions hold true for three different sulf--spe methods, BiGAN (Don- ahue et al., 2017), RotNet (Gidaris et al., 2018) and DeepCluster (Caron et al., 2018), which are representative of the spectrum of techniques that are currently popular. We find that performance as a function of the amount of data is dependent on the method, but all three methods can indeed leverage a single image learn the first few layers of a deep network almost "perfectly".
Overall, while our results do not improve self-supervision per-se, they help to characterize the limi- tations of current methods and to better focus on the important open challenges.
RELATED WORK
Our paper relates three broad areas of research: (a) seffissuprriissdnssuurriss learning, (b) learning from a single sample, and (c) designing/leariig low-level feature extractors. We discuss closely related work for each.
Self-supervised learning: A wide variety of proxy tasks, requiring no manual annotations, have been proposed for the self-training of deep convolutional neural networks. These methods use various cues and tasks namely, in-painting (Pathak et al., 2016), patch context and jigsaw puz- zles (Doersch al., 2015; Noroozi & Favaro, 2016; Noroozi et al., 2018; Mundhenk et al., 2017), clustering (Caron et al., 2018), noise-as-targed (Bojanowski & Joulin, 2017), colorization (Zhang et al., 2016; Larsson al., 2017), generation (Jenni & Favaro, 2018; Ren & Lee, 2018; Donahue et al., 2017), geometry (Dosovitskiy et al., 2016; Gidaris et al., 2018) and counting (Noroozi et al., 2017). The idea is that the pretext task can be constructed automatically and easily on images alone. Thus, methods often modify information in the images and require the network to recover them. In- painting or colorization techniques fall in this category. However these methods have the downside that the features are learned on modified images which potentially harms the generalization to un- modified ones. For example, colorization uses gray scale image as input, thus the network cannot learn to extract color information, which can be important for other tasks.
Slightly less related are methods that use additional information to learn features. Here, often temporal information is used in the form of videos. Typical pretext tasks are based on temporal- context (Misra et al., 2016; Wei et al., 2018 8; Lee et al., 2017; Sermanet et al., 2018), spatio-temporal
'Example applications that only rely on low-level feature extractors include template matching (Kat et al., 2018; Talmi et al., 2017) and style transfer (Gatys et al., 2016; Johnson et al., 2016), which currently rely on pre-training with millions of images.
2

---

Published as a conference paper at ICLR 2020
cues (Isola et al., 2015; Gao et al., 2016; Wang et al., 2017), beegrouund-acckrrouud segmentation via video segmentation (Pathak et al., 2017), optical-flow (Gan et al., 2018; Mahendran et al., 2018), synthesis (Srivastava et al., 2015), audio prediction from video (de Sa, 1994; Owens et al., 2016), audio-video alignment (Arandjelović & Zisserman, 2017), ego-motion estimation (Ja- yaraman & Grauman, 2015), slow feature analysis with higher order temporal coherence (Jayara- man & Grauman, 2016), transformation between frames (Agrawal et al., 2015) and patch tracking in videos (Wang& & Gupta, 2015). Since we are interested in learning features from as little data as one image, we cannot make use of methods that rely on video input.
Our contribution inspects three unsupervised feature learning methods that use very different means of extracting information from the data: BiGAN (Donahue et al., 2017) utilizes generative adver- sarial task, RotNet (Gidaris al., 2018) exploits the photographic bias in the dataset and DeepClus- ter (Caron et al., 2018) 18) learns stable feature representations under number of image transformations by proxy labels obtained from clustering. These are described in more detail in the Methods section.
Learning from a single sample: In some applications of computer vision, the bold idea of learn- ing from a single sample comes out of necessity. For general object tracking, methods such as max margin correlation filters (Rodriguez et al., 2013) learn robust tracking templates from a single sample of the patch. A single image also be used to learn and interpolate multi-scale textures with a GAN framework (Rott Shaham et al., 2019). Single sample learning was pursued by the semi-parametric exemplar SVM model (Malisiewicz et al., 201 1). They learn one SVM per positive sample separating it from all negative patches mined from the background. While only one sam- ple used for the positive set, the negative set consists of thousands of images and a necessary component of their method. The negative space was approximated by a by the Exemplar LDA (Hariharan al.. 2012). These SVMs, one per positive sample, are pooled together using max aggregation. We differ from both of these approaches in that we do not use large collection of negative images to train our model. Instead we restrict ourselves to a single or a few images with systematic augmentation strategy.
Gaussian
Classical learned and hand-crafted low-level feature extractors: Learning and hand-crafting features pre-dates modern deep learning approaches and self-supervision techniques. For example the classical work of (Olshausen & Field, 1997) shows that edge-like like filters can be learned via sparse coding of just 10 natural scene images. SIFT (Lowe, 2004) and HOG (Dalal & Triggs, 2005) have been used extensively before the advent of convolutional neural networks and, in many ways, they resemble the first layers of these networks. The scatter transform of Bruna & Mallat (2013); Oyallon et al. (2017) is handerafted design that aims replacing least the first few layers of a deep network. While these results show that effective low-level features can be handcrafted, this insufficient to clarify the power and limitation of self-supervision in deep networks. For instance, it is not obvious whether deep networks can learn better low level features than these, how many images may be required to learn them, and how effective self-supervision may be in doing so. For instance, as we also show in the experiments, replacing low-level layers in a convolutional networks with handcrafted features such as Oyallon et al. (2017) may still decrease the overall performance of the model. Furthermore, this says little about deeper layers, which we also investigate.
In this work we show that current deep learning methods learn slightly better low-level representa- tions than hand crafted features such the scattering transform. Additionally, these representations can be learned from one single image with augmentations and without supervision. The results show how current self-supervised learning approaches that one million images yield only rel- atively small gains when compared to what can be achieved from one image and augmentations, and motivates renewed focus on augmentations and incorporating prior knowledge into feature extractors.
3 METHODS
We discuss first our data and data augmentation strategy (section 3.1) and then we summarize the three different methods for unsupervised feature learning used in the experiments (section 3.2).
3

---

Published as a conference paper at ICLR 2020
3.1 DATA
Our goal is to understand the performance of representation learning methods as a function of the image data used to train them. To make comparisons as fair as possible, we develop protocol where only the nature of the training data is changed, but all other parameters remain fixed.
In order to do so, given baseline method trained on d source images, we replace those with another set of d images. Of these, now only N d are source images (i.e. i.i.d. samples), while the remaining d-N .N are augmentations of the source ones. Thus, the amount of information in the training data is controlled by N and we can generate a continuum of datasets that vary from one extreme, utilizing single source image 1, to the other extreme, using all N= d original training set images. For example, if the baseline method is trained on ImageNet, then d= 1,281,167, When N = 1, it means that we train the method using single source image and generate the remaining 1,281,166 images via augmentation. Other baselines use CIFAR-10/100 images, so in those cases d 50,000 instead.
The data augmentation protocol, is an extreme version of augmentations already employed by most deep learning protocols. Each method we test, in fact, already performs some data augmentation internally. Thus, when the method is applied on our augmented data, this can be equivalently thought of as incrementing these "native" augmentations by concatenating them with our own.
Choice of augmentations. Next, we describe how the N source images are expanded to additional d-N images so that the models can be trained on exactly d images, independent from the choice of N. The idea is to use an aggressive form of data augmentation involving cropping, scaling, rotation, contrast changes, and adding noise. These transformations are representative of invariances that one may wish to incorporate in the features. Augmentation can be seen as imposing prior on how we expect the manifold of natural images to look like. When training with very few images, these priors become more important since the model cannot extract them directly from data.
Given a source image of size size H x W, we first extract certain number of random patches of size (w,h) where w: < W and h < H satisfy the additional constraints / S wh and y S h I y- WH Thus, the smallest size of the crops is limited to be at least 3WH and at most the whole image. Additionally, changes to the aspect ratio are limited by 7. In practice we use B 10-3 and
Second, good features should not change much by small image rotations, so images are rotated (before cropping to avoid border artifacts) by E (-35, 35) degrees. Due to symmetry in image statistics, images are also flipped left-to-right with 50% probability.
Illumination changes common in natural images, thus expect image features to be robust to color and contrast changes. Thus, we employ a set of linear transformations in RGB space to model this variability in real data. Additionally, the color/intensity of single pixels should not affect the feature representation, as this does not change the contents of the image. To this end, color jitter with additive brightness, contrast and saturation are sampled from three uniform distributions in (0.6, 1.4) and hue noise from (-0.1, 0.1) is applied to the image patches. Finally, the cropped and transformed patches are scaled to the color range (-1, 1) and then rescaled to full S S resolution to be supplied to each representation learning method, using bilinear interpolation. This formulation ensures that the patches are created in the target resolution S, independent from the size and aspect ratio W. H of the source image.
Real samples. The images used for the 1 and N = 10 experiments are shown in Figure I and the appendix respectively (this is all the training data used in such experiments). For the special case of using a single training image, i.e. N 1, we have chosen one photographic (2560x x 1920) and one drawn image (600 225), which we call Image and Image B, respectively. The two images were manually selected as they contain rich texture and are diverse, but their choice was not optimized for performance. We test only two images due to the cost of running a full set of experiments (each image is expanded up to 1 1.2M times for training some of the models, as explained above). However, this is sufficient to prove our main points. We also test another (1165 x 585) Image C to ablate the "crowdedness" of an image, as this latter contains large areas covering objects. While resolution matters to some extent as a bigger image contains more pixels, the information within is still far more correlated, and thus more redundant than sampling several smaller images. In particular, the resolution difference in Image A and B appears to be negligible in our experiments. For CIFAR-10, where S = 32 we only use Image B due the resolution difference. In direct comparison, Image
4

---

Published as a conference paper at ICLR 2020
is the size of about 132 CIFAR images which is still much less than d = 50,000. For N> 1, select the source images randomly from each method's 's training set.
3.2 REPRESENTATION LEARNING METHODS
Generative models. Generative Adversarial Networks (GANs) (Goodfellow al., 2014) learn to generate images using an adversarial objective: a generator network maps noise samples to image samples, approximating a target image distribution and discriminator network is tasked with dis- tinguishing generated and real samples. Generator and discriminator are pitched one against the other and learned together, when an equilibrium is reached, the generator produces images indistin- guishable (at least from the viewpoint of the discriminator) from real ones.
Bidirectional Generative Adversarial Networks (BiGAN) (Donahue et al., 2017: Dumoulin et al,, 2016) are an extension of GANs designed to learn a useful image representation as an approximate inverse of the generator through joint inference on an encoding and the image. This method's na- tive augmentation uses random crops and random horizontal flips to learn features from S = 128 sized images. As opposed to the other two methods discussed below it employs leaky ReLU non- linearities is typical in GAN discriminators.
Rotation. Most image datasets contain pictures that are 'upright' this is how humans prefer to take and look at them. This photographer bias can be understood as a form implicit data labelling. RotNer (Gidaris et al., 2018) exploits this by tasking a network with predicting the upright direction of a picture after applying to it a random rotation multiple of 90 degrees (in practice this is formulated as 4-way classification problem). The authors reason that the concept of 'upright' requires learning high level concepts in the image and hence this method is not vulnerable to exploiting low-level visual information, encouraging the network to learn more abstract features. In our experiments, we test this hypothesis by learning from impoverished datasets that may lack the photographer bias. The native augmentations that RotNet uses on the S 256 inputs only comprise horizontal flips and non-scaled random crops to 224 x 224.
Clustering. DeepCluster (Caron et al., 2018) is recent state-of-the-art unsupervised represen- tation learning method. This approach alternates k-means clustering to produce pseudo-labels for the data and feature learning to fit the representation to these labels. The authors attribute the suc- cess of the method to the prior knowledge ingrained in the structure of the convolutional neural network (Ulyanov et al., 2018).
The method alternatives between a clustering step, in which k-means is applied on the PCA-reduced features with A 10a, and learning step, in which the network is trained predict the cluster ID for each image under a set of augmentations (random resized crops with 3 = 0.08, & and horizontal flips) that constitute its native augmentations used on top of the S= 256 input images.
EXPERIMENTS
We evaluate the representation learning methods ImageNet and CIFAR-10/100 using linear probes (Section 4.).. After ablating various choices of transformations in our augmentation pro- tocal (Section 4.2), we move to the question of the paper: whether large dataset is beneficial to unsupervised learning, especially for learning early convolutional features (Section 4.3).
4.1 LINEAR PROBES AND BASELINE ARIINNEERRCHEEE ARCHITECTURE
In order to quantify if neural network has learned useful feature representations, we follow the standard approach of using linear probes (Zhang et al., 2017). This amounts to solving difficult task such as ImageNet classification by training a linear classifier on top of pre-trained feature rep- resentations, which are kept fixed. Linear classifiers heavily rely on the quality of the representation since their discriminative power is low.
We apply linear probes to all intermediate convolutional layers of networks and train on the Ima- geNet LSVRC-12 (Deng et al., 2009) and SEFAR-10/100 (Krizhevsky, 2009) datasets, which are the standard benchmarks for evaluation in self-supervised learning. Our base encoder architecture is AlexNet (Krizhevsky et al., 2012) with BatchNorm, since this is good representative model and is most often used in other unsupervised learning work for the purpose of benchmarking. This model
5

---

Published as a conference paper at ICLR 2020
CIFAR-10
Table Ablating data augmentation using
convl
conv2
conv3 conv4
MonoGAN (left). Training linear classifier on the features extracted different depths
(a) Fully sup.
66.5
70.1
75.9
(b) Random feat.
57.8
55.5
54.2
47.3
of the network for CIFAR-10.
(c) No aug.
57.9
56.2
54.2
47.8
(d) Jitter
58.9
58.0
49.8
Table 2: ImageNet LSVRC-12 linear prob- ing evaluation (below). A linear classifier is trained on the (downsampled) activations of each layer in the pretrained model. We re-
(e) Rotation
61.4
58.8
47.5
(f) Scale
67.9
69.3
67.9
59.1
(
Rot. & jitter
64.9
63.6
61.0
53.4
(h) Rot. scale
67.6
69.9
68.0
60.7
i Jitter & scale
68.1
71.3
69.5
62.4
port classification accuracy averaged 10
crops. The indicated that numbers are taken from (Zhang et al., 2017).
All
68.1
72.3
70.8
63.5
ILSVRC-12
Method, Reference
#images
convl
conv2
conv3
conv4
conv5
Full-supervision
1,281,167
19.3
36.3
44.2
48.3
50.5
(b)
(Oyallon al., 2017): Scattering
0
18.9
(c) Random
11.6
17.1
16.9
16.3
14.1
(d)
(Krähenbühl al., 2016)sk-means*
R: 160
17.5
23.0
24.5
23.2
20.6
(c)
(Donahue et al,, 2017): BIGAN=
1,281,167
24.5
31.0
29.9
28.0
(f)
mono, Image
20.4
30.9
33.4
28.4
16.0
(g) mono, Image
20.5
30.4
31.6
27.0
16.8
(h) deka
10
16.2
16.5
16.5
13.1
7.5
kilo
1,000
16.1
17.7
18.3
17.6
13.5
(
(Gidaris et al., 2018): RotNet
1,281,167
18.8
31.7
38.7
38.2
36.5
(K)
mono, Image A
19.9
30.2
30.6
27.6
21.9
(I)
mono, Image B
17.8
27.6
27.9
25.4
20.2
(m) deka
19.6
30.7
32.6
28.9
22.6
kilo
1,000
21.0
33.5
36.5
34.0
29.4
(0)
(Caron et al., 2018): DeepCluster
1,281,167
18.0
32.5
39.2
37.2
30.6
mono, Image A
20.7
31.5
32.5
28.5
21.0
mono, Image B
19.7
30.1
31.6
28.5
20.4
mono, Image C
18.9
29.2
31.5
28.9
23.5
(S) deka
10
18.5
29.0
31.1
28.2
21.9
kilo
1,000
19.5
29.8
33.0
31.7
26.8
Table 3: CIFAR-10/100. Accuracy of linear classifiers on different network layers.
Dataset
CIFAR-10
CIFAR-100
Model
convl
conv2
conv3
conv4
convl
conv2
conv3
conv4
Fully supervised
66.5
70.1
72.4
75.9
38.7
43.6
44.4
46.5
Random
57.8
55.5
54.2
47.3
30.9
29.8
28.6
24.1
RotNet
64.4
65.6
65.6
59.1
36.0
35.9
34.2
25.8
GAN (CIFAR-10)
67.7
73.0
72.5
69.2
39.6
46.0
45.1
39.9
GAN (CIFAR-100)
38.1
42.2
44.0
46.6
MonoGAN
68.1
72.3
70.8
63.5
39.9
46.9
44.5
38.8
has five convolutional blocks (each comprising a linear convolution later followed by ReLU and optionally max pooling). We insert the probes right after the ReLU layer in each block, and denote these entry points convl to conv5. Applying the linear probes at each convolutional layer allows studying the quality of the representation learned different depths of the network.
Details. While linear probes conceptually straightforward, there several technical details that affect the final accuracy by few percentage points. Unfortunately, prior work has used several slightly different setps,, so that comparing results of different publications must be done with cau- tion. To make matters more difficult, not all papers released evaluation source code. We prove this standardized testing code here?.
Phttps://gr thub. cot/ www.kiimmaa/////c inear-probes

---

Published as a conference paper at ICLR 2020
In our implementation, we follow the original proposal (Zhang et al., 2017) in pooling each rep- resentation to a vector with 9600, 9216, 9600, 9600, 9216 dimensions for convl-5 using adaptive max max-pooling, and absorb the batch normalization weights into the preceding convolutions. For eval- uation on ImageNet we follow RotNet to train linear probes: images are resized such that the shorter edge has length of 256 pixels, random crops of 224 224 are computed and flipped horizontally with 50% probability. Learning lasts for 36 epochs and the learning rate schedule starts from 0.01 and is divided by five epochs 5, 15 and 25. The top-1 accuracy of the linear classifier is then measured on the ImageNet validation subset. This uses DeepCluster's 's protocol, extracting 10 crops for each validation image (four the corners and one at the center along with their horizontal flips) and averaging the prediction scores before the accuracy is computed. For CIFAR-10/100 data, we follow the same learning rate schedule and for both training and evaluation we do not reduce the dimensionality of the representations and keep the images' original size of 32 x22.
4.2 EFFECT OF AUGMENTATIONS
In order to better understand which image transformations are important to learn a good feature representations, we analyze the impact of augmentation settings. For speed, these experiments are conducted using the CIFAR- 10 images (d 50, 000 in the training set) and with the smaller source Image B and a GAN using the Wasserstein GAN formulation with gradient penalty (Gulrajani et al., 2017). The encoder is a smaller Alex CNN consisting of four convolutional layers (kernel sizes: 7, 3, 3; strides: 3,2, 2, 1) followed by a single fully connected layer as the discriminator. Given that the GAN is trained on a single image (w/ augmentations), we call this setting MonoGAN.
Table I reports all 2 combinations of the three main augmentations (scale, rotation, and jitter) and a randomly initialized network baseline (see Table I (b)) using the linear probes protocol discussed above. Without data augmentation the model only achieves marginally better performance than the random network (which also achieves a non-negligible level of performance (Ulyanov et al., 2017; Caron al., 2018)). This is understandable since the dataset literally consists of a single training image cloned d times. Color jitter and rotation slightly improve the performance of all probes by 1- 2% points, but random rescaling adds least ten points every depth (see Table (f,h,i)) and is the most important single augmentation. A similar conclusion can be drawn when two augmentations are combined, although there are diminishing returns as more augmentations are combined. Overall, we find all three types of augmentations are of importance when training in the ultra-low data setting.
4.3 BENCHMARK EVALUATION
We analyze how performance varies function N, the number of actual samples that are used to generated the augmented datasets, and compare it to the gold-standard setup (in terms of choice of training data) defined in the papers that introduced each method. The evaluation is again based on linear probes (Section 4...
Mono is enough. From Table 2 we make the following observations. Training with just a single source image (f.g.k.l.p.)) is much better than random initialization (c) for all layers. Notably, these models also outperform Gabor-like filters from Scattering networks (Bruna & Mallat, 2013), which are hand crafted image features, replacing the first two convolutional layers as in (Oyallon et al., 2017). Using the same protocol as in the paper, this only achieves an accuracy of 18.9% compared to (p)'s conv2 > 30%.
More importantly, when comparing within pretext task, even with one image we are able to improve the quality of convl-conv3 features compared to full (unsupervised) ImageNet training for GAN based self-supervision (e-i). For the other methods (i-n, o-s) we reach and also surpass the perfor- mance for the first layer and are within 1.5% points for the second. Given that the best unsupervised performance for conv2 is 32.5, our method using a single source Image A (Table 2, p) is remarkably close with 31.5.
Image contents. While we surpass the GAN based approach of (Donahue et al., 2017) for both single source images, we find more nuanced results for the other two methods: For RotNet, as ex- pected, the photographic bias cannot be extracted from a single image. Thus its performance is low with little training data and increases together with the number of images (Table 2, j-n). When comparing Image A and B trained networks for RotNet, we find that the photograph yields better performance than the hand drawn animal image. This indicates that the method can extract rotation
7

---

Published as a conference paper at ICLR 2020
Method, Image
Method, Image B
BiGAN
RotNet
DeepCluster
BiGAN
RotNet
DeepCluster
20.4
19.9
20.7
20.5
17.8
19.7
Figure 2: conv1 filters trained using a single image. The 96 learned (3 x x1x11) filters for the first layer of AlexNet are shown for each single training image and method along with their linear classifier performance. For visualization, each filter is normalized to be in the range of (-1,
information from low level image features such patches which is at first counter intuitive. Con- sidering that the hand-drawn image does not work well, we can assume that lighting and shadows even in small patches can indeed give important cues on the up direction which can be learned even from a single (real) image. DeepCluster shows poor performance convl which we can improve upon in the single image setting (Table 2, o-r). Naturally, the image content matters: trivial image without any image gradient (e.g. picture of white wall) would not provide enough signal for any method. To better understand this issue, also train DeepCluster on the much less cluttered Im- age C to analyze how much the image influences our claims. We find that even though this image contains large parts of sky and sea, the performance is only slightly lower than that of Image A. This finding indicates that the augmentations can even compensate for large untextured areas and the exact choice of image is not critical.
More than one image. While BiGAN fails converge for N E {10, 1000), most likely due to is- sues in learning from distribution which is neither whole images nor only patches, we find that both RotNet and DeepCluster improve their performance in deeper layers when increasing the number of training images. However, for convl and conv2, single image is enough. In deeper layers, DeepCluster seems to require large amounts of source images to yield the reported results as the deka- and kilo- variants start improving over the single image case (Table 2, o-t). This need for data also explains the gap between the two input images which have different resolutions. Summarizing Table 2, conclude that learning convl, conv2 and for the most part conv3 (33.4 vs. 39.4) on over 1M images does not yield a significant performance increase over using one single training image highly unexpected result.
Generalization. In Table 3, we show the results of training linear classifiers for the CIFAR- 10 dataset and compare against various baselines. We find that the GAN trained on the smaller Image outperforms all other methods including the sully-sueervised trained one for the first convolutional layer. We also outperform the same architecture trained the full CIFAR-10 training set using RotNet, which might be due to the fact that either CIFAR images do not contain much information about the orientation of the picture because they do not contain as many objects in ImageNet. While the GAN trained on the whole dataset outperforms the MonoGAN on the deeper layers, the gap stays very small until the last layer. These findings CIFAR- 100 dataset shown in Table 3. We find that our method obtains the best performance for the first layers, even against the fully supervised version. The gap between our mono variant and the other methods increases again with deeper layers, hinting to the fact that we cannot learn very high level concepts in deeper layers from just one single image. These results corroborate the finding that our method allows learning very generalizable early features that are not domain dependent.
also reflected in the experiments the
4.4 QUALITATIVE ANALYSIS
Visual comparison weights. In Figure 2, we compare the learned filters of all first-layer con- volutions of AlexNet trained with the different methods and single image. First, find that the filters closely resemble those obtained via supervised training: Gibor-like edge detectors and various color blobs. Second, we find that the look is not easily predictive of its performance, e.g.
8

---

Published as a conference paper at ICLR 2020
while generatively learned filters (BiGAN) show many edge detectors, its linear probes performance about the same as that of DeepCluster which seems to learn many somewhat redundant point fea- tures. However, also find that some edge detectors required, as confirm from RotNet and DeepCluster trained on Image B, which yield less crisp filters and worse performances.
Table 4: Finetuning experiments The pre- trained model's first two convolutions are left frozen (or replaced by the Scattering trans- form) and the nework is retrained using Ima- geNet LSVRC-12 training set.
Our Mand Fostuts
Superssdd Imagahat Foolares
Content
Top-1
Full sup.
59.4
Random
42.6
Scattering
49.2
Figure 3: Style transfer with single-image pre- training. We show two style transfer results us- ing the Image trained BiGAN and the ImageNet pretrained AlexNet.
BiGAN, A
51.4
RotNet,
49.5
DeepCluster A
52.5
Fine-tuning instead of freezing. In Tab. 4, we show the results of retraining a network with the first two convolutional filters, the scattering transform from (Oyallon et al., 2017), left frozen. We observe that our single image trained DeepCluster and BiGAN models achieve performances closes to the supervised benchmark. Notably, the scattering transform performs slightly worse than the analyzed single image methods. We also show in the appendix the results of retraining network initialized with the first two convolutional layers obtained from a single image and subsequently linearly probing the model. The results are shown in Appendix Tab. and we find that can recover the performance of fully-supervised networks, i.e. the first two convolutional filters trained from just single image generalize well and do not get stuck in an image specific minimum.
replacement for conv 1-2
Neural style transfer. Lastly, we show how our features trained on only single image can be used for other applications. In Figure 3 we show two basic style transfers using the method of (Gatys et al., 2016) from official PyTorch tutorial". Image content and style are separated and the style is transferred from the source to target image using all CNN features, not just the shallow layers. We visually compare the results of using our features and from full ImageNet supervision. We find almost no visual differences in the stylized images and can conclude that our early features equally powerful as fully supervised ones for this task.
CONCLUSIONS
We have made the surprising observation that we can learn good and generalizable features through self-supervision from one single source image, provided that sufficient data augmentation is used. Our results complement recent works (Mahajan et al., 2018: 8; Goyal et al., 2019) that have investigated self-supervision in the very large data regime. Our main conclusion is that these methods succeed perfectly in capturing the simplest image statistics, but that for deeper layers a gap exist with strong supervision which is compensated only in limited manner by using large datasets. This novel finding motivates a renewed focus on the role of augmentations in self-supervised learning and critical rethinking of how better leverage the available data.
ACKNOWLEDGEMENTS.
We thank Aravindh Mahendran for fruitful discussions. Yuki Asano gratefully acknowledges sup- port from the EPSRC Centre for Doctoral Training in Autonomous Intelligent Machines & Systems (EP//101599//). The work is supported by ERC IDIU-638009.
http//vvvnee//eeeeeee http:///o...rrteaaaaeee/eeeaaaaa euutorraat html

---

Published as a conference paper at ICLR 2020
REFERENCES
Pulkit Agrawal, Joao Carreira, and Jitendra Malik, Learning to see by moving. In Proc. ICCV, pp- 37-45. IEEE, 2015. 3
R. Arandjelović and A. Zisserman. Look, listen and learn. In Proc. ICCV, 2017. 3
Piotr Bojanowski and Armand Joulin. Unsupervised learning by predicting noise. In Proc. ICMI.pp.511. 517-526.. PMLR, 2017. 2
Joan Bruna and Stéphane Mallat. Invariant seattering convolution networks, IEEE transactions on pattern analysis and machine intelligence, 35(8):1872 1888988888888 2013. 3.7
M. Caron, P. Bojanowski, A. Joulin, and M. Douze. Deep clustering for unsupervised learning of visual features. In Proc. ECCV, 2018. 2, 6, 7,
N. Dalal and B Triggs. Histogram of Oriented Gradients for Human Detection. In Proc. CVPR, volume 2, pp. 886-893, 2005. 2,3
Virginia de Sa. Learning classification with unlabeled data. In NIPS, pp. 112-119, 1994.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image
database. Proc. CVPR, 2009.
Carl Doersch, Abhinav Gupta, and Alexei Efros. Unsupervised visual representation learning by context prediction. Proc. ICCV, pp. 1422-1430, 2015.
Jeff Donahue, Philipp Krhenbhl, and Trevor Darrell. Adversarial feature learning. Proc. ICLR, 2017. 3, 5, 6,
Dosovitskiy, P. Fischer, T. Springenberg. Riedmiller, Brox. Discriminative unsupervised feature learning with exemplar convolutional neural networks. IEEE PAMI, 38(9):133-11477, Sept 2016. ISSN 0162-8828, doi: 10. 1199/TAA...0015229911144. 2
V. Dumoulin, I. Belghazi, B. Poole, O. Mastropictro, A. Lamb, M. Arjovsky, and A. Courville. Adversarially learned inference. arXiv preprint 1606::/606.00004, 2016. 5
D. Erhan, Y. Bengio, A. Courville, and P. Vincent. Visualizing higher-layer features of a deep network. Tech- nical Report 1341, University of Montreal, Jun 2009. 13
Chuang Gan, Boqing Gong, Kun Liu, Hao Su, and Leonidas J Guibas. Geometry guided convolutional neural networks for self-supervised video representation learning. In Proc. CVPR, 2018. 3
Rouhan Gao, Dinesh Jayaraman, and Kristen Grauman. Object-centric representation learning from unlabeled videos. In Proc. ACCV, 2016.
Leon Gatys, Alexander Eoker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proc. CVPR, pp. 2414-2423, 2016.
Geirhos, Rubisch, C. Michaelis, M. Bethge. F. Wichmann, and Brendel. Imagenet CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. International Conference on Learning Representations, 2019. 14
Spyros Gidaris, Praveen Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In Proc. ICLR, 2018. 3,
R. B. Girshick. Fast R-CNN. In Proc. ICCV, 2015.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, pp. 2672-2680, 2014. 5
Priya Goyal, Dhruv Mahajan, Abhinay Gupta, and Ishan Misra. Scaling and henchmarking self-supervised visual representation learning. arXiv preprint arxi: hat::/905.00 /905.0/255 2019.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved train- ing of wasserstein gans. In NIPS, pp. 5767-5777, 2017. 7
B. Hariharan, J. Malik, and D. Ramanan. Discriminative decorrelation for clustering and classification. In Proc. ECCV, 2012. 3
10

---

Published as a conference paper at ICLR 2020
M. Huh, P. Agrawal, and A. A. Efros. What makes imagenet good for transfer learning? arXiv preprint a60iv 1608.0861, 2016.
Phillip Isola, Daniel Zoran, Dilip Krishnan, and Edward H Adelson. Learning visual groups from occurrences space and time. Proc. ICLR, 2015.
Dinesh Jayaraman and Kristen Grauman. Learning image representations tied ego-motion. In Proc. ICCV, 2015. 3
Dinesh Jayaraman and Kristen Grauman. Slow and steady feature analysis: higher order temporal coherence in video. In Proc. CVPR, 2016. 3
Simon Jenni and Paolo Favaro. Self-supervised feature learning by learning to spot artifacts. In Proc. CVPR, 2018. 2
J. Johnson, A. Alahi, and L. Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In Proc. ECCV, pp. 694-711, 2016.
R. Kat, R. Jevnisek, and S. Avidan. Matching pixels using co-occurrence statistics. In Proc. ICCV, 1751- 1759, 2018. 2
Philipp Krähenbühl, Carl Doersch, Jeff Donahue, and Trevor Darrell. Data-dependent initializations of convo- lutional neural networks, Proc. ICLR, 2016.
A. Krizhevsky, I. Sutskever, and G. E. Hinton. mlaeesftt with deep convolutional neural networks. In NIPS. pp. 1106-1114, 2012.
Alex Krizhevsky, Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009. 5
Gustav Larsson, Michael Maire, Gregory Shakhnarovich. Colorization as a proxy task visual under- standing. In Proc. CVPR, 2017.
Hsin- Hsin-Ying Lee, Bin-Bin Huang, Mancesh Kumar Singh, and Ming-Hsuan Yang. Unsupervised representation learning by sorting sequence. In Proc. ICCV, 2017. 2017. 2
D. Lowe. Distinctive image features from scale-invariant keypoints. IJC,, 60(2):91-100, 2004. 2,
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. In Pro- ceedings of the European Conference on Computer Vision (ECCV), pp. 181-196, 2018.
Mahendran, Thewlis, and Vedaldi. Cross pixel optical-flow similarity for self-supervised learning. In Proc. ACCV, 2018.
Malisiewicz, Gupta, A. Efros. Ensemble exemplar-SV SVMs for object detection beyond. In Proc. ICCV, 2011. 3
Ishan Misra, Lawrence Zitnick, and Martial Hebert. Shuffle and learn: Unsupervised learning using temporal order verification. Proc. ECCV, 2016.
T Mundhenk, Daniel Ho, and Barry Chen. Improvements context based self-supervised learning. In Proc. CVPR, 2017.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In Proc. ECCV, pp. 69-84. Springer, 2016.
Mehdi Noroozi, Hamed Pirsiavash, and Paolo Favaro. Representation learning by learning count. In Proc. ICCV, 2017. 2
Mehdi Noroozi, Ananth Vinjimoor, Paolo Favaro, and Hamed Pirsiavash. Boosting self supervised learning via knowledge transfer. Proc. CVPR, 2018.
Chris Olah, Arvind Satyanarayan, lan Johnson, Shan Carter, Ludwig Schubert, Katherine Ye, and Alexander Mordvintsev. The building blocks of interpretabilityy Distill, 3(3),e00, 2018. 13
B. A. Olshausen and D. J. Field. Sparse coding with overcomplete basis set: A strategy employed by V1? Vision Research, 37(23)33311-333 1997. 2,
M. Oquah, I.. Botton, I. Laptev, and J. Sivic. Learning and Transferring Mid-Level Image Representations using Convolutional Neural Networks. In Proc. CVPR, 2014.

---

Published as a conference paper at ICLR 2020
Andrew Owens, Phillip Isola, Josh H. McDermott, Antonio Torralba, Edward H. Adelson, and William T. Freeman. Visually indicated sounds. Proc. CVPR, pp. 2405 2405-2413, 2016.
Edouard Oyallon, Eugene Belilovsky, and Sergey Zagoruyko. Scaling the scattering transform: Deep hybrid networks. pp. 5618-5627, 2017.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. Proc. CVPR. pp. 2536-2544, 2016.
Deepak Pathak, Ross Girshick, Piotr Dollár, Trevor Darrell, and Bharath Hariharan. Learning features by watching objects move. In Proc. CVPR, 2017. 3
Zhongzheng Ren and Yong Jae Lee. Cross-domain self-supervised multi-task feature learning using synthetic imagery. In Proc. CVPR. 2018.
A. Rodriguez, Naresh Boddeti, BVK V. Kumar, and Mahalanobis. Maximum margin correlation filter: A new approach localization and classifieation. IEEE Transactions on Image Processing. 22(2):631-64 2013.
Tamar Rott Shaham, Tali Dekel, and Tomer Michaeli. Singan: Learning generative model from a single natural image. Computer Vision (ICCV), IEEE International Conference on, 2019. 3
Pierre Sermanet et al. Time-contrastive networks: Self-supervised learning from video. In Proc. Iull Conf. on Robotics and Automation, 2018. 2
Н. Shin, H. R. Roth, Gao, Lu, Xu, I. Nogues, Yao, D. Mollura, and R. Summers. Deep convolutional neural networks for computer-aided detection: Can architectures, dataset characteristics and transfer learning, IEEE Trans. on Medical Imaging, 35(5): (215):1285-122) 1298, May 2016. ISSN 0278-0065
N. Srivastava, E. Mansimov, and R. Salakhudinov. Unsupervised learning of video representations using Istms.
In Proc. ICML, 2015. 3
I. Talmi, R. Mechrez, and L. Zelnik-Manor. Template matching with deformable diversity similarity. In Proc.
CVPR, pp. 175 183, 2017. 2
D. Ulyanov, A. Vedaldi, and V. Lempitsky. Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis. In Proc. CVPR, 2017. 7
D. Ulyanov, A. Vedaldi, and V. Lempitsky. Deep image prior. In Proc. CVPR, 2018. 5
X. Wang and A. Gupta. Unsupervised learning visual representations using videos. In Proc. ICCV, pp. 2794-2802, 2015.
Xiaolong Wang, Kaiming Hc, and Abhinav Gupta. Transitive invariance for self-supervised visual representa- tion learning. In Proc. ICCV, 2017.
D. Wei, Lim, A. Zisserman, and W. T. Freeman. Learning and using the arrow of time. Proc. CVPR, 2018.
Yosinski, Clune, Bengio, and H. Lipson. How transferable are features deep neural networks? In NIPS, pp. 3320-3328, 2014.
M. D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Proc. ECCV, 2014.
13
Richard Zhang, Phillip Isola, and Alexei Efros. Colorful image colorization. In Proc. ECCV, pp. 649-66996 666. Springer, 2016. 2
Richard Zhang, Phillip Isola, and Alexei A. Efros. Split-brain autoencoders: Unsupervised learning by cross- channel prediction. In Proc. CVPR, 2017. 5, 6, 13
12

---

Published as a conference paper at ICLR 2020
A APPENDIX
A.1
IMAGENET TRAINING IMAGES
NUR
Figure 4: ImageNet images for the N = 10 experiments. The images used for the = 10 experiments are shown in fig. 4.
A.2
VISUAL COMPARISON OF FILTERS
Activation Maximization
Maximal Feature Activation
convi
conv2
conv3
conv5 conv4
Figure 5: Filter visualization. We show activation maximization (left) and retrieval of top ac- tivated images from the training set of ImageNet (right) for four random non-cherrypicked target filters. From top bottom: convl-5 of the BiGAN trained single image A. The filter visual- ization is obtained by learning a (regularized) input image that maximizes the response to the target filter using the library Lucid (Olah et al., 2018).
In order to understand what deeper neurons are responding to in our model, we visualize random neurons via activation maximization (Erhan et al., 2009; Zeiler & Fergus, 2014) in each layer. Ad- ditionally, we retrieve the top-9 images in the ImageNet training set that activate each neuron most in Figure Since the mono networks are not trained on the ImageNet dataset, it can used here for visualization. From the first convolutional layer we find typical neurons strongly reacting to oriented edges. In layers 2-4 we find patterns such as grids (conv2:3), and textures such as leop- ard skin (conv2:2) and round grid cover (conv4:4). Confirming our hypothesis that the neural network is only extracting patterns and not semantic information, we do not find any neurons partic- ularly specialized to certain objects even in higher levels for example dog faces similar which can be fund in supervised networks. This finding aligns with the observations of other unsuper- vised methods (Caron al., 2018; Zhang al., 2017). As most neurons extract simple patterns and
13

---

Published as a conference paper at ICLR 2020
Table Finetuning experiments Models are initialized using convl and conv2 from various sin- gle image trained models and the whole network is fine-tuned using ImageNet LSVRC-12 training set. Accuracy is averaged over 10 crops.
c1
c2
c3
c4
c5
Full sup.
19.3 36.3
44.2
48.3
50.5
BiGAN, A RotNet,
22.5 37.6
44.2
47.6
48.3
22.0 38.2
44.8
49.2
51.8
DeepCluster. A
21.8 35.9
43.6
48.8
50.4
textures, the surprising effectiveness of training a network using a single image can be explained by the recent finding that even CNNs trained on ImageNet rely on texture (as opposed shape) information to classify (Geirhos et al., 2019).
A.3 RETRAINING FROM SINGLE IMAGE INITIALIZATION
In Table 5, initialize AlexNet models using the first two convolutional filters learned from single image and retrain them using ImageNet. We find that the networks recover their performance fully and the first filters do make the network stuck in bad local minimum despite having been trained on a single image from a different distribution. The difference from the BiGAN to the full supervision model likely due to using a smaller input resolution (112 instead of 224), as the BiGAN's output resolution is limited.
A.4 LINEAR PROBES ON IMAGENET
We show plots of the ImageNet linear probes results (Table 2 of the paper) in fig. 6. On the left we plot performance per layer in absolute scale. Naturally the performance of the supervised model improves with depth, while all unsupervised models degrade after conv3. From the relative plot on the right, it becomes clear that with our training scheme, we can even slightly surpass supervised performance on convl presumably since our model trained with sometimes very small patches, thus receiving an emphasis on learning good low level filters. The gap between all self reff-supervised methods and the supervised baseline increases with depth, due to the fact that the supervised model trained for this specific task, whereas the celf-super models learn from surrogate task without labels.
Linear Classifier on ImageNet
Supervised
Random
RotNet
1-RotNet
absolute performance
BIGAN
1-BiGAN
DeepCluster
DeepCluster
10
conv1
ConV2
CODV3
convt
oo195
Figure 6: Linear Classifiers on ImageNet. Classification accuracies of linear classifiers trained on the representations from Table 2 are shown in absolute scale.
A.5 EXAMPLE AUGMENTED TRAINING DATA
In figs. to 10 we show example patches generated by our augmentation strategy for the datasets with different N. Even though the images and patches are very different in color and shape distribu-
14

---

Published as a conference paper at ICLR 2020
tion, our model learns weights that perform similarly in the linear probes benchmark (see Table 2 in the paper).
Figure 7: Example crops of Image A (N dataset.
15

---

Published as a conference paper at ICLR 2020
Figure 8: Example crops of Image (N 1) dataset. samples were selected randomly.
STON
AMAUR
ST200
Figure 9: Example crops of deka (N 10) dataset. 50 samples selected randomly.
NN
Figure 10: Example crops of kilo (N 1000) dataset. 50 samples were selected randomly.
16


Reviews:
[{"review_id": "r1x7498kcS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper explores self-supervised learning in the low-data regime, comparing results to self-supervised learning on larger datasets.  BiGAN, RotNet, and DeepCluster serve as the reference self-supervised methods.  It argues that early layers of a convolutional neural network can be effectively learned from a single source image, with data augmentation.  A performance gap exists for deeper layers, suggesting that larger datasets are required for self-supervised learning of useful filters in deeper network layers.\n\nI believe the primary claim of this paper is neither surprising nor novel.  The long history of successful hand-designed descriptors in computer vision, such as SIFT [Lowe, 1999] and HOG [Dalal and Triggs, 2005], suggest that one can design (with no data at all) features reminiscent of those learned in the first couple layers of a convolutional neural network (local image gradients, followed by characterization of those gradients over larger local windows).\n\nMore importantly, it is already well established that it is possible to learn, from only a few images, filter sets that resemble the early layers of filters learned by CNNs.  This paper fails to account for a vast amount of literature on modeling natural images that predates the post-AlexNet deep-learning era.\n\nFor example, see the following paper (over 5600 citations according to Google scholar):\n\n[1] Bruno A. Olshausen and David J. Field.  Emergence of simple-cell receptive field properties by learning a sparse code for natural images.  Nature, 1996.\n\nFigure 4 of [1] shows results for learning 16x16 filters using \"ten 512x512 images of natural scenes\".  Compare to the conv1 filters in Figure 2 of the paper under review.  This 1996 paper clearly established that it is possible to learn such filters from a small number of images.  There is long history of sparse coding and dictionary learning techniques, including multilayer representations, that follows from the early work of [1].  The paper should at minimum engage with this extensive history, and, in light of it, explain whether its claims are actually novel.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Ske6OaZ-qr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update 11/21\nWith the additional experiments (testing a new image, testing fine-tuning of hand-crafted features), additions to related work, and clarifications, I am happy to raise my score to accept. Overall, I think this paper is a nice sanity check on recent self-supervision methods. In the future, I am quite curious about how these mono-image learned features would fare on more complex downstream tasks (e.g., segmentation, keypoint detection) which necessarily rely less on texture.\n\nSummary\nThis paper seeks to understand the role of the *number of training examples* in self-supervised learning with images. The usefulness of the learned features is evaluated with linear probes at each layer for either ImageNet or CiFAR image classification. Empirically, they find that a single image along with heavy data augmentation suffices for learning the first 2-3 layers of convolutional weights, while later layers improve with more self-supervised training images. The result holds for three state-of-the-art self-supervised methods, tested with two single-image training examples.\n\nIn my view, learning without labels is an important problem, and it is interesting what can be learned from a single image and simple data augmentation strategies. \n\nComments / Questions\nIt seems to me that for completeness, Table 4 should include the result of training a supervised network on top of random conv1/2 and Scattering network features, because this experiment is actually testing what we want - performance of the features when fine-tuned for a downstream task. So for example, even if a linear classifier on top of Scattering features does poorly, if downstream fine-tuning results in the same performance as another pre-training method, then Scattering is a perfectly fine approach for initial features. Could the authors please either correct this logic or provide the experiments?\nFurther, it seems that the results in Table 4 might be a bit obscured by the size of the downstream task dataset. I wonder if the learned features require fewer fully supervised images to obtain the same performance on the downstream task?\nCan the authors clarify how the neural style transfer experiment is performed? The method from Gatys et al. requires features from different layers of the feature hierarchy, including deeper layers. Are all these features taken directly from the self-supervised network or is it fine-tuned in some way?\nWhile I appreciate the computational burden of testing more images, it does feel that Image A and B are quite cherry-picked in being very visually diverse. Because of this, it seems like a precise answer to what makes a good single training image remains unknown. I wonder how feasible it is to find a proxy metric that corresponds to the performance on downstream tasks which is expensive to compute. It might be interesting to try to generate synthetic images (or modify real ones) that are good for this purpose and observe their properties.\nI disagree with the claim of practicality in the introduction (page 2, top). While training on one image does reduce the burden of number of images, the computational burden remains the same. And as mentioned above, it doesn’t seem likely that *any* image would work for this method. Finally, more images are needed to learn the deeper layers for the downstream task anyway. \n\nThe paper is well-written and clear. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BkgPU07CFS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper studies self-supervised learning from very few unlabeled images, down to the extreme case where only a single image is used for training. From the few/single image(s) available for training, a data set of the same size as some unmodified reference data set (ImageNet, Cifar-10/100) is generated through heavy data augmentation (cropping, scaling, rotation, contrast changes, adding noise). Three popular self-supervised learning algorithms are then trained on this data sets, namely (Bi)GAN, RotNet, and DeepCluster, and the linear probing accuracy on different blocks is compared to that obtained by training the same methods on the reference data sets. The linear probing accuracy from the first few conv layers of the network trained on the single/few image data set is found to be comparable to or better than that of the same model trained on the full reference data set.\n\nI enjoyed the paper; it addresses the interesting setting of an extremely small data set which complements the large number of studies on scaling up self-supervised learning algorithms. I think it is not extremely surprising that using the proposed strategy allows to learn low level features as captured by the first few layers, but I think it is worth studying and quantifying. The experiments are carefully described and presented, and the paper is well-written.\n\nHere are a few questions and concerns:\n\n- How much does the image matter for the single-image data set? The selected images A and B are of very high entropy and show a lot of different objects (image A) and animals (image B). How do the results change if e.g. a landscape image or an abstract architecture photo is used?\n\n- How general is the proposed approach? How likely is it to generalize to other approaches such as Jigsaw (Doersch et al., 2015) and Exemplar (Dosovitskiy et al., 2016)? It would be good to comment on this.\n\n- [1] found that the network architecture for self-supervised learning can matter a lot, and that by using a ResNet architecture, performance of SSL methods can be significantly improved. In particular, the linear probing accuracy appears to be often monotonic as a function of the depth of the layer it is computed from. This is in contrast to what is observed for AlexNet in Tables 2 and 3, where the conv5 accuracy is lower than the conv4. It would therefore be instructive to add experiments for ResNet to see how well the results generalize to other network architectures.\n\n- Does the MonoGAN exhibit stable training dynamics comparable to training WGAN on CIFAR-10, or do the training dynamics change on the single-image data set?\n\n\nOverall, I’m leaning towards accepting the paper, but it would be important to see how well the experiments generalize to i) ResNet and ii) other (lower entropy) input images.\n\n[1] Kolesnikov, A., Zhai, X. and Beyer, L., 2019. Revisiting self-supervised visual representation learning. arXiv preprint arXiv:1901.09005.\n\n\n\n---\nUpdate after rebuttal:\nI thank the authors for their detailed response. I appreciate the efforts of the authors into investigating the issues raised, the described experiments sound promising. Unfortunately, the new results are not presented in the revision. I will therefore keep my rating.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the effectiveness of self-supervised approaches by characterising how much information they can extract from a given dataset of images on a per-layer basis. Based on an empirical evaluation of RotNet, BiGAN, and DeepCluster, the authors argue that the early layers of CNNs can be effectively learned from a single image coupled with strong data augmentation. Secondly, the authors also provide some empirical evidence that supervision might still necessary to learn the deeper layers (even in the presence of millions of images for self-supervision). 
Overall, the reviews agree that the paper is well written and timely given the growing popularity of self-supervised methods. Given that most of the issues raised by the reviewers were adequately addressed in the rebuttal, I will recommend acceptance. We ask the authors to include additional experiments requested by the reviewers (they are valuable even if the conclusions are not perfectly aligned with the main message).

Decision: Accept (Poster)