A critical analysis of self-supervision, or what we can learn from a single image

1 INTRODUCTION . Despite tremendous progress in supervised learning , learning without external supervision remains difficult . Self-supervision has recently emerged as one of the most promising approaches to address this limitation . Self-supervision builds on the fact that convolutional neural networks ( CNNs ) transfer well between tasks ( Shin et al. , 2016 ; Oquab et al. , 2014 ; Girshick , 2015 ; Huh et al. , 2016 ) . The idea then is to pre-train networks via pretext tasks that do not require expensive manual annotations and can be automatically generated from the data itself . Once pre-trained , networks can be applied to a target task by using only a modest amount of labelled data . Early successes in self-supervision have encouraged authors to develop a large variety of pretext tasks , from colorization to rotation estimation and image autoencoding . Recent papers have shown performance competitive with supervised learning by learning complex neural networks on very large image datasets . Nevertheless , for a given model complexity , pre-training by using an off-theshelf annotated image datasets such as ImageNet remains much more efficient . In this paper , we aim to investigate the effectiveness of current self-supervised approaches by characterizing how much information they can extract from a given dataset of images . Since deep networks learn a hierarchy of representations , we further break down this investigation on a per-layer basis . We are motivated by the fact that the first few layers of most networks extract low-level information ( Yosinski et al. , 2014 ) , and thus learning them may not require the high-level semantic information captured by manual labels . Concretely , in this paper we answer the following simple question : “ is self-supervision able to exploit the information contained in a large number of images in order to learn different parts of a neural network ? ” We contribute two key findings . First , we show that as little as a single image is sufficient , when combined with self-supervision and data augmentation , to learn the first few layers of standard deep networks as well as using millions of images and full supervision ( Figure 1 ) . Hence , while selfsupervised learning works well for these layers , this may be due more to the limited complexity of such features than the strength of the supervisory technique . This also confirms the intuition that early layers in a convolutional network amounts to low-level feature extractors , analogous to early conv1 conv2 conv3 conv4 conv5 0 20 40 60 80 100 % su pe rv ise d pe rfo rm an ce Linear Classifier on ImageNet Random RotNet 1-RotNet BiGAN 1-BiGAN DeepCluster 1-DeepCluster Figure 1 : Single-image self-supervision . We show that several self-supervision methods can be used to train the first few layers of a deep neural networks using a single training image , such as this Image A , B or even C ( above ) , provided that sufficient data augmentation is used . learned and hand-crafted features for visual recognition ( Olshausen & Field , 1997 ; Lowe , 2004 ; Dalal & Triggs , 2005 ) . Finally , it demonstrates the importance of image transformations in learning such low-level features as opposed to image diversity.1 Our second finding is about the deeper layers of the network . For these , self-supervision remains inferior to strong supervision even if millions of images are used for training . Our finding is that this is unlikely to change with the addition of more data . In particular , we show that training these layers with self-supervision and a single image already achieves as much as two thirds of the performance that can be achieved by using a million different images . We show that these conclusions hold true for three different self-supervised methods , BiGAN ( Donahue et al. , 2017 ) , RotNet ( Gidaris et al. , 2018 ) and DeepCluster ( Caron et al. , 2018 ) , which are representative of the spectrum of techniques that are currently popular . We find that performance as a function of the amount of data is dependent on the method , but all three methods can indeed leverage a single image to learn the first few layers of a deep network almost “ perfectly ” . Overall , while our results do not improve self-supervision per-se , they help to characterize the limitations of current methods and to better focus on the important open challenges . 2 RELATED WORK . Our paper relates to three broad areas of research : ( a ) self-supervised/unsupervised learning , ( b ) learning from a single sample , and ( c ) designing/learning low-level feature extractors . We discuss closely related work for each . Self-supervised learning : A wide variety of proxy tasks , requiring no manual annotations , have been proposed for the self-training of deep convolutional neural networks . These methods use various cues and tasks namely , in-painting ( Pathak et al. , 2016 ) , patch context and jigsaw puzzles ( Doersch et al. , 2015 ; Noroozi & Favaro , 2016 ; Noroozi et al. , 2018 ; Mundhenk et al. , 2017 ) , clustering ( Caron et al. , 2018 ) , noise-as-targets ( Bojanowski & Joulin , 2017 ) , colorization ( Zhang et al. , 2016 ; Larsson et al. , 2017 ) , generation ( Jenni & Favaro , 2018 ; Ren & Lee , 2018 ; Donahue et al. , 2017 ) , geometry ( Dosovitskiy et al. , 2016 ; Gidaris et al. , 2018 ) and counting ( Noroozi et al. , 2017 ) . The idea is that the pretext task can be constructed automatically and easily on images alone . Thus , methods often modify information in the images and require the network to recover them . Inpainting or colorization techniques fall in this category . However these methods have the downside that the features are learned on modified images which potentially harms the generalization to unmodified ones . For example , colorization uses a gray scale image as input , thus the network can not learn to extract color information , which can be important for other tasks . Slightly less related are methods that use additional information to learn features . Here , often temporal information is used in the form of videos . Typical pretext tasks are based on temporalcontext ( Misra et al. , 2016 ; Wei et al. , 2018 ; Lee et al. , 2017 ; Sermanet et al. , 2018 ) , spatio-temporal 1Example applications that only rely on low-level feature extractors include template matching ( Kat et al. , 2018 ; Talmi et al. , 2017 ) and style transfer ( Gatys et al. , 2016 ; Johnson et al. , 2016 ) , which currently rely on pre-training with millions of images . cues ( Isola et al. , 2015 ; Gao et al. , 2016 ; Wang et al. , 2017 ) , foreground-background segmentation via video segmentation ( Pathak et al. , 2017 ) , optical-flow ( Gan et al. , 2018 ; Mahendran et al. , 2018 ) , future-frame synthesis ( Srivastava et al. , 2015 ) , audio prediction from video ( de Sa , 1994 ; Owens et al. , 2016 ) , audio-video alignment ( Arandjelović & Zisserman , 2017 ) , ego-motion estimation ( Jayaraman & Grauman , 2015 ) , slow feature analysis with higher order temporal coherence ( Jayaraman & Grauman , 2016 ) , transformation between frames ( Agrawal et al. , 2015 ) and patch tracking in videos ( Wang & Gupta , 2015 ) . Since we are interested in learning features from as little data as one image , we can not make use of methods that rely on video input . Our contribution inspects three unsupervised feature learning methods that use very different means of extracting information from the data : BiGAN ( Donahue et al. , 2017 ) utilizes a generative adversarial task , RotNet ( Gidaris et al. , 2018 ) exploits the photographic bias in the dataset and DeepCluster ( Caron et al. , 2018 ) learns stable feature representations under a number of image transformations by proxy labels obtained from clustering . These are described in more detail in the Methods section . Learning from a single sample : In some applications of computer vision , the bold idea of learning from a single sample comes out of necessity . For general object tracking , methods such as max margin correlation filters ( Rodriguez et al. , 2013 ) learn robust tracking templates from a single sample of the patch . A single image can also be used to learn and interpolate multi-scale textures with a GAN framework ( Rott Shaham et al. , 2019 ) . Single sample learning was pursued by the semi-parametric exemplar SVM model ( Malisiewicz et al. , 2011 ) . They learn one SVM per positive sample separating it from all negative patches mined from the background . While only one sample is used for the positive set , the negative set consists of thousands of images and is a necessary component of their method . The negative space was approximated by a multi-dimensional Gaussian by the Exemplar LDA ( Hariharan et al. , 2012 ) . These SVMs , one per positive sample , are pooled together using a max aggregation . We differ from both of these approaches in that we do not use a large collection of negative images to train our model . Instead we restrict ourselves to a single or a few images with a systematic augmentation strategy . Classical learned and hand-crafted low-level feature extractors : Learning and hand-crafting features pre-dates modern deep learning approaches and self-supervision techniques . For example the classical work of ( Olshausen & Field , 1997 ) shows that edge-like filters can be learned via sparse coding of just 10 natural scene images . SIFT ( Lowe , 2004 ) and HOG ( Dalal & Triggs , 2005 ) have been used extensively before the advent of convolutional neural networks and , in many ways , they resemble the first layers of these networks . The scatter transform of Bruna & Mallat ( 2013 ) ; Oyallon et al . ( 2017 ) is an handcrafted design that aims at replacing at least the first few layers of a deep network . While these results show that effective low-level features can be handcrafted , this is insufficient to clarify the power and limitation of self-supervision in deep networks . For instance , it is not obvious whether deep networks can learn better low level features than these , how many images may be required to learn them , and how effective self-supervision may be in doing so . For instance , as we also show in the experiments , replacing low-level layers in a convolutional networks with handcrafted features such as Oyallon et al . ( 2017 ) may still decrease the overall performance of the model . Furthermore , this says little about deeper layers , which we also investigate . In this work we show that current deep learning methods learn slightly better low-level representations than hand crafted features such as the scattering transform . Additionally , these representations can be learned from one single image with augmentations and without supervision . The results show how current self-supervised learning approaches that use one million images yield only relatively small gains when compared to what can be achieved from one image and augmentations , and motivates a renewed focus on augmentations and incorporating prior knowledge into feature extractors . 3 METHODS . We discuss first our data and data augmentation strategy ( section 3.1 ) and then we summarize the three different methods for unsupervised feature learning used in the experiments ( section 3.2 ) .

Summary: The paper studies self-supervised learning from very few unlabeled images, down to the extreme case where only a single image is used for training. From the few/single image(s) available for training, a data set of the same size as some unmodified reference data set (ImageNet, Cifar-10/100) is generated through heavy data augmentation (cropping, scaling, rotation, contrast changes, adding noise). Three popular self-supervised learning algorithms are then trained on this data sets, namely (Bi)GAN, RotNet, and DeepCluster, and the linear probing accuracy on different blocks is compared to that obtained by training the same methods on the reference data sets. The linear probing accuracy from the first few conv layers of the network trained on the single/few image data set is found to be comparable to or better than that of the same model trained on the full reference data set.