A critical analysis of self-supervision, or what we can learn from a single image

1 INTRODUCTION . Despite tremendous progress in supervised learning , learning without external supervision remains difficult . Self-supervision has recently emerged as one of the most promising approaches to address this limitation . Self-supervision builds on the fact that convolutional neural networks ( CNNs ) transfer well between tasks ( Shin et al. , 2016 ; Oquab et al. , 2014 ; Girshick , 2015 ; Huh et al. , 2016 ) . The idea then is to pre-train networks via pretext tasks that do not require expensive manual annotations and can be automatically generated from the data itself . Once pre-trained , networks can be applied to a target task by using only a modest amount of labelled data . Early successes in self-supervision have encouraged authors to develop a large variety of pretext tasks , from colorization to rotation estimation and image autoencoding . Recent papers have shown performance competitive with supervised learning by learning complex neural networks on very large image datasets . Nevertheless , for a given model complexity , pre-training by using an off-theshelf annotated image datasets such as ImageNet remains much more efficient . In this paper , we aim to investigate the effectiveness of current self-supervised approaches by characterizing how much information they can extract from a given dataset of images . Since deep networks learn a hierarchy of representations , we further break down this investigation on a per-layer basis . We are motivated by the fact that the first few layers of most networks extract low-level information ( Yosinski et al. , 2014 ) , and thus learning them may not require the high-level semantic information captured by manual labels . Concretely , in this paper we answer the following simple question : “ is self-supervision able to exploit the information contained in a large number of images in order to learn different parts of a neural network ? ” We contribute two key findings . First , we show that as little as a single image is sufficient , when combined with self-supervision and data augmentation , to learn the first few layers of standard deep networks as well as using millions of images and full supervision ( Figure 1 ) . Hence , while selfsupervised learning works well for these layers , this may be due more to the limited complexity of such features than the strength of the supervisory technique . This also confirms the intuition that early layers in a convolutional network amounts to low-level feature extractors , analogous to early conv1 conv2 conv3 conv4 conv5 0 20 40 60 80 100 % su pe rv ise d pe rfo rm an ce Linear Classifier on ImageNet Random RotNet 1-RotNet BiGAN 1-BiGAN DeepCluster 1-DeepCluster Figure 1 : Single-image self-supervision . We show that several self-supervision methods can be used to train the first few layers of a deep neural networks using a single training image , such as this Image A , B or even C ( above ) , provided that sufficient data augmentation is used . learned and hand-crafted features for visual recognition ( Olshausen & Field , 1997 ; Lowe , 2004 ; Dalal & Triggs , 2005 ) . Finally , it demonstrates the importance of image transformations in learning such low-level features as opposed to image diversity.1 Our second finding is about the deeper layers of the network . For these , self-supervision remains inferior to strong supervision even if millions of images are used for training . Our finding is that this is unlikely to change with the addition of more data . In particular , we show that training these layers with self-supervision and a single image already achieves as much as two thirds of the performance that can be achieved by using a million different images . We show that these conclusions hold true for three different self-supervised methods , BiGAN ( Donahue et al. , 2017 ) , RotNet ( Gidaris et al. , 2018 ) and DeepCluster ( Caron et al. , 2018 ) , which are representative of the spectrum of techniques that are currently popular . We find that performance as a function of the amount of data is dependent on the method , but all three methods can indeed leverage a single image to learn the first few layers of a deep network almost “ perfectly ” . Overall , while our results do not improve self-supervision per-se , they help to characterize the limitations of current methods and to better focus on the important open challenges . 2 RELATED WORK . Our paper relates to three broad areas of research : ( a ) self-supervised/unsupervised learning , ( b ) learning from a single sample , and ( c ) designing/learning low-level feature extractors . We discuss closely related work for each . Self-supervised learning : A wide variety of proxy tasks , requiring no manual annotations , have been proposed for the self-training of deep convolutional neural networks . These methods use various cues and tasks namely , in-painting ( Pathak et al. , 2016 ) , patch context and jigsaw puzzles ( Doersch et al. , 2015 ; Noroozi & Favaro , 2016 ; Noroozi et al. , 2018 ; Mundhenk et al. , 2017 ) , clustering ( Caron et al. , 2018 ) , noise-as-targets ( Bojanowski & Joulin , 2017 ) , colorization ( Zhang et al. , 2016 ; Larsson et al. , 2017 ) , generation ( Jenni & Favaro , 2018 ; Ren & Lee , 2018 ; Donahue et al. , 2017 ) , geometry ( Dosovitskiy et al. , 2016 ; Gidaris et al. , 2018 ) and counting ( Noroozi et al. , 2017 ) . The idea is that the pretext task can be constructed automatically and easily on images alone . Thus , methods often modify information in the images and require the network to recover them . Inpainting or colorization techniques fall in this category . However these methods have the downside that the features are learned on modified images which potentially harms the generalization to unmodified ones . For example , colorization uses a gray scale image as input , thus the network can not learn to extract color information , which can be important for other tasks . Slightly less related are methods that use additional information to learn features . Here , often temporal information is used in the form of videos . Typical pretext tasks are based on temporalcontext ( Misra et al. , 2016 ; Wei et al. , 2018 ; Lee et al. , 2017 ; Sermanet et al. , 2018 ) , spatio-temporal 1Example applications that only rely on low-level feature extractors include template matching ( Kat et al. , 2018 ; Talmi et al. , 2017 ) and style transfer ( Gatys et al. , 2016 ; Johnson et al. , 2016 ) , which currently rely on pre-training with millions of images . cues ( Isola et al. , 2015 ; Gao et al. , 2016 ; Wang et al. , 2017 ) , foreground-background segmentation via video segmentation ( Pathak et al. , 2017 ) , optical-flow ( Gan et al. , 2018 ; Mahendran et al. , 2018 ) , future-frame synthesis ( Srivastava et al. , 2015 ) , audio prediction from video ( de Sa , 1994 ; Owens et al. , 2016 ) , audio-video alignment ( Arandjelović & Zisserman , 2017 ) , ego-motion estimation ( Jayaraman & Grauman , 2015 ) , slow feature analysis with higher order temporal coherence ( Jayaraman & Grauman , 2016 ) , transformation between frames ( Agrawal et al. , 2015 ) and patch tracking in videos ( Wang & Gupta , 2015 ) . Since we are interested in learning features from as little data as one image , we can not make use of methods that rely on video input . Our contribution inspects three unsupervised feature learning methods that use very different means of extracting information from the data : BiGAN ( Donahue et al. , 2017 ) utilizes a generative adversarial task , RotNet ( Gidaris et al. , 2018 ) exploits the photographic bias in the dataset and DeepCluster ( Caron et al. , 2018 ) learns stable feature representations under a number of image transformations by proxy labels obtained from clustering . These are described in more detail in the Methods section . Learning from a single sample : In some applications of computer vision , the bold idea of learning from a single sample comes out of necessity . For general object tracking , methods such as max margin correlation filters ( Rodriguez et al. , 2013 ) learn robust tracking templates from a single sample of the patch . A single image can also be used to learn and interpolate multi-scale textures with a GAN framework ( Rott Shaham et al. , 2019 ) . Single sample learning was pursued by the semi-parametric exemplar SVM model ( Malisiewicz et al. , 2011 ) . They learn one SVM per positive sample separating it from all negative patches mined from the background . While only one sample is used for the positive set , the negative set consists of thousands of images and is a necessary component of their method . The negative space was approximated by a multi-dimensional Gaussian by the Exemplar LDA ( Hariharan et al. , 2012 ) . These SVMs , one per positive sample , are pooled together using a max aggregation . We differ from both of these approaches in that we do not use a large collection of negative images to train our model . Instead we restrict ourselves to a single or a few images with a systematic augmentation strategy . Classical learned and hand-crafted low-level feature extractors : Learning and hand-crafting features pre-dates modern deep learning approaches and self-supervision techniques . For example the classical work of ( Olshausen & Field , 1997 ) shows that edge-like filters can be learned via sparse coding of just 10 natural scene images . SIFT ( Lowe , 2004 ) and HOG ( Dalal & Triggs , 2005 ) have been used extensively before the advent of convolutional neural networks and , in many ways , they resemble the first layers of these networks . The scatter transform of Bruna & Mallat ( 2013 ) ; Oyallon et al . ( 2017 ) is an handcrafted design that aims at replacing at least the first few layers of a deep network . While these results show that effective low-level features can be handcrafted , this is insufficient to clarify the power and limitation of self-supervision in deep networks . For instance , it is not obvious whether deep networks can learn better low level features than these , how many images may be required to learn them , and how effective self-supervision may be in doing so . For instance , as we also show in the experiments , replacing low-level layers in a convolutional networks with handcrafted features such as Oyallon et al . ( 2017 ) may still decrease the overall performance of the model . Furthermore , this says little about deeper layers , which we also investigate . In this work we show that current deep learning methods learn slightly better low-level representations than hand crafted features such as the scattering transform . Additionally , these representations can be learned from one single image with augmentations and without supervision . The results show how current self-supervised learning approaches that use one million images yield only relatively small gains when compared to what can be achieved from one image and augmentations , and motivates a renewed focus on augmentations and incorporating prior knowledge into feature extractors . 3 METHODS . We discuss first our data and data augmentation strategy ( section 3.1 ) and then we summarize the three different methods for unsupervised feature learning used in the experiments ( section 3.2 ) .

Summary: This paper explores self-supervised learning in the low-data regime, comparing results to self-supervised learning on larger datasets.  BiGAN, RotNet, and DeepCluster serve as the reference self-supervised methods.  It argues that early layers of a convolutional neural network can be effectively learned from a single source image, with data augmentation.  A performance gap exists for deeper layers, suggesting that larger datasets are required for self-supervised learning of useful filters in deeper network layers.