When Does Self-supervision Improve Few-shot Learning?
Authors: Jong-Chyi Su
Subhransu Maji
Bharath Hariharan
Venue:  (2020)
Keywords: Few-shot learning
Self-supervised learning
Meta-learning
Multi-task learning

Abstract: We present a technique to improve the generalization of deep representations learned on small labeled datasets by introducing self-supervised tasks as auxiliary loss functions. Although recent research has shown benefits of self-supervised learning (SSL) on large unlabeled datasets, its utility on small datasets is unknown. We find that SSL reduces the relative error rate of few-shot meta-learners by 4%-27%, even when the datasets are small and only utilizing images within the datasets. The improvements are greater when the training set is smaller or the task is more challenging. Though the benefits of SSL may increase with larger training sets, we observe that SSL can have a negative impact on performance when there is a domain shift between distribution of images used for meta-learning and SSL. Based on this analysis we present a technique that automatically select images for SSL from a large, generic pool of unlabeled images for a given dataset using a domain classifier that provides further improvements. We present results using several meta-learners and self-supervised tasks across datasets with varying degrees of domain shifts and label sizes to characterize the effectiveness of SSL for few-shot learning.

Under review as a conference paper at ICLR 2020
WHEN DOES SELF-SUPERVISION IMPROVE FEW-SHOT LEARNING?
Anonymous authors Paper under double-blind review
ABSTRACT
We present a technique to improve the generalization of deep representations learned on small labeled datasets by introducing self-supervised tasks as auxiliary loss functions. Although recent research has shown benefits of self-supervised learning (SSL.) on large unlabeled datasets, its utility on small datasets is un- known. We find that SSL reduces the relative error rate of few-shot meta-lcarners by 4%-27%, even when the datasets are small and only utilizing images within the datasets. The improvements are greater when the training set smaller or the task is more challenging. Though the benefits of SSL may increase with larger train- ing sets, we observe that SSL can have negative impact on performance when there is a domain shift between distribution of images used for meta-learning and SSL. Based on this analysis we present technique that automatically select im- ages for SSL from a large, generic pool of unlabeled images for a given dataset using domain classifier that provides further improvements. We present results using several meta-learners and self- supervisdd tasks across datasets with varying degrees of domain shifts and label sizes to characterize the effectiveness of SSL for few-shot learning.
INTRODUCTION
Current machine learning algorithms require enormous amounts of training data to learn new tasks. This is problem for many practical problems across domains such as biology and medicine where labeled data is hard to come by. In contrast, we humans can quickly learn new concepts from limited training data. We are able to do this by relying on our past "visual experience". Recent work attempts to emulate this by training feature representation to classify a training dataset of "base" classes with the hope that the resulting representation generalizes not just to unseen examples of the same classes but also to novel classes, which may have very few training examples (called few- shot learning). However, training for base class classification can force the network to only encode features that useful for distinguishing between base classes. In the process, it might discard semantic information that is irrelevant for base classes but critical for novel classes. This might be especially true when the base dataset is small when the class distinctions are challenging.
One way to recover this useful semantic information is to leverage representation learning techniques that do not use class labels, namely, unsupervised or self-supervised learning. The key idea is to learn about statistical regularities (e.g., spatial relationship between patches, orientation of an images) that might be a cue to semantics. Despite recent advances, these techniques have only been applied to a few domains (e.g., entry-level classes on internet imagery), and under the assumption that large amounts of unlabeled images are available. Their applicability to the general few-shot scenario described above is unclear. In particular, can these techniques help prevent overfitting to base classes and improve performance on novel classes in the few-shot setting? so, does the benefit generalize across domains and to more challenging tasks? Moreover, training to boost performance in domains where even unlabeled images are hard to get?
use self-supervised
This paper seeks to answer these questions. We show that with additional training data, adding a self-supervised task as an auxiliary task (Figure 1) improves the performance of existing few-shot techniques benchmarks across multitude of domains (Figure 2). Intriguingly, we find that the benefits of self-supervision increase with the difficulty of the task, for example when training from a smaller base dataset, or with degraded inputs such as low resolution greyscale images (Figure 3).

---

Under review as a conference paper at ICLR 2020
One might surmise that as with traditional SSL, additional unlabeled images might improve perfor- mance further. But what unlabeled images should we use for novel problem domains where unla- beled data is not freely available? To answer this, we conduct a series of experiments with additional unlabeled data from different domains. We find that adding more unlabeled images improves perfor- mance only when the images used for self-supervision are within the same domain the base classes (Figure 4a); otherwise they can even negatively impact the performance of the few-shot learner (Fig- ure 4b, 4c, 4d). Based on this analysis we present simple approach that uses a domain classifier to pick similar- -imiaa unlabeled images for self-suuprviiion from a large, generic pool of images. The resulting method improves over the performance of a model trained with super-useee learning from images within the domain (Figure 4c). Taken together, this results in powerful, general and practical approach for improving few-shot learning from small datasets in novel domains. Finally, these benefits are also observed on standard classification tasks (Appendix A.3).
RELATED WORK
Few-shot learning Few-shot learning aims to learn representations that generalize well to the novel classes where only few images are available. To this end several meta-learning approaches have been proposed that evaluate representations by sampling many few-shot tasks within the do- main of base dataset. These include optimization-basee meta-learners, such as model-agnostic meta-learner (MAML) (Finn et al., 2017), gradient unrolling (Ravi & Larochelle, 2017), closed form solvers (Bertinetto al., 2019). convex learners (Lee et al., 2019). etc. Others such as match- ing networks (Vinyals et al., 2016) and prototypical networks (Snell et al., 2017) rely on distance based classifiers, while Garcia & Bruna (2018) models higher-order relationships in data using a graph network. Another class of methods (Gidaris & Komodakis, 2018; Qi et al., 2018; Qiao et al., 2018) model the mapping between training data and classifier weights using a feed-forward network.
While the literature is rapidly growing, recent study by Chen et al. (2019) has shown that the differences between meta-learners are diminished when deeper network architectures are used and they develop a strong baseline for few-shot learning based on ResNet18 architecture and prototypical networks Snell et al. (2017). We build our experiments on top of this work and show that auxiliary self-supervised tasks provide additional benefits with tfte-of-thhear deep networks across several few-shot benchmarks. We also find that prototypical networks are often the best performing, though the benefits of SSL generalize across
such as MAML.
Self-supervised learning Human labels are expensive to collect and hard to scale up. To this end, there has been increasing research interest to investigate learning representations from unlaheled data. In particular, the image itself already contains structural information which can be utilized. Self-supervised learning approaches attempt to capture this.
There is rich line of work sulf-supervised learning. One class of methods removes part of the visual data, and tasks the network with predicting what has been removed from the rest in a dis- criminative manner (Larsson et al., 2016; Zhang al., 2016; 2017). Doersch et al. (2015) proposed to predict the relative position of patches cropped from images. Wang & Gupta (2015) used the similarity of patches obtained from tracking as self-supervised task and combined position and similarity predictions of patches ( (Wang et al., 2017). Other tasks include predicting rotation (Gi- daris et al., 2018), noise (Bojanowski & Joulin, 2017), clusters (Caron et al., 2018; 2019; Wu al., 2018), number of objects (Noroozi et al., 2017), missing patches (Pathak et al., 2016; Trinh et al., 2019), motion segmentation labels (Pathak et al., 2017), artifacts (Jenni & Favaro, 2018), next patch feature (Oord et al., 2018; Hénaff et al., 2019), or by maximizing mutual information (Hjelm et al., 2019: Bachman et al., 2019), and training an adversarial generative model (Donahue Simonyan, 2019). Doersch & Zisserman (2017) jointly trained four self-superviser tasks and found it to be ben- eficial. Recently, Goyal et al. (2019) and Kolesnikov et al. (2019) compared various self-supervised learning tasks at scale and concluded that solving jigsaw puzzles and predicting image rotations are among the most effective, motivating the choice of self-supervised tasks in our experiments.
The focus of most prior works on self-supervised learning is to supplant traditional fully supervised representation learning with unsupervised learning on large unlabeled datasets for downstream tasks. Crucially in almost all prior works, self-supervised representations consistently lag behind fully- supervised representations trained on the same dataset with the same architecture (Goyal et al., 2019; Kolesnikov et al., 2019). In contrast, our work focuses on important counterexample: self-
2

---

Under review as a conference paper at ICLR 2020
Prothonotary Warbier
image
supervised label
Permutation Rotation (8-180)
shared feature backbone
self-supervised label
rotation task jigsaw puzzle task
Figure Our framework combines supervised and self-supervised losses to learn represen- tations on small datasets. Self-supervised tasks, such as jigsaw puzzle or rotation prediction, can be derived from the images alone and act as a data-dependent regularizer for the shared feature backbone.
supervision in fact augment standard supervised training for few-shot transfer learning in the low training data regime without relying on any external dataset. Our work also investigates the effect of domain shifts across datasets used for SSL and given task.
Multi-task learning Our work is related to multi-task learning, a class of techniques that train on multiple task objectives together improve each one. Previous works in the computer vision literature have shown moderate benefits by combining tasks such as edge, normal, and saliency estimation for images, or part segmentation and detection for humans (Kokkinos, 2017; Maninis et al., 2019; 9; Ren & Lee, 2018). However, there is significant evidence that training on multiple tasks together often hurts performance on individual tasks (Kokkinos, 2017; Maninis al., 2019). Only certain task combinations appear to be mutually beneficial, and sometimes specific architectures are needed. Our key contribution here showing that self-supervised tasks and few-shot learning are indeed mutually beneficial in this sense. The fact that self-supervised tasks require no annotations makes our finding especially practically significant.
Concurrent works At the time of submission several concurrent works appear draw similar conclusions to ours. We briefly describe these, deferring a detailed comparison to the future when code and experimental details become available, Asano al. (2019) finds that self-supervised pre- training is useful even with a single image. However, very large images and extreme data augmen- tations are used. Carlucci al. (2019) daris et al. (2019), the most related work to ours, also shows that self-supervision improves few-shot learning. Although this initial result is the same, we further show these benefits on smaller datasets with harder recognition problems fine-grained classification) and with deeper models. Moreover, we also present novel analysis of the impact of the domain of unlabeled data, and new, simple approach to automatically select similar-domain unlabeled data for self-supervision.
self-supervision improve domain generalization. Gi-
3
METHOD
We adopt the commonly used setup for few-shot learning where one is provided with labeled training data for of base classes and much smaller training set (typically 1-5 examples per class) for novel classes D,,. The goal of the few-shot learner is to learn representations lead good generalization on novel classes. Although in theory the base classes are assumed have a large number of labeled examples, in practice this number can be quite small for novel domains, e.g. less than 5000 examples for the birds dataset (Welinder al., 2010), making it challenging to learn generalizable representation.
the base classes that
Our framework, as seen in Figure 1, combines meta-learning approaches for few-shot learning that rely supervised labels with self-supervised learning unlabeled images. Denote training dataset as { H consisting pairs images x; E X' and labels yi E V. feed-forward convolutional network f()) maps the input to embedding space, which is then mapped to the label space using classifier g. The overall mapping from the input to the label can be written as g f(x) g X + I Learning consists of estimating functions and g that minimize loss l on the

---

Under review as a conference paper at ICLR 2020
predicted labels on the training data along with suitable regularization R over the functions f and g- This can be written as:
C. -, Σ -= f(.)))))))) R)f.g).
A commonly used loss is the cross-entropy loss and a regularizer is the &g norm of the parameters of the functions. For transfer g is discarded and relearned on training data for novel classes.
In addition, we consider self-supervised losses Les based on labeled data æ + (#; f that can be derived from inputs alone. Figure 1 shows two examples: the jigsaw task rearranges the input image and uses the index of the permutation as the target label, while the rotation task uses the angle of the rotated image the target label. In our framework, separate function h is used to predict these labels from the shared feature backbone / with a self-supervised loss:
Las = Σ ((h (B:),
Our final loss function combines the two losses: C L. + Las. and thus the self-supervised losses act a data-dependent regularizer for representation learning. Note that the domain of images used for supervised and supervieed losses denoted by D3 and Das need not be identical. In particular often would like to use significantly larger sets of images for self-supervised learning from related domains. Below we describe various losses we consider for image classification tasks.
3.1 SUPERVISED LOSSES (L.)
Most of our results are presented using a meta-learner based on prototypical networks (Snell et al., 2017) that perform episodic training and testing over sampled datasets in stages called meta-training and meta-testing. During meta-training, we randomly sample N classes from the base set D,, then select support
S, with K images per class and another query set 2s with M images per class, We call this an N-way K-shot classification task. The embeddings are trained to predict the labels of the query set Q6 conditioned on the support set S using a nearest mean (prototype) model. The objective is to minimize the prediction loss on the query set. Once training is complete, given the novel dataset D,,, class prototypes are recomputed for classification and query examples are classified based on the distances to the class prototypes.
Prototypical networks are related to distance-based learners such as matching networks (Vinyals et al., 2016) or metric-learning based on label similarity (Koch et al., 2015). In the Appendix we also present few-shot classification results using a gradient- based meta-learner called MAML (Finn et al., 2017), and one trained with a standard cross-entropy loss on the base classes. We also present standard classification results where the test set contains images from the same base categories.
3.2 SELF-SUPERVISED LOSSES (C..)
We consider two losses motivated by a recent large-scale comparison of the effectiveness of self- supervised learning tasks (Goyal et al., 2019) described below:
Jigsaw puzzle task loss. In this case the input image æ is tiled into 3x3 regions and permuted randomly to obtain input &. The target label is the index of the permutation. The index (one of 9!) is reduced to one of 35 following the procedure outlined in Noroozi & Favaro (2016), which grouped the possible permutations based on hamming distance control the difficulty of the task.
Rotation task loss. We follow the method of Gidaris et al. (2018) where the input image x rotated by an angle 0 (00, 9090 180", 270%) to obtain f and the target label is the index of the angle.
In both cases we use the cross-entropy loss between the target and prediction.
3.3 STOCHASTIC SAMPLING AND TRAINING
When the source images used for SSL and meta-learning are identical the same batch of images used for computing L, and L.s- For experiments investigating domain shifts (Sec. 4.2), where SSL and meta-learne are trained on different distribution of images, a different batch (size of 64) is used for computing f... After the two forward passes, one for the supervised task and one for the
4

---

Under review as a conference paper at ICLR 2020
self self-supervised task, the two losses are combined. While other techniques exist (Chen et al., 2018; Kendall et al., 2018; Sener& & Koltun, 2018), simply combining the two losses performed well.
EXPERIMENTS
We first describe the datasets and experimental details. We then present results for few-shot learning on different datasets and SSL tasks in Section 4.1. Last, we investigate the effectiveness of SSL with unlabeled data sampled from different domains in Section 4.2.
Datasets and benchmarks We experiment with datasets across diverse domains: Caltech-UCSD) birds (Welinder et al., 2010), Stanford cars (Krause et al., 2013), FGVC aircrafts (Maji al., 2013), Stanford dogs (Khosla et al., 10, and Oxford flowers (Nilsback & Zisserman, 2006). Each dataset contains between 100 to 200 classes with few thousands of images. We also experi- ment with the widely-used mini-ImageNet (Vinyals et al., 2016) and liered-ImageNet (Ren et al., 2018) benchmarks for few-shot learning. In Imag-em, each class has 600 images, where in tiered- -mree-el each class has 732 to 1300 images.
We split classes within dataset into three disjoint sets: base, val, and novel. For each class all the images in the dataset are used in the corresponding set. A model is trained on the base set of categories, validated on the val set, and tested on the novel set of categories given a few examples per class. For birds, we use the same split as Chen et al. (2019), where (buse, val, novel) sets have {100, 50, 50} classes respectively. The same ratio used for the other four fine-grained datasets. We use the original splits for mini-ImageNet and tiered Netgeee.
We also present results on a setting where the base set is "degraded" either by reducing the resolu- tion, removing color, or the number of training examples to 20% of the original set. This allows us to study the effectiveness of SSL on even smaller datasets and as a function of the difficulty of the task. Further details of the datasets are in the Appendix A.
Meta-learners and feature backbone We follow the best practices and use the codebase for few- shot learning described in Chen et al. (2019). In particular, we use prototypical network (Snell et al., 2017) with ResNet18 (He et al., 2016) as the feature backbone. Their experiments found this to often be the best performing: however, we also experiment our method with other meta-learners such as MAML (Finn et al., 2017) and softmax classifiers.
Learning and optimization use use 5-way (classes) and 5-shot (examples per-class) with 16 query images for training. For experiments using 20% of labeled data, we use 5 query images for training since the minimum number of images per class is 10. The models are trained with ADAM (Kingma & Ba, 2015) with a learning rate of 0.001 for 60,000 episodes. We report the mean accuracy and 95% confidence interval over 600 test experiments. In each test episode, N classes are selected from the novel set, and for each class 5 support images and 16 query images (or 5 query images when using 20% labeled data) are used. We report results for = (5. 20) classes.
Image sampling and data augmentation Data augmentation has a significant impact on few-shot learning performance. We follow the data augmentation procedure outlined in Chen al. (2019) which resulted in a strong baseline performance of meta-learning, For label and rotation predictions, images are first resized to 224 pixels for the shorter edge while maintaining the aspect ratio, from which a central crop of 224 224 is obtained. For jigsaw puzzles, we first randomly crop 255 x 255 region from the original image with random scaling between [0.5, 1.0). then split into 3x3 regions, from which a random crop of size 64 is picked. While it might appear that with self-supervision the model effectively sees more images, we wish to point out that all the experiments use signif- icant data augmentation, including cropping, flipping, and color jittering, which also significantly increases the amount of training data. The gains from the self-supervised auxiliary tasks are over and above any gains from data augmentation alone. More experimental details are in Appendix A.5.
Other experimental results In Appendix A.3, we show the benefits of using self- sulf-supervision for standard classification tasks when training the model from scratch. We further visualize these models in Appendix A.4 to show that models trained with self-supervissio tend to avoid accidental correlation of background features to class labels.
5

---

Under review as a conference paper at ICLR 2020
ProtoNet
ProtoNet + Jigsaw
ProtoNet Rotation
ProtoNet
Jigsaw Rotation
95
90
Accuracy
65
80
75-
mini--mageNet
tiered-ImageNet
Birds
Cars
Aircrafts
Dogs
Flowers
Figure 2: Benefits of SSL for few-shot learning tasks. We show the accuracy over the ProtoNet baseline of using different SSL tasks. The jigsaw task results in improvement of the 5-way 5-shot classification accuracy for a ProtoNet across datasets. Combining SSL tasks can be beneficial for some datasets. Here SSL was performed images within the base classes only. See Table in the Appendix for a tabular version and results for 20-way 5-shot classification.
ProtoNet (8-waa)
ProtoNet Jigsaw (3 wway)
ProtoNet (20wayy)
ProtoNet Jigeaw (20-way)
Acouracy
70
50-
40
Birds Grayscalo
Cars Lowiros.
Airoratis
Dogs Groyscale
Flowers Creyscale
Blrds 20%
Cars 20%
Aircrats 20%
Dogs 20%
Flowers 23%
Low-ros.
Figure 3: Benefits SSL for harder few-shot learning tasks. We show the accuracy of using jigsaw puzzle task over ProtoNet baseline on harder versions of the datasets. We see that SSL is effective even on smaller datasets and the relative benefits are higher.
4.1 RESULTS ON FEW-SHOT LEARNING
Self-supervised learning improves few-shot learning Figure shows the accuracies of various models for 5-way 5-shot classification on few-shot benchmarks. Our ProtoNet bascline matches the results of the mini- ImageNet and birds datasets presented in Chen et al. (2019) (in their Table A5). Our results show that jigsaw puzzle task improves the ProtoNet baseline on all seven datasets. Specifically, it reduces the relative error rate by 4.0%, 8.7%, 19.7%, 8.4%, 4.7%, 15.9%, and 27.8% on mini- Imagee,, tiered-Iman Imtgee,,, birds, cars, airerafts, dogs, and flowers datasets respectively. Predicting rotations also improves the ProtoNet baseline most of the datasets, except for aircrafts and flowers. We speculate this is because most flower images are symmetrical, and airplanes are usually horizontal, making the rotation task too hard too trivial to benefit main task. In addition, combining these two SSL tasks can be beneficial sometimes. Our results also echo those of Gidaris al. (2019) who find that the rotation task improves mini- and lirgee- A tabular version of the figure and the results for 20-way classification are included in Table Appendix A.2.
Gains larger for harder tasks Figure 3 shows the performance the degraded version of the same datasets (first five groups). For cars and aircrafts we use low-resolution images where the factor of four and up-sampled back to x24x224 with bilinear inter- polation. For natural categories we discard color. Low-resolution images are considerably harder to classify for man-made categories while color information most useful for natural categories (Su & Maji, 2017). On birds and dogs datasets, the improvements using self-supervision (3.2% and 2.9% on 5-way 5-shot) are higher compared to color images (2.5% and 2.7%), similarly on the cars and aircrafts datasets with low-resolution images (2.2% and 2.1% vs. 0.7% and 0.4%). We also conduct an experiment where only 20% of the images the base categories are used for both SSL and meta-learning (last five groups in Figure 3). This results in much smaller training set than standard few-shot benchmarks: 20% the birds dataset amounts to only roughly 3% of the popular mini- ImageNe dataset. We find larger benefits from SSL in this setting. For example, the gain from the jigsaw puzzle loss for 5-way 5-shot car classification increases from 0.7% (original dataset) to 7.0% (20% training data).
images down-sampled

---

Under review as a conference paper at ICLR 2020
Brde
Cars Arceis
Doge Flowers
Arcneta
Dage Flowrs
702-
images
(a) Effecct of number of images SST..
(h) Effect of domain shift SSL.
Selecting unlabeled data for SSL.
Birds
Cara
Amtraitt
Dogs
F Fowerss
Nott Aves
Nait other
mini-traseNet
Dy Birds
D, - Cars
Airorafts
D, Dogs
Dj - Flowers
Acarmny
32:0
623
Domain distance between D. and Das
(d) Effectivvneess of SSL as function domain distance between 2), and D.n. (shown on lop).
Figure 4: Effect of size and domain of SSL on 5-way 5-shot accuracy. (a) More unlabeled
data from the same domain for SSL improves the performance of the meta-leamer. (b) Replacing a fraction (X-axis) of the images with those from other domains makes SSL less effective. (c) comparison of approaches to sample images from pool for SSL. With random sampling, the extra unlabeled data often hurts the performance, while those sampled using the "domain weighs" improves performance on most datasets. When the pool contains images from related domains, egg. birds and dogs, the performance using selected images matches that of using 100% within dattsst images. (d) Accuracy of the meta-learner with SSL on different domains as function distance between the supervised domain D, and the self-supervised domain D,.- See text for details.
Improvements generalize to other meta-learners We combine SSL with other meta-learners and find the combination to be effective. In particular we use MAML (Finn et al., 2017) and a standard feature extractor trained with cross-entropy loss (softmax). We observe that the average 5-way 5- shot accuracies across five fine- fine-graing grained datasets for softmax, MAML, and ProtoNet improve from 85.5%, 82.6% and 88.5% 86.6% 83.8% and 90.4% respectively when combined with the jigsaw puzzle task. Detailed results are shown in Table 4 in Appendix A.2.
Self-supervision alone is not enough SSL alone significantly lags behind supervised learning in our experiments. For example, ResNet18 trained with SSL alone achieve 32.9% (w/ jigsaw) and 33.7% (w/ rotation) 5-way 5-shot accuracy averaged across five fine-grained datasets. While this is better than random initialization (29.5%), it is dramatically worse than one trained with simple cross-entropy loss (85.5%) on the labels (Details in Table 2). Surprisingly, we found that initialization with SSL followed by meta-learning did not yield improvements over meta-learning starting from random initialization, supporting the view that SSL acts as a feature regularizer.
4.2 SELECTING DOMAINS FOR SELF-SUPERVISION
promising direction of research in SSL is by scaling it massive unlabeled datasets which are readily available for some domains. However, does more unlabeled data always help for a task in hand? This question hasn't been sufficiently addressed in the literature as most prior works study the effectiveness of SSL on curated set of images, such as ImageNet, and their transferability to handful tasks. In this section we conduct a series experiments to characterize effect of size and distribution D.. of images used for SSL in the context of few-shot learning domain D..
First we investigate SSL on unlabeled data from the same domain improves the meta-learned We use 20% of the images in the base categories for meta-learming identical setting in Figure 3. The labels of the remaining 80% data are withheld and only the images are used for SSL. We systematically vary the number of images used by the SSL from 20% to 100%. Results presented
7

---

Under review as a conference paper at ICLR 2020
in Figure 4a. The accuracy improves with the size of the unlabeled set with diminishing returns. Note that 0% corresponds to no SSL and 20% corresponds to using only the labeled images for SSL.
Figure 4b shows experiment where fraction of the unlabeled images are replaced with images from other datasets. For example 20% along the x-axis for birds indicate that 20% of the images in base set replaced by images drawn uniformly at random from other datasets. Since the number of images used for SSL are identical, the x-axis from left to right represents increasing amounts of domain shifts between D, and D... We observe that the effectiveness of SSL decreases as the fraction of out-of-domain images increases. Importantly, training with SSL on the available 20% within domain images (shown as crosses) often (3 out of datasets) better than increasing the set of images by five times to include out of domain images.
Figure 4d analyzes how the magnitude of the domain shift affects performance. Here all the images in base classes are used (instead of using 20% data in previous experiment), and all the images for SSL are sampled from different datasets. In addition to the five datasets (birds, cars, aircrafts, dogs, flowers), we include 1) 5000 images of Aves from iNaturalist 2018 dataset (iNat) (Van Horn et al., 2018), 2) 5000 images of other super-classes from iNat, and 3) 5760 images sampled from mini- ImageNet. These domains are roughly of the same sizes (between 5k to 10k images each) but exhibit varying degrees of domain shifts. To compute a "distance" " between a pair of domains, we first es- timate an embedding for each domain D using the mean image feature: (DD) EmED (xx,)/,, where N is the number of images within 'D, and use the Fuclidean distance between their em- beddings. Here ((x) is image embedding from ResNet101 network pretrained ImageNet. Figure 4d shows the accuracy of the meta-learner when combined with SSL on different domains as a function of the domain distance between D, and D... Once again we see that the effectiveness of SSL decreases with the distance from the supervised domain across all datasets.
Based on the above analysis we propose a simple method to select images for SSL from large, generic pool of unlabeled images in a dataset dependent manner. We use a "domain weighted" model select the top images based on domain classifier, in our case a binary logistic regression model trained with images from the source domain DD as the positive class and from pool Dp as the negative class based on ResNet101 image features. The top images are selected according to the ratio p( E D.)/p(x E Dp). Note that these importance weights account for the domain shift. We evaluate this approach using a pool of images from all five datasets as well as 30000 images each from Aves of iNat, other super-classes of iNat and mini-Imaged of the labeled images for meta-learning, then use the pool in a leave-one-or out manner: all images from dataset excluded from the pool to simulate scenario where one does not have additional within-domain images. The compositions of the selected images are shown in Appendix A.6.
For each dataset we use 20%
Figure 4c shows results across datasets with meta-learner trained 20% labeled examples. SSL with 20% and 100% dataset denotes baselines when SSL uses 20% and 100% of the images within domain. The latter is reference "upper bound". SSL pool "(random)" and "(weight)" denote two approaches for selecting the remaining 80% of images from the pool. The former selects images uniformly at random, which is detrimental for cars, dogs, and flowers. For dogs the performance is even worse than no SSL. The pool selected according to the importance weights provides signifi- cant improvements on birds and dogs as the pool has relevant images, and outperforms the random selection on most datasets. These results are likely to improve by collecting an even larger pool.
5 CONCLUSION
We have shown that self-supervision improves transferability of representations on few-shot learn- ing tasks across a range of different domains. Surprisingly, we found that self-supervision is more beneficial for more challenging problems, especially when the number of images used for self- supervision is small, orders of magnitude smaller than previously reported results. This has practi- cal benefit that the images within small datasets can be used for self-supervision without relying large-scale external dataset. We have also shown that additional unlabeled images can improve per- formance only if they are from the same or similar domains. Finally, for domains where unlabeled data is limited, we present a novel, simple approach to automatically identify such similar- domain images from larger pool.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Yuki M Asano, Christian Rupprecht, and Andrea Vedaldi. Surprising effectiveness of few-image
unsupervised feature learning. arXiv preprint arXiv: 1199044133222 2019. 3
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. arXiv preprint arXiv:1906. 00900, 2019. 2
Luca Bertinetto, Joao Henriques, Philip HS Torr, and Andrea Vedaldi. Meta-learning with dif- ferentiable closed-form solvers. International Conference Learning Representations (ICLR), 2019. 2
Piotr Bojanowski and Armand Joulin. Unsupervised learning by predicting noise. In International Conference on Machine Learning (ICML), 2017.
Fabio M Carlucci, Antonio D I'nnocente, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. Do- main generalization by solving jigsaw puzzles. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), 2019. 3
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In European Conference Computer Vision (ECCV), 2018.
Mathilde Caron, Piotr Bojanowski, Julien Mairal, and Armand Joulin. Leveraging large-scale un- curated data for unsupervised pre-training of visual features. arXiv preprint N0Xi::990500220 2019. 2
Wei- Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Wang, and Jia-Bin Huang. A Closer Look at Few-shot Classification. In International Conference on Learning Representations (ICLR), 2019.
2, 5, 6, 15
Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In International Conference on Machine Learning (ICML), 2018. 5
Carl Doersch and Andrew Zisserman. Multi-task self-supervised visual learning. In International Conference on Computer Vision (ICCV), 2017. 2
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In International Conference on Computer Vision (ICCV), 2015.
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. arXiv preprint 1 2019. 2
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning (ICML), 2017. 2, 4, 5,
Victor Garcia and Joan Bruna. Few-shot learning with graph neural networks. In International Conference on Learning Representations (ICLR), 2018. 2
Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Spyros Gidaris. Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In International Conference on Learning Representations (ICLR), 2018. 2,
Spyros Gidaris, Andrei Bursuc, Nikos Komodakis, Patrick Pérez, and Matthieu Cord. Boosting few-shot visual learning with self-supervision. arXiv preprint arXiv:1906.05186, 2019.
Priya Goyal, Dhruv Mahajan, Abhinav Gupta, and Ishan Misra. Scaling and benchmarking self- supervised visual representation learning- arXiv preprint arXiv:1955.0223 2019. 2,

---

Under review as a conference paper at ICLR 2020
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Computer Vision and Pattern Recognition (CVPR), 2016. 5
Olivier J Hénaff, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. arXiv preprint arXiv: (eto::99599222 2019. 2
R Devon Hjelm, Alex Fedorov, Samuel Gavoie-Marchiloe Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations (ICLR), 2019. 2
Simon Jenni and Paolo Favaro. Self-supervised feature learning by learning spot artifacts. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2018.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2018. 5
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Li Fei-Fei. Novel dataset for fine- grained image categorization. In First Workshop on Fine-Grained Visual Categorization, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2011.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In International
Conference on Learning Representations (ICLR), 2015. 5
Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. Siamese neural networks for one-shot image recognition. In ICML deep learning workshop, volume 2, 2015. 4
Iasonas Kokkinos. Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR), 2017. 3
Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting self-supervised visual represen- tation learning. arXiv preprint arXiv: 190100900, 2019. 2
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3D Object Representations for Fine- Grained Categorization. In 4th International IEEE Workshop on 3D Representation and Recog- nition (3dRR-13), Sydney, Australia, 2013. 5
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Learning representations for auto- matic colorization. In European Conference on Computer Vision (ECCV), 2016. 2
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In Computer Vision and Pattern Recognition (CVPR), 2019. 2
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.515,, 2013. 5
Kekis-Kokitsi Maninis, Ilija Radosavovic, and lasonas Kokkinos. Attentive single-tasking of multi- ple tasks. arXiv preprint /tti::/994.0800 2019. 3
M-E. Nilsback and A. Zisserman. A visual vocabulary for flower classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), volume 2, 2006. 5
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. European Conference on Computer Vision (ECCV), 2016. 4, 15
Mehdi Noroozi, Hamed Pirsiavash, and Paolo Favaro. Representation learning by learning to count.
In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017. 2
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv: 807.003748 2018. 2
10

---

Under review as a conference paper at ICLR 2020
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vitito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NeurlPS Autodiff Workshop, 2017. 15
Deepak Pathak, Philipp Krähenbühl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros. Context encoders: Feature learning by inpainting. In Computer Vision and Pattern Recognition (CVPR), 2016. 2
Deepak Pathak, Ross B. Girshick, Piotr Dollár, Trevor Darrell, and Bharath Hariharan. Learning features by watching objects move. In Computer Vision and Pattern Recognition (CVPR), 2017.
Hang Qi, Matthew Brown, and David G Lowe. Low-shot learning with imprinted weights. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Siyuan Qiao, Chenxi Liu, Wei Shen, and Alan Yuille. Few-shot image recognition by predicting parameters from activations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. 2
Sachin Ravi and Hugo Larochelle. Optimization model for few-shot learning. In International Conference on Learning Representations (ICLR), 2017. 2
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenenbaum, Hugo Larochelle, and Richard S Zemel. Meta-learning for semi-supervised few-shot classifica- tion. International Conference Learning Representations (ICLR), 2018. 5
Zhongzheng Ren and Yong Jae Lee. Cross-domain self-supervised multi-task feature learning us- ing synthetic imagery. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2018. 3
Ozan Sener and Vladlen Koltun. Multi-task learning multi-objective optimization. In Advances in Neural Information Processing Systems (NeurIPS), 2018. 5
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems (NeurIPS), 2017. 2, 4,
Jong-Chyi Su and Subhransu Maji. Adapting models signal degradation using distillation. In British Machine Vision Conference (BMVC). 2017. 6
Trieu H Trinh, Minh-Thang Luong, and Quoc V Le. Selfie: Self-supervised pretraining for image embedding. arXiv preprint arXiv: 02940, 2019.
Grant Van Hom, Oisin Mae Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems (NeurIPS), 2016. 2, 4.
Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using videos. In International Conference on Computer Vision (ICCV), 2015. 2
Xiaolong Wang, Kaiming Hc, and Abhinav Gupta. Transitive invariance for sulervupe visual representation learning. In International Conference Computer Vision (ICCV), 2017. 2
Welinder, S. Branson, Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-UCS Birds 200. Technical Report CNS-TR-2010-001, Califomia Institute of Technology, 2010. 3,5
Davis Wertheimer and Bharath Hariharan. Few-shot learning with localization realistic settings. In Computer Vision and Pattern Recognition (CVPR), 2019. 14
11

---

Under review as a conference paper at ICLR 2020
Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Computer Vision and Pattern Recognition (CVPR), 2018.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European Con- ference on Computer Vision (ECCV), 2016.
Richard Zhang, Phillip Isola, and Alexei A Efros. Split-brain autoencoders: Unsupervised learning by cross-channel prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 2
A APPENDIX
A.1 DATASET DETAILS
The statistics of various datasets used in experiments shown in Table 1. We show results on several fine- tinengraieed classification datasets in addition to standard few-shot benchmarks such as mini- and tiered- ferred-ImageNet.. Notably, the fine-grained datasets significantly smaller.
Birds
Cars
Aircrafts
Dogs
Flowers
mini- ImageNet
tiered- ImageNet
Setting
Set
Stats
Birds
Cars
Aircrafts
Dogs
Flowers
class
64
351
100
98
50
60
51
Base
# images
38,400
448,695
5885
8162
5000
10337
4129
Few-shot transfer
# class
16
97
50
49
25
30
26
Val
# images
9,600
124.261
2950
3993
2500
5128
2113
class
20
160
50
49
25
30
25
Novel
# images
12,000
206,209
2953
4030
2500
5115
1947
Table Example images and dataset statistics. For few-shot learning experiments the classes are split into base, val, and novel set. Image representations learned on base set are evaluated the novel set while val set is used for cross-validation. These datasets vary in the number classes but orders of magnitude smaller than ImageNet dataset.
A.2 RESULTS ON FEW-SHOT LEARNING
Table 2 shows the performance of ProtoNet with different self-supervision on seven datasets. We also test the accuracy of the model on novel classes when trained only with self-supervision on the base set of images. Compared to the randomly initialized model ("None" rows), training the network to predict rotations gives around 2% to 21% improvements on all datasets, while solving jigsaw puzzles only improves on aircrafts and flowers. However, these numbers are significantly worse than learning with supervised labels on the base set, in line with the current literature.
Table 3 shows the performance of ProtoNet with jigsaw puzzle loss harder benchmarks. The results on degraded version of the datasets are shown in the top part, and the bottom part shows the results of using only 20% of the images in the base categories. The gains using SSL are higher in this setting.
Table compares meta-leamers based ResNet18 network trained with and without jigsaw puzzle loss. Self-supervision improves the performance across different meta-learners and different datasets. ProtoNet trained with self-supervision the best model across all datasets.
12

---

Under review as a conference paper at ICLR 2020
mini- ImageNet
tiered- ImageNet
Birds
Cars
Aircrafts
Dogs
Flowers
Loss
5 way 5-shot
ProtoNet
7..2=0.6
73.9=0.7
07.3=0.5
91.7=0.4
91.4+0.4
83.0=0.6
89.2=0.66
ProtoNet Jigsaw
76.2+0.6
78.0+0.7
89.8
92.440.4
91.840.4
85.7
92.2.1044
Rel. error reduction
4.0%
8.7%
19.7%
8.4%
4.7%
15.9%
27.8%
Jigsaw
25.610.5
24.910.4
25.740.5
25.3:0.5
38.810.6
24.3:0.5
50.510.7
ProtoNet + Rotation
76.0=0.6
78.9=0.7
89.4+0.4
92.3#0.4
91.4+0.4
84.3=0.5
89.0=0.5
Rotation
51.4.0.77
50.7.0.8
33.110.6
29.410.5
29.510.5
27.310.5
49.411..7
ProtoNet + Jig. + Rot.
76.6=0.7
77.2+0.7
90.2+0.4
92.7=0.4
91.9=0.4
85.9
91.4+0.5
None
31.010.5
28.910.5
26.710.5
25.210.5 28.1
25.3
42.310.8
20-way 5-shot
20
ProtoNet
46.6#0.3
49.7+0.4
69.3+0.3
78.7=0.3
78.6=0.3
61.6=0.3
75.4+0.3
ProtoNet + Jigsaw
4..
5.44
73.7.0.3
79.1+0.3
79.140.2
65.4
79.2-1.3
Jigsaw
9.2+0.2
7.5+0.1
8.140.1
7.1=0.1
15.4+0.2
7.1+0.1
25.7=0.2
ProtoNet + Rotation
48.240.3
52.4.0.4
72.910.3
80.0
78.410.2
63.410.3
73.910.3
Rotation
27.4+0.2
25.7+0.3
12.9+0.2
9.3=0.2
9.8+0.2
8.8+0.1
26.3+0.2
ProtoNet + Jig. + Rot.
49.0
51.210.4
75.0
79,810.3
79.0.0.2
66.210.3
78.610.3
None
10.8+0.1
11.0+0.2
9.3+0.2
7.5=0.1
8.940.1
7.8+0.1
22.6+0.2
Table 2: Performance on few-shot transfer task. The mean accuracy (%) and the 95% confidence interval of 600 randomly chosen experiments are reported for various combinations of loss functions. The top part shows the accuracy on 5-way 5-shot classification tasks, while the bottom part shows the same on 20-way 5-shot. Adding self-supervised losses the ProtoNet loss improves the performance on all seven datasets on 5-way classification results. On 20-way classification, the improvements are even larger. The last row indicates results with a randomly initialized network. The top part of this table corresponds to Figure 2.
Birds
Cars Low-resolution
Airerafts Low-resolution
Dogs Greyscale
Flowers Greyscale
Loss
Greyscale
5-way 5-shot
ProtoNet
82.2=0.6
84.8=0.5
85.0+#.55
80.7=0.6 83.6=0.5
86.1=0.6
ProtoNet Jigsaw
85.4=0.6
87.0=0.5
87.1
87.6=0.5
20-way 5-shot 64.1#0.3
ProtoNet
60.8=0.4
64.7=0.3
57.4=0.3
69.7=0.3
ProtoNet Jigsaw
65.7+0.3
68.6=0.3
68.3+0.3
61.2=0.3
71.6=0.3
Loss
20% Birds
20% Cars
20% Aircrafts
20% Dogs
20% Flowers
5-way 5-shot
ProtoNet
73.0=0.7
75.8=0.7
77.7+0.6
68.5=0.7 69.1=0.7
82.2=0.7
ProtoNet Jigsaw
75.4=0.7
82.8=0.6
78.4#0.6
86.0=0.6
20-way 5-shot
ProtoNet 46.4:1.33
51.8=0.4
52.340.3
40.8=0.3
62.8=0.3
ProtoNet Jigsaw
49,8+0.3
61.5=0.4
53.6#0.3
42.2=0.3
68.5=0.3
Table 3: Performance few-shot transfer task with degraded inputs or less training data. Accuracies are reported on novel set for 5-way 5-shot and 20-way 5-shot classification with degraded inputs, and with subset (20%) of the images in the base set. The loss of color resolution, and the smaller training set size make the tasks more challenging as seen by the drop the performance of the ProtoNet baseline. However the improvements of using the jigsaw puzzle loss higher in comparison to the results presented in Table 2.
13

---

Under review as a conference paper at ICLR 2020
Birds
Cars
Airerafts
Dogs
Flowers
Loss
5-way 5-shot
Softmax
81.5=0.5
87.7=0.5
89.2=0.4
77.6=0.6
91.3+0.5
Softmax + Jigsaw
83.9
90.6
89.610.4
77.8
90.9
MAML
81.2+0.7
86.9
88.8
77.3
79.0:0.9
MAML Jigsaw
81.1=0.7
89.0=0.5
89.1
77.3=0.7
82.6#0.7
ProtoNet
87.3=0.5
91.7=0.4
91.4=0.4
83.0=0.6
89.2=0.6
ProtoNet
Jigsaw
89,8=0.4
92.4=0.4
91.8=0.4
85.7=0.5
92.2=0.4
Table 4: Performance on few-shot transfer task using different meta-learners. Using jigsaw puzzle loss gives improvements across different meta-learners on most of the datasets. ProtoNet with jigsaw loss performs the best on all five datasets.
A.3 RESULTS ON STANDARD FINE-GRAINED CLASSIFICATION
Here we present results on standard classification task. Different from few-shot transfer learning, all the classes are seen in the training and test set contains novel images from the classes. We use the standard training and test splits provided in the datasets. We investigate if SSL can improve training of deep networks (e.g. ResNet18 network) when trained from scratch (i.e. with random initialization) using images and labels in the training set only. The accuracy using various loss functions are shown in Table 5. Training with sule-suupn improves performance across datasets. On birds, cars, and dogs, predicting rotation gives 4.1%. 3.1% 1,, and 3.0% improvements, while on aircrafts and flowers, the jigsaw puzzle loss yields 0.9% and 3.6% improvements.
Loss
Birds
Cars
Aircrafts
Dogs
Flowers
Softmax
47.0
72.6
69.9
51.4
72.8
Softmax Jigsaw
49.2
73.2
70.8
53.5
76.4
Softmax Rotation
51.1
75.7
70.0
54.4
73.5
Table 5: Performance on standard classification task. Per-image accuracy (%) on the test set are reported. Using self-supervision improves the accuracy of ResNet18 network trained from scratch over the baseline of supervised training with cross-entropy (softmax) loss on all five datasets.
A.4 VISUALIZATION OF LEARNED MODELS
To understand why the representation generalizes, we visualize what pixels contribute the most to the correct classification for various models. In particular, for each image and model compute the gradient of the logits (predictions before softmax) for the correct class with respect to the input image. The magnitude of the gradient at each pixel is proxy for its importance and is visualized as "saliency maps". Figure 5 shows these maps for various images and models trained with and without self-supervision
the standard classification task. It appears that the self supervipe models tend to focus more on the foreground regions, as seen by the amount of bright pixels within the bound- ing box. One hypothesis that self-supervised tasks force the model rely less background features, which might be accidentally correlated to the class labels. For fine-gra fine-grained recognition, localization indeed improves performance when training from few examples (see Wertheimer & Hariharan (2019) for contemporary evaluation of the role of localization for few-shot learning).
A.5 EXPERIMENTAL DETAILS
Additional optimization details on fhot learning During training. especially for jigsaw puz- zle task, we found it to be beneficial to track the running mean and variance for the batch nor- malization layer, and instead estimate them for each batch independently. We hypothesize that this is because the inputs contain both full-sized images and small patches, which might have different
14

---

Under review as a conference paper at ICLR 2020
Softmex
Softmaa Jigesw
Softmax- Rotation
Figure 5: Saliency maps for various images and models. For each image visualize the magni- tude of the gradient with respect to the correct class for models trained with various loss functions. The magnitudes are scaled to the same range for easier visualization. The models trained with self- supervision often have lower energy on the background regions when there is clutter. We highlight few examples with blue borders and the bounding- box of the object for each image is shown in red.
statistics. At test time we do the same. We found the accuracy goes up as the batch size increases but saturates size of 64.
When training with supervised and self-supervised loss, a trade-off term À between the losses can be used, thus the total loss is L (1 X)C, AL.s. We find that simply use À 0.5 works the best, except for training on mini- and tiered-ImageNet with jigsaw loss, where we set À = 0.3. We suspect that this is because the variation of the image size and the categories are higher, making the self-supervision harder to train with limited data. When both jigsaw and rotation losses are used, we set À 0.5 and the two self-supervised losses are averaged for L.s-
For training meta-learners, we use 16 query images per class for each training episode. When only 20% of labeled data are used, 5 query images per class are used. For MAML, we use 10 query images and the approximation method for backpropagation as proposed in Chen al. (2019) to reduce the GPU memory usage. When training with self-supervised loss, added when computing the loss in the outer loop. We use PyTorch (Paszke et al., 2017) for our experiments.
For the domain classifier, we first obtain features from the penultimate- llyyer (2048 dimensional) from ResNet101 model pretrained ImageNet. We then train binary logistic regression model with weight decay using LBFGS for 1000 iterations. The images from labeled dataset are the posi- tive class and images from pool the negative. loss for the positive class is scaled by the inverse of its frequency to account for the significantly larger number of negative examples.
Optimization details for standard classification For standard classification (Appendix A.3) train a ResNet18 network from scratch. All the models are trained with ADAM optimizer with a learning rate of 0.001 for 600 epochs with batch size of 16. We track the running statisties for the batch normalization layer for the softmax baselines following the conventional setting, i.e. w/o self-supervised loss, but do not track these statistics when training with self-supervision.
Architectures for self-supervised tasks For jigsaw puzzle task, we follow the architecture of Noroozi & Favaro (2016) where it was first proposed. The ResNet18 results in 512-dimension
15

---

Under review as a conference paper at ICLR 2020
feature for each imput, and add fully-connected (fc) layer with 512-units top. The nine patches gives nine 512-dimensional feature vectors, which are concatenated. This followed by fc layer, projecting the feature vector from 4608 to 4096 dimensions, and a fc layer with 35- dimensional outputs corresponding to the 35 permutations for the jigsaw task.
For the rotation task, the 512-dimensional output of ResNet18 is passed through three fc layers with {512, 128, 128, 4} units. The predictions correspond to the four rotation angles. Between each fo layer, ReLU activation and dropout layer with dropout probability of 0.5 added.
A.6 DETAILS THE DATASET SELECTED FROM POOL
Table describes the composition of the unlabeled images sampled from a generic pool for SSL experiments presented Section 4.2 (Figure 4c). "Random sampling" uniformly samples images proportional to the size of each dataset, while the "domain weighted" one tends to pick more images from related domains. This is also indicated by the domain distance of the sampled images with respect to the source domain. The improvements are larger when the domain distance is small, e.g., for birds, dogs, and flowers.
Labeled dataset
20% Birds
20% Cars
20% Aircrafts
20% Dogs
20% Flowers
Dataset
Selection method
random
weight random
weight random weight random weight
random
weight
Birds
1177
1177
325
87
184
114
410
118
156
Cars
303
1632
1632
242
466
532
594
230
12
Aircrafts
214
3
259
259
1000
1000
356
285
129
Dogs
413
19
554
657
334
251
2067
2067
283
91
Flowers
167
230
162
140
40
290
145
825
825
iNat aves
1133
4176
1599
792
957
815
1997
744
767
119
iNat others
1089
108
1546
1555
935
240
2048
964
793
2533
mini-ImageNee
1389
397
2017
3018
1208
2074
2637
5420
946
540
Domain distance-
6.69
0.94
11.42
9.69
13.41
11.13
8.72
3.68
13.38
4.57
Accuracy (%)
75.7
78.5
79.2
80.8
79.7
79.2
66.6
74.6
83.4
85.6
Table 6: Composition of the unlabeled datasets selected from a pool of images used in Figure 4c. We list the number of images in each source datasets. We also provide domain distance between the selected images and the labeled dataset. The 5-way 5-shot accuracy of the models trained with SSL are shown in the last row. The performance correlates with domain distance: when the distribution of unlabeled data is closer to labeled data, the performance is better.
16


Reviews:
[{"review_id": "rygzcR7qYB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary & Pros\n- This paper proposes a few-shot learning method that uses self-supervision as an auxiliary label and trains primary and auxiliary labels via multi-task learning.\n- This paper provides extensive experiments for analyzing the effect of self-supervision on various few-shot learning settings: (1) self-supervision can improve various few-shot learning algorithms, ProtoNet & MAML; (2) self-supervision with similar samples can provide more improvements.\n\nConcerns #1: Novelty of the proposed method\n- This paper uses a multi-task learning approach with self-supervision. But this approach is already used in various tasks, e.g., domain adaptation, semi-supervised learning, training GANs. Thus, the proposed method (in Section 3) using a multi-task learning objective with self-supervised losses seems to be incremental.\n\nConcerns #2: Somewhat unsurprising experimental results\n- This paper shows various experimental results, but some experiments seem to be trivial. For example, the performance gap is typically increased when learning harder tasks, e.g., when the number of training samples is decreasing, the performance gap between methods is typically increasing in a fully-supervised setting. Thus I think results in the paragraph \"Gains are larger for harder tasks\" might be predictable. Other examples are Figure 4a and 4b in Section 4.2 because one can easily expect that using training more in-domain samples can provide more performance gain.\n- In the case of Figure 4d in Section 4.2, the authors claimed that the effectiveness of SSL decreases as the distance from the supervised domain increases. However, I think Figure 4d is not matched to the claim. For example, in the case of Dogs, better performance is achieved when using a more dissimilar domain for self-supervision except for D_s=D_ss. So I wonder how to draw the lines in Figure 4d.\n\nSome experimental results provide meaningful messages, e.g., single self-supervision can improve performance significantly while joint self-supervision does marginally. However, the contribution of the methodology is limited and some experimental results seem to incremental.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1eEEC9lqS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper considers the addition of self-supervised learning techniques in the few-shot learning setting. Extensive experiments are done to show that it can be helpful, including in cases where the labeled data is corrupted. The paper also considers the domain mismatch issue where unlabeled images come from a different domain.\n\nReview: This paper is thorough and clearly written. Applying self-supervised learning techniques to the few-shot learning regime is a simple idea, and this paper clearly shows that it can be beneficial. It includes extensive experiments on a wide variety of image datasets, including many additional studies in the appendix. The only possible criticisms of this paper are that it is limited to the image domain (as are most few-shot learning/self-supervised learning studies, so we can probably ignore this) and that it does not produce a huge delta in understanding compared to Gidaris et al. (2019). Gidaris et al. (2019) was posted to arxiv in June, I'm not sure if this work was also on arxiv around the same time, and even if it wasn't I'm not sure what to consider \"concurrent\". However, as the authors note they include additional experimental settings (like the domain-selection idea) that are not in Gidaris et al. (2019), so the works are somewhat complementary. The only other comment I have is that the paper is quite large in scope for a conference submission and as a result there are many details and experiments that are left for the appendix. I could also see the domain selection experiments constituting their own submission. For example the definition of the \"'distance' between a pair of domains\" is only introduced in passing in the midst of Section 4.2 covering domain shift experiments, and the method for training a domain classifier is similarly only mentioned in passing in 4.2.  Of course, it is not really valid to criticize a paper for being too exhaustive. Overall, I recommend acceptance.\n\nSpecific comments:\n- Truly a minor suggestion but I suggest moving Figure 1 to the top of page 2.\n- Snell et al. (2017) needs a \\citep\n- You ought to cite \"S4L: Self-Supervised Semi-Supervised Learning\", which is related to your discussion of connections between self- and semi-supervised learning (though not few-shot).\n- The paragraph beginning \"The focus of most prior work...\" in the Related Work section provides a nice framing of your work and so might make more sense in the introduction.\n- \"we consider self-supervised losses based on labeled data ... that can be derived from inputs x alone\" All labels (can be derived from in the inputs x alone, given an oracle (or human labeler). I think you mean \"that can be derived automatically without any human labeling\".\n- You state \"Our final loss function combines the two losses\". Is there no scalar multiplier on either loss term to trade-off the importance of each?\n- The gains from the self-supervised auxiliary tasks are over and above any gains from data augmentation alone. More experimental details are in Appendix A.5.\" I don't see any information to substantiate the claim that the self-supervised tasks result in a bigger improvement than data augmentation alone, can you provide those details?\n- \"However, does more unlabeled data always help for a task in hand?\" This question actually was addressed somewhat in \"Realistic Evaluation of Semi-Supervised Learning Algorithms\", see sections 4.4 and 4.5 therein.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "H1gxFLNtYS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThere are three main contributions of the paper:\ni) The authors present an empirical study of different self-supervised learning (SSL) methods in the context of self-supervised learning.\nii) They point out how SSL helps more when the dataset is harder.\niii) They point out how domain matters while using SSL for training and present a method to choose samples from an unlabeled dataset.\n\nStrengths:\n1) They confirm the results of [7] and provide additional evidence of the benefit of the self-supervised learning in the few-shot setting.  They also showcase an interesting new result that self-supervised learning helps more in case of harder problems. \n2) The authors have a done a commendable job of coming up with a meaningful set of experiments by varying base-models, self-supervised methods, datasets, and few-shot learning methods in Section 4.1. This is quite a comprehensive study. \n3)  The paper is well-written and well-motivated.\n\nWeaknesses:\n1) The main weakness of the paper is Section 4.2's experimental setup. \n\ni) The definition of domain distance in not quite meaningful. Since the chosen datasets have very different classes (airplanes vs dogs etc), the average embeddings for different datasets/classes will be far from each other.  In Figure 4d, it is misleading to show a trendline that includes the same domain as that will always be 0. If that datapoint is removed the trend line is mostly flat. The authors want to present a quantifiable way to show how domain distribution affect performance on self-supervised learning methods. But this definition of domain distance is more meaningful in the domain adaptation setting (like Amazon-Office dataset used for domain adaptation (https://people.eecs.berkeley.edu/~jhoffman/domainadapt/)) as in that case the requirement is to get the embeddings close to each other for the same class but from different domains.\n\nii) The authors go on to create an \"unlabeled pool\" by combining images from many domains. Then they train a domain classifier by labeling in-domain images as positive and  labeling the images in pool as negative. Considering all images in the pool as negative is not correct as there can be images of same class in unlabeled pool. \n\niii) Then they choose the samples which the classifier predicts comes from the same domain. This probably succeeds as it is done on top of ResNet-101 features (that has seen all of ImageNet). This technique probably works possibly due to the ResNet being pre-trained with so many labeled classes. One baseline might have just been to choose the k-nearest neighbors (from unlabled pool)  to the average embedding of all the images of the chosen dataset. Since, it is quite easy for classifiers to detect from which dataset an image came from [6] it might be easy to choose an image from similar domain by using nearest neighbor in the embedding space. \n\nI am not sure how well their heuristic will work for an unlabeled pool that the classifier has never seen. Additionally, a portion of the dataset has been created by combining existing datasets. Since statistics of different datasets vary a lot artificially, the creation of an unlabeled pool by combining different datasets might work in the favor of the proposed heuristic of looking at \"domain distance\" to choose the samples.\n\n2) Effect of domain shift in SSL (Figure 4b) is studying an extreme case where the domain shift results in almost no common classes in the SSL training phase. It would make more sense to include datasets where at least some of the classes are shared so that self-supervised learning methods get to see some relevant classes. In practice, a self-supervised learning method would be applied on a large unlabeled pool of images. Hopefully with increasing diversity and number of images, there might be some images on which ding SSL helps the downstream few-shot task. Hence comparison with such a dataset like ImageNet/iNatrualist is important here.\n\nDecision:\nThe paper presents an well thought-out empirical study on self-supervised learning for few-shot learning. But there are  major concerns with the empirical setup and methods presented in the section where they propose a method to choose samples for SSL from an unlabeled pool of images. \n\n\nMinor Comments:\n1)  “In contrast, we humans can quickly learn new concepts from limited training data” - Remove we. \n2) “Despite recent advances, these techniques have only been applied to a few domains (e.g., entry-level classes on internet imagery), and under the assumption that large amounts of unlabeled images are available.” This is not true. Self-supervised learning methods have been used for continuous control in reinforcement learning[1, 2], cross-modal learning[3], navigation [5], action recognition[4] etc. Later on in related work the authors list many papers that use self-supervised learning in different contexts. This line should be modified to reflect how common self-supervised learning methods are in other fields as well and with less data.\n3) “making the rotation task too hard or too trivial to benefit main task” - add respectively.\n4) \"With random sampling, the extra unlabeled data often hurts the performance, while those sampled using the “domain weighs” improves performance on most datasets.\" replace weighs with weights\n5) [8] points out how self-supervised learning on a large dataset but with a domain shift (YFCC100M) is not as effective for pre-training as it is doing self-supervised learning on the downstream task's dataset (ImageNet). While it is a different setting it is inline with one of the main conclusions of the paper.\n\nReferences:\n[1] “PVEs: Position-Velocity Encoders for Unsupervised Learning of Structured State Representations” Rico Jonschkowski, Roland Hafner, Jonathan Scholz, and Martin Riedmiller.\n[2] “Learning Actionable Representations from Visual Observations” Debidatta Dwibedi, Jonathan Tompson, Corey Lynch, Pierre Sermanet\n[3] “Look, Listen and Learn” Relja Arandjelović, Andrew Zisserman\n[4] “Self-supervised Spatiotemporal Learning via Video Clip Order Prediction” Dejing Xu, Jun Xiao, Zhou Zhao, Jian Shao, Di Xie, Yueting Zhuang.\n[5] “Scaling and Benchmarking Self-Supervised Visual Representation Learning” Priya Goyal, Dhruv Mahajan, Abhinav Gupta, Ishan Misra\n[6] \"Unbiased Look at Dataset Bias\" Antonio Torralba and Alexei A. Efros.\n[7] \"Boosting ´few-shot visual learning with self-supervision.\" Spyros Gidaris, Andrei Bursuc, Nikos Komodakis, Patrick Perez, and Matthieu Cord. \n[8] \"Deep clustering for unsupervised learning of visual features\" Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers agree that this paper contains interesting results and simple, but good ideas. However, a few severe concerns were raised by reviewers. Most prominent one was the experiment set up - authors use a pre trained ResNet101 (which has seen many classes of Imagenet) for testing which makes is unclear how well their proposed method would work for unlabeled pool of dataset that classifiers has never seen.
 While authors claim that their the dataset used for testing was disjoint from Imagenet, a reviewer pointed out that dogs dataset, bird datasets both state that they overlap with Imagenet. A few other concerns are raised (need more meaningful metric in Figure 4d, which wasn’t addressed in rebuttal). We look forward to seeing an improved version of the paper in your future submissions. 
Decision: Reject