arXiv:1910.01319v1 [cs.LG] 3 Oct 2019

An empirical study of pretrained representations forfew-shot classification
Tiago Ramalho, Thierry Sousbie, Stefano Peluchetti
Cogent Labs
Tokyo, Japan{tramalho, tsousbie, speluchetti}@cogent.co.jp

Abstract
Recent algorithms with state-of-the-art few-shot classification results start theirprocedure by computing data features output by a large pretrained model. In thispaper we systematically investigate which models provide the best representations
for a few-shot image classification task when pretrained on the Imagenet dataset.
We test their representations when used as the starting point for different fewshot classification algorithms. We observe that models trained on a supervisedclassification task have higher performance than models trained in an unsupervisedmanner even when transferred to out-of-distribution datasets. Models trained withadversarial robustness transfer better, while having slightly lower accuracy thansupervised models.

1

Introduction

Deep learning systems achieve remarkable performance for several classification problems when
given large enough datasets [15, 33]. In the case of small amounts of training data, these models caneasily overfit as they contain a very large number of parameters [3]. There are two main techniques
to address these issues: if we have a medium-sized dataset, we can fine-tune the weights of a modeltrained on a larger dataset [12]; for very small datasets we rely on methods which learn from the’training’ dataset without multiple steps of gradient descent over all parameters in the model, known
as meta-learning or few-shot classification [26].
Just as in the fine-tuning regime, in the few-shot classification case we also want to leverage large
datasets for better final performance. This is almost always achieved by feeding the data representation
produced by a deep neural network as input to the algorithm. For example, in the case of image
classification practitioners will use a deep residual convolutional network pre-trained on a larger
dataset to compute the features [31, 32].
Intuitively, if the representations produced by the pre-trained model are more discriminative of thedifferent classes under consideration at test time, it will be easier for a few-shot classification method
to produce better results [31]. Previous work has shown that deeper convolutional models have higher
accuracy and their feature quality appears to be the limiting factor in performance of most few-shotclassification algorithms [4].
In this paper we will perform a systematic exploration of whether deep convolutional networks
pretrained on the ImageNet dataset without few-shot classification in mind can transfer well to thistask. Previous work has done a similar exploration for the case of fine-tuning [12], but systematic
reviews of few-shot classification methods have not looked at the transfer properties of pretrainedmodels, instead focusing on the adaptation algorithm or architectural choices [4, 35].
After evaluating multiple models for 14 different datasets, we have come to the following conclusions:
Preprint. Under review.

• Models pre-trained on a supervised classification task on the ImageNet dataset transfer wellto other natural image datasets, but poorly to other types of image (e.g. MNIST, SVHN).
The more data models are trained on, the better their transfer performance.
• Models trained with adversarial robustness [38, 11] suffer a smaller performance decrease
when transferred to out-of-distribution datasets. Their absolute performance is slightly lower
than non-robust models.
• Models trained with an unsupervised learning loss [10, 2] do not reach the performance ofmodels trained in a supervised manner, and do not seem to transfer better.
• Unlike what was reported in [32] for simpler architectures trained from scratch, similaritybased few-shot classification methods used with pretrained models perform best when thesimilarity function is based on cosine similarity.

2

Related work

The earliest research in the field of few-shot classification focused on embedding space similarity
based methods such as Matching [36] and Prototype networks [32]. These models are trained on
permutations of the meta-training dataset for a number of epochs, and then tested on unseen classes
from the same distribution. Similarity based methods can be augmented with a decoder on top of theretrieved features to provide more flexibility in cases where the required output isn’t a direct classmapping [8, 25].
MAML is a gradient-descent based method for metalearning [7], where the network is trained suchthat its weights converge to a configuration from which the model can be finetuned in very few stepsof gradient descent on the support set. Recent work has demonstrated state of the art results forgradient-based methods [31]. In addition, MAML and related models have been shown to work
particularly well for reinforcement learning scenarios [21].
A third way of doing meta learning is to train a recurrent neural network to iterate over the whole
support set and the new query point [28, 20]. These methods require fewer assumptions on the
structure of the activation space, but are more computationally intensive.
Most literature on metalearning tends to use the same distribution for the meta training set and thetesting set, only holding out specific images or specific classes from training. This is a weaker formof generalization than cross-distribution generalization, which has recently been tested in [35], andis the setting in which we are interested. The authors of this paper show that the performance of
all metalearning methods degrades significantly when tested on a distribution significantly different
to the one of the meta training set; and that a model which has been meta-trained on a number ofdifferent datasets generalizes better.
The authors of [12] study transfer learning in the setting of models pretrained on imagenet. Theyshow that models with better final accuracy on imagenet classification tend to have better transfer
learning performance, which is encouraging for their application in few-shot classification.
The authors of Meta-Dataset explore the performance of few-shot classification methods whentransferred across datasets [35]. The authors survey a large number of state-of-the-art adaptation
methods across a large variety of datasets with a convolutional backbone trained from scratch. They
conclude that a model trained on a large variety of datasets has better transfer performance than amodel trained exclusively on ImageNet. Their comparison across multiple adaptation methods leads
us to conclude that quality of the features produced by a backbone affects performance much morestrongly than the particular adaptation method. Unlike our work, they do not consider pretrainedmodels and instead train a comparatively shallow backbone (ResNet-18) from scratch.
In [4] we find another survey of few-shot classification methods. Here the authors measure the effectof convolutional backbone depth as well as classification algorithm on final performance. In thisstudy only two datasets are considered, CUB and mini-ImageNet. As before, the authors find thatdeeper backbones provide better performance, but again restrict their analysis to shallow models
trained from scratch.
Given the thorough analyses in the two papers mentioned above, we are encouraged to study the useof very deep convolutional networks trained on large datasets as feature extractors, as that appears tobe the main bottleneck in current few-shot classification methods.
2ModelTraining method

Dataset#params

Short name

ResNet-50 [9]Supervised

ILSVRC2012

24M

resnet50

EfficientNet-B0 [34]EfficientNet-B7 [34]Supervised

ILSVRC2012

5.3M

efficientB0

Supervised

ILSVRC2012

66M

efficientB7

WSL(resnext101_32x8d_wsl)[18]Supervised

ILSVRC2012 + Instagram (950M images)

88M

wsl

Robust-50 [6]Supervised + Adv. RobustnessILSVRC2012

24M

robust50

Denoising Resnet [38]Supervised + Adv. RobustnessILSVRC2012

70M

denoise

Local DIM [2]Unsupervised

ILSVRC2012

431M

amdim

Table 1: Pretrained deep convolutional networks used as feature extractors considered in our analysis.

3

Few-shot classification methods

The few-shot classification problem consists of a set of datasets D = {d0 , d1 , ...} where dn is anindividual dataset with image and target label pairs dn = {(x0 , y0 ), (x1 , y1 ), ...}. We consider eachdataset d split into two parts: a support set s = {(xs , ys )} and a query set q = {(xq , yq )}. The model
can access both data and labels for all examples in s and is asked to predict labels for xq ∈ q. The
model is trained to minimize the cross-entropy loss:
L = Ed∼D −X

yq log (fθ,s (xq )) ,(1)(xq ,yq )∈q

where fθ,s (x) stands for a parametric model which outputs the vector of inferred class probabilities.
Following established nomenclature, we define k-shot, N -way classification learning as the casewhere the support set s contains N classes, and for each class we have k observations.
Following [31, 29], we decompose the meta-classification model into two modules: the representation
network φ, and the adaptation method a such that fθ,s (x) = aθ (φ(xq ), {φ(xs ), ys }). In the following
subsections we detail the choices made for φ and a in this study.
3.1

Pretrained models

To extract representations from an image we use different convolutional network backbones, allbased on the residual network architecture introduced in [9]. These models consist of a stack of
convolutional layers followed by a single linear layer applied to a final, flattened activation vector.
For all our experiments we take this pre-logits activation vector (which we denote as φ(x)) as therepresentation to be fed to the adaptation method.
We consider a number of recent architectures trained on the Imagenet ILSVRC2012 dataset [30]. For
models trained on the standard supervised classification task we consider the baseline ResNet-50 [9],as well as the highest and lowest parameter count variants of EfficientNets [34]. Additionally weconsider the WSL model [18], which is pretrained on an image dataset with billions of images withlow quality labels before full finetuning on Imagenet. We also analyse two ResNet architectures
trained with adversarial robustness [38, 6] to investigate whether such representations generalize
better. Finally we test the Local DIM model [2], trained in a fully unsupervised manner on ImageNet.
3.2

Adaptation methods

We focus our analysis on three simple popular adaptation methods: Matching networks [36], Prototype
networks [32], and Logistic regression via SGD.

3

In prototype networks the mean vector of the activations corresponding to all support examples ofthat class is used as a ’prototype’ for that class. Class conditional probabilities are calculated as a thesoftmax over the similarity of a test point’s activation to all ’prototypes’. Concretely, a prototype isdefined as the mean vector:
1 X
φ(s x)µc =#Xc
xs ∈Xcwhere Xc is the set of support set inputs with label corresponding to class c. This is equivalent
to assuming the activations for class c follow a Gaussian distribution with mean µc and identity
covariance matrix.
Assuming the labels yc of inputs belonging to class c are represented by a one hot vector, theunnormalized logits ` for a query input xq are given by a similarity measure between ai (xq ) and thecentroids of each class:` (xq ) =NX

S(φ(xq ), µc ) yc

c=1

where the similarity S(·, ·) can be for instance the cosine similarity, the inner product, or the negativeof a distance function such as the Euclidean distance.
In matching networks [36] the activations for all examples in the support set are kept in memory. Fora test point we query the K-nearest neighbors in activation to compute a kernel density estimate of
the class conditional probabilities. The unnormalized logits are given by:` (xq ) =X

S (φ(xq ), φ(x)) y,(x,y)∈NK (xq )where NK (xq ) is the set of input and label pairs corresponding to the K nearest neighbors of xq .
One goal of our analysis is to evaluate whether the topology of the representation space impacts fewshot classification: if the representations are clustered in multiple disjoint volumes in representation
space then matching networks should perform better than prototype networks, as the prototypes canonly represent one single connected region of space.
We also examine the performance of simple logistic regression on the support set’s representations.
For a large number of shots, we expect this method to provide an upper bound on performance as
has more flexibility than similarity-based methods. For low number of shots, this method has an
infinite number of solutions and therefore SGD defines an implicit form of generalization. For theexperiments we use logistic regression implementation in scikit-learn [24] with all regularization
terms set to zero.

4

Experiments

We evaluate the performance of all methods on 14 natural image datasets: ILSVRC2012 Imagenet
validation set [30]; MNIST [17]; Omniglot [16]; VGG flowers [23]; FGVC-Aircraft [19]; Cars [13];SVHN [22]; CIFAR-10 and CIFAR-100 [14]; DTD [5]; Fungi [1]; Caltech Birds [37]; ImageNetv2 [27]; and a subset of Imagenet classes not contained in the ILSVRC2012 1000-class set (weprovide a full list of synsets in the Appendix).
For each dataset, we generate a new episode by sampling N classes (N ∈ {5, 10, 20, 50, 100}), andcreate a k-shot support set (k ∈ {1, 2, 3, 5, 10, 20}) using the features generated by the network understudy. Unless otherwise noted, we always use the last layer’s features. The accuracy of an episode iscalculated over a batch of new query datapoints (batch size 32).

4

Accuracy
Accuracy
Accuracy

1.00
0.75
0.50
0.25

1.00
0.75
0.50
0.25

1.00
0.75
0.50
0.25
0

imagenet-val

imagenet-v2

imagenet-ood

cifar10

cifar100

birds

cars

planesflowers

funghiomniglot

dtd

mnist

svhn

0

10
20
Number of shots

Architecture
amdim
denoiseefficientB0
efficientB7
resnet50
robust50
wsl

10
20 0
10
20 0
10
20 0
10
20
Number of shots
Number of shots
Number of shots
Number of shots

Figure 1: Accuracy comparison for different datasets as a function of number of shots (elements inthe support set) for 10-way classification. The accuracy value reported is the average accuracy over 25
unique episodes (where each episode is a random sample of 10 classes, from which the examples arethen randomly split into a support set with k examples and a query set with max (zc − k, 32) examples,with zc the total number of examples available in class c). Top row: we show imagenet and cifar
datasets, where the best performance is obtained by networks trained on the supervised classification
task. Middle row: more specialized natural image datasets where the performance decreases but
supervised architectures still do better. Note that better architectures and the wsl network trainedon more data perform better. Bottom row: out-of-distribution datasets. Here networks trained withadversarial robustness do better.
imagenet-val

Accuracy

1.0

imagenet-v2

imagenet-ood

cifar100

birds

0.5
0.0
omniglot

Accuracy

1.0

flowers

cars

planesfunghi0.5
0.00Architecture
amdim
denoiseefficientB0
efficientB7
resnet50
robust50
wsl

50
100 0
50
100 0
50
100 0
50
100 0
50
100
Number of classes
Number of classes
Number of classes
Number of classes
Number of classes

Figure 2: Accuracy comparison for different datasets as a function of number of classes (elements inthe support set) for 5-shot classification. The accuracy value reported is the average accuracy over
25 unique episodes (as described in the figure above). We show only results for datasets where 100classes were available.
4.1

Pretrained features comparisonFirstly we seek to quantify how the features calculated by the various pretrained backbones underconsideration affect the final performance of the few-shot classification task across a range of numberof shots and number of classes under consideration. Are certain backbones universally better thanothers? Do unsupervised models and models trained with adversarial robustness generalize better?
In Figure 1 we compare the 10-way classification performance across all architectures under consideration for different datasets as a function of number of shots (elements in the support set). To focus

5

Methodbirds
MN (dot)
PT (dot)
MN (L2)PT (L2)
MN (cosine)
PT (cosine)
SGD

cars
0.80
0.72
0.64
0.56

Methodfunghi
0.88
0.80
0.72
0.64
0.56
0.48

Method1

2 3 5 10 20
Number of shots

0.56

0.48

0.48
0.40
0.32
imagenet-ood

0.60

dtd

0.72

0.66

0.64

0.60

0.56

0.54

0.48

0.48

0.40

0.42

imagenet-v2

0.84

imagenet-val
0.80

0.54

0.78

0.48

0.72

0.75

0.42

0.66

0.70

0.36

0.60

0.65

0.30

0.54

0.60

omniglotplanes

svhn

0.72

0.88

0.64

0.80

0.56

0.72

0.48

0.64

0.40

0.56

0.32

0.32

0.48

0.24

1

2 3 5 10 20
Number of shots

0.80
0.75
0.70
0.65

0.24

0.48

0.21

0.40

2 3 5 10 20
Number of shots

2 3 5 10 20
Number of shots

0.27

0.56

1

0.90
0.85

1

mnist
MN (dot)
PT (dot)
MN (L2)PT (L2)
MN (cosine)
PT (cosine)
SGD

0.56

0.32

flowers

cifar100
0.64

0.40

0.48

MN (dot)
PT (dot)
MN (L2)PT (L2)
MN (cosine)
PT (cosine)
SGD

cifar10
0.64

0.18
0.15
1

2 3 5 10 20
Number of shots

0.12

Figure 3: Accuracy comparison for different datasets as a function of adaptation method and similarity
function for 10-way classification. The abbreviations MN, PT and SGD are respectively used forMatching Networks, Prototype Networks, and Logistic Regression with Stochastic Gradient Descent.
Light grey values represent missing data (when the dataset does not contain enough examples perclass for that number of shots). The accuracy value reported is the average accuracy over 25 uniqueepisodes (as described in Fig.1).
on the feature quality, we report the maximum accuracy achieved over all adaptation methods for thatparticular architecture and number of shots. In the next section we will break the results down by
adaptation method.
Firstly, we observe that supervised classification architectures all do very well on the three datasets
derived from imagenet, as would be expected since there is little distributional shift. As expected fromthe results in [27], there is a small drop in performance for ImageNet-v2, and a larger performancedrop for the ImageNet dataset with unseen classes. Performance on the cifar datasets is also high, inspite of a very different image resolution (we upscale all images where the resolution does not matchthe original model’s input resolution).
The same architectures also perform well in various natural image based datasets such as birds, cars,
flowers. For these datasets we empirically observe that architectures with better final performance on
the classification task perform better. In particular the wsl network, pre-trained on much more data,
stands out.
Out-of-distribution datasets such as omniglot, dtd, mnist and svhn present a different picture. Not
only does the performance degrade, networks trained with adversarial robustness do better than their
non-robust counterparts. The architecture trained in a fully unsupervised manner does not generalize
better, in spite of its features not having been tuned in a class-specific manner by a supervised loss.
In Figure 2 we plot the same quantity as above, but now as a function of the number of classes fora fixed number of shots (5 shot per class). We can confirm with this plot that the relative ranking
between models stays roughly constant as they degrade in performance with increasing number ofclasses. In fact, performance for a low number of classes seems to be predictive of final performancefor high number of classes across architectures and datasets, with higher drops for lower initialaccuracy numbers.

6

4.2

Adaptation method comparison

Next we quantify the effect of the adaptation method on the final performance of few-shot classification when used with pre-trained models. Since pretrained models were not trained with few-shotclassification in mind, conclusions reached by previous studies [4, 35, 32] may not hold true.
Specifically we are interested in the following questions: if a model is trained in a supervised mannerwith a final softmax layer, the representations have no reason to cluster together in the sense of
the euclidean distance, as what matters to produce discriminative logits is the dot product of therepresentation vector with a given entry in the weights matrix of the last linear layer. Therefore, theaveraging done by prototype networks could be detrimental, and the use of L2 distance might not be
optimal in terms of determining proximity in this space.
In Figure 3 we compare the accuracy different datasets as a function of adaptation method and similarity function for 10-way classification. The accuracy is reported as an average over all architectures
under consideration. From these heatmaps we can observe that contrary to our expectation Prototype
Networks seems to consistently beat Matching Networks. On the other hand we do find that cosine
similarity is consistently superior to L2 distance as we expected. This is different to the results
originally reported in Prototype Networks, where L2 worked better for a convolutional backbone
pretrained from scratch.
Furthermore we observe that the unnormalized dot product tend to perform worse than the othersimilarity measures. We hypothesise this is due to the fact that there are no learnable parameters inthese adaptation methods and therefore an unnormalized method suffers if there are scale differences
between the average features of different classes. We leave this investigation to futre work.
Finally we observe that Logistic regression with SGD is the best performer for 10 shots and above,
while being significantly worse than other adaptation methods for a small number of shots. Thisagrees with the intuition of the early papers introducing few-shot classification [36], which propose
their methods only in the 5-shot and below regime.
This suggests that at least in the case of pretrained models there is a clear hierarchy of which methods
are best as a function of number of shots: for 1-5 shots, methods such as Prototype and Matching
networks will work best; for over 10 shots, retraining of the last layer will do better; and finally forhigh number of examples per class fine-tuning becomes a viable option [12].
In Figure 4 we break down the accuracy per pre-trained architecture. We find that there are no
significant differences to the analysis described above, with Prototype Networks with cosine similarity
generally performing best among similarity-based methods. Unnormalized dot product still exhibits
the worst performance, although with EfficientNets its performance drop is not so significant. Overall
the most robust architecture across all datasets appears to be efficientB7, while the individual
highest performer for the best hyperparameter choices is wsl.

5

Conclusions

In this paper we systematically investigate the performance of pretrained models as backbones tocalculate feature representations for few-shot image classification tasks. We tested their accuracy
when used as starting point for popular similarity-based few-shot classification methods across a
range of datasets.
Encouragingly we find that these models can provide very high accuracy for a number of few-shotclassification tasks. We found that models pre-trained on a supervised classification task on the
ImageNet dataset transfer well to other natural image datasets, but poorly to other types of image(e.g. MNIST, SVHN). The more data models are trained on, the better their transfer performance.
Contrary to our expectation, models trained with adversarial robustness or trained with an unsupervised loss do not seem to outperform models trained on the popular 1000-class ILSVRC2012
ImageNet classification task. However, the models tested here do not achieve the same performanceon the original ILSVRC2012 test set as the purely supervised models, so their performance should be
revisited when the field matures and performance improves.

7

Methodamdim
MN (dot)
PT (dot)
MN (L2)PT (L2)
MN (cosine)
PT (cosine)
SGD

Methodresnet50
MN (dot)
PT (dot)
MN (L2)PT (L2)
MN (cosine)
PT (cosine)SGD
1

2 3 5 10 20
Number of shotsdenoise0.8

efficientB0

0.8

efficientB7

0.8

0.8

0.7

0.7

0.7

0.7

0.6

0.6

0.6

0.6

0.5

0.5

0.5

0.5

0.4

0.4

0.4

0.4

0.3

0.3

0.3

robust50

0.8

wsl

0.8

0.8

0.7

0.7

0.7

0.6

0.6

0.6

0.5

0.5

0.5

0.4

0.4

0.4

0.3

0.3

1

2 3 5 10 20
Number of shots

1

2 3 5 10 20
Number of shots

1

2 3 5 10 20
Number of shots

0.3

0.3

Figure 4: Accuracy comparison for different pretrained backbones as a function of adaptation method
and similarity function for 10-way classification. The abbreviations MN, PT and SGD are respectively
used for Matching Networks, Prototype Networks, and Logistic Regression with Stochastic Gradient
Descent. The accuracy value reported is the average accuracy over all datasets.
We also found that some best practices on few-shot classification do not transfer to the use of pretrained models (e.g. we find that the cosine similarity provides better performance than L2 distance,
contrary to what is the case with end-to-end trained few-shot classification models).
We hope our empirical investigation will spur the use of pretrained models in applied few-shotclassification and online learning tasks, as they can provide excellent performance with very little
training costs. All convolutional backbones we experimented with have publicly available weights
and code implementations and we thank all the respective authors for releasing their code forexperimentation.

References[1] Schroeder B. and Y. Cui. Fgvcx fungi classification challenge 2018. 2018.
[2] Philip Bachman, R. Devon Hjelm, and William Buchwalter. Learning Representations by
Maximizing Mutual Information Across Views. arXiv:1906.00910 [cs, stat], June 2019. arXiv:
1906.00910.
[3] Yoshua Bengio. Deep learning of representations for unsupervised and transfer learning. In
Proceedings of ICML workshop on unsupervised and transfer learning, pages 17–36, 2012.
[4] Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A
Closer Look at Few-shot Classification. arXiv:1904.04232 [cs], April 2019. arXiv: 1904.04232.
[5] M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, , and A. Vedaldi. Describing textures in the wild.
In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.
[6] Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Brandon Tran, andAleksander Madry. Learning Perceptually-Aligned Representations via Adversarial Robustness.
arXiv:1906.00945 [cs, stat], June 2019. arXiv: 1906.00945.
[7] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-Agnostic Meta-Learning for Fast
Adaptation of Deep Networks. arXiv:1703.03400 [cs], March 2017. arXiv: 1703.03400.
[8] Marta Garnelo, Dan Rosenbaum, Chris J. Maddison, Tiago Ramalho, David Saxton, Murray
Shanahan, Yee Whye Teh, Danilo J. Rezende, and S. M. Ali Eslami. Conditional Neural
Processes. arXiv:1807.01613 [cs, stat], July 2018. arXiv: 1807.01613.
[9] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image
Recognition. arXiv:1512.03385 [cs], December 2015. arXiv: 1512.03385.
[10] Olivier J. Hénaff, Ali Razavi, Carl Doersch, S. M. Ali Eslami, and Aaron van den Oord. DataEfficient Image Recognition with Contrastive Predictive Coding. arXiv:1905.09272 [cs], May
2019. arXiv: 1905.09272.
8

[11] Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, andAleksander Madry. Adversarial Examples Are Not Bugs, They Are Features. arXiv:1905.02175[cs, stat], May 2019. arXiv: 1905.02175.
[12] Simon Kornblith, Jonathon Shlens, and Quoc V. Le. Do Better ImageNet Models Transfer
Better? arXiv:1805.08974 [cs, stat], May 2018. arXiv: 1805.08974.
[13] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations forfine-grained categorization. In 4th International IEEE Workshop on 3D Representation and
Recognition (3dRR-13), Sydney, Australia, 2013.
[14] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 and cifar-100 datasets. URl:
https://www. cs. toronto. edu/kriz/cifar. html, 6, 2009.
[15] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep
convolutional neural networks. In Advances in neural information processing systems, pages
1097–1105, 2012.
[16] Brenden M. Lake, Ruslan Salakhutdinov, and Joshua B. Tenenbaum. Human-level concept
learning through probabilistic program induction. Science, 350(6266):1332–1338, December
2015.
[17] Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mnist/.
[18] Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li,
Ashwin Bharambe, and Laurens van der Maaten. Exploring the Limits of Weakly Supervised
Pretraining. arXiv:1805.00932 [cs], May 2018. arXiv: 1805.00932.
[19] S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classificationof aircraft. Technical report, 2013.
[20] Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A Simple Neural Attentive
Meta-Learner. July 2017.
[21] Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S. Fearing, Pieter Abbeel, Sergey
Levine, and Chelsea Finn. Learning to Adapt in Dynamic, Real-World Environments ThroughMeta-Reinforcement Learning. arXiv:1803.11347 [cs, stat], March 2018. arXiv: 1803.11347.
[22] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng.
Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on
Deep Learning and Unsupervised Feature Learning 2011, 2011.
[23] M-E. Nilsback and A. Zisserman. A visual vocabulary for flower classification. In Proceedingsof the IEEE Conference on Computer Vision and Pattern Recognition, volume 2, pages 1447–
1454, 2006.
[24] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel,
P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher,
M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine
Learning Research, 12:2825–2830, 2011.
[25] Tiago Ramalho and Marta Garnelo. Adaptive Posterior Learning: few-shot learning with asurprise-based memory module. arXiv preprint arXiv:1902.02527, 2019.
[26] Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. 2016.
[27] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet
classifiers generalize to imagenet? arXiv preprint arXiv:1902.10811, 2019.
[28] Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B. Tenenbaum, Hugo Larochelle, and Richard S. Zemel. Meta-Learning for Semi-Supervised Few-Shot
Classification. arXiv:1803.00676 [cs, stat], March 2018. arXiv: 1803.00676.
[29] James Requeima, Jonathan Gordon, John Bronskill, Sebastian Nowozin, and Richard E. Turner.
Fast and Flexible Multi-Task Classification Using Conditional Neural Adaptive Processes.
arXiv:1906.07697 [cs, stat], June 2019. arXiv: 1906.07697.
[30] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng
Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei.
ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision(IJCV), 115(3):211–252, 2015.

9[31] Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-Learning with Latent Embedding Optimization. arXiv:1807.05960[cs, stat], July 2018. arXiv: 1807.05960.
[32] Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical Networks for Few-shot Learning.
arXiv:1703.05175 [cs, stat], March 2017. arXiv: 1703.05175.
[33] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting Unreasonable
Effectiveness of Data in Deep Learning Era. arXiv:1707.02968 [cs], July 2017. arXiv:
1707.02968.
[34] Mingxing Tan and Quoc V. Le. EfficientNet: Rethinking Model Scaling for Convolutional
Neural Networks. arXiv:1905.11946 [cs, stat], May 2019. arXiv: 1905.11946.
[35] Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Kelvin Xu, Ross Goroshin,
Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle. Meta-Dataset:
A Dataset of Datasets for Learning to Learn from Few Examples. arXiv:1903.03096 [cs, stat],March 2019. arXiv: 1903.03096.
[36] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra.
Matching Networks for One Shot Learning. arXiv:1606.04080 [cs, stat], June 2016. arXiv:
1606.04080.
[37] P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-UCSD
Birds 200. Technical Report CNS-TR-2010-001, California Institute of Technology, 2010.
[38] Cihang Xie, Yuxin Wu, Laurens van der Maaten, Alan Yuille, and Kaiming He. FeatureDenoising for Improving Adversarial Robustness. arXiv:1812.03411 [cs], December 2018.
arXiv: 1812.03411.

10