A Baseline for Few-Shot Image Classification
Authors: Guneet Singh Dhillon
Pratik Chaudhari
Avinash Ravichandran
Stefano Soatto
Venue:  (2020)
Keywords: few-shot learning
transductive learning
fine-tuning
baseline
meta-learning

Abstract: Fine-tuning a deep network trained with the standard cross-entropy loss is a strong baseline for few-shot learning. When fine-tuned transductively, this outperforms the current state-of-the-art on standard datasets such as Mini-ImageNet, Tiered-ImageNet, CIFAR-FS and FC-100 with the same hyper-parameters. The simplicity of this approach enables us to demonstrate the first few-shot learning results on the ImageNet-21k dataset. We find that using a large number of meta-training classes results in high few-shot accuracies even for a large number of few-shot classes. We do not advocate our approach as the solution for few-shot learning, but simply use the results to highlight limitations of current benchmarks and few-shot protocols. We perform extensive studies on benchmark datasets to propose a metric that quantifies the "hardness" of a few-shot episode. This metric can be used to report the performance of few-shot algorithms in a more systematic way.

Published as a conference paper at ICLR 2020
A BASELINE FOR FEW-SHOT IMAGE CLASSIFICATION
Guneet S. Dhillon', Pratik Chaudhari Avinash Ravichandran', Stefano Soatto 1,3 Amazon Web Services, University of Pennsylvania, UUiieessty of California, Los Angeles (guneetsd, ravinash, soattos) @amazon.com, pratikac Praiiac@seas.upenaa seas.upenn.edu
ABSTRACT
Fine-tuning deep network trained with the standard cross-entropy loss is strong baseline for few-shot learning. When fine-tuned transductively, this outperforms the current state-of-the-art on standard datasets such as Mini-ImageNet, Tiered- ImageNet, CIFAR-FS and FC-100 with the same hyper-parameters. The simplicity of this approach enables to demonstrate the first few-shot learning results the ImageNet-21k k dataset. We find that using a large number of meta-training classes results in high few-shot accuracies even for large number of few-shot classes. We do advocate our approach as the solution for few-shot learning, but simply use the results to highlight limitations of current benchmarks and few-shot protocols. We perform extensive studies on benchmark datasets to propose a metric that quantifies the "hardness" of a few-shot episode. This metric can be used report the performance of few-shot algorithms in more systematic way.
INTRODUCTION
Mm Mmmiiageent (N)
80
60
40
1-chot,
20
Protatynical (2017)
MotaCpt SVM [2019)
Transductive Fina-Tuning
(2017]
[2018)
Figure 1: Are we making progress? The box-plo illustrates the performance of state-of of--te--rttt few-shot algorithms the Mini-ImageNer (Vinyals et al., 2016) dataset for the 1-shot 5-way protocol. The boxes show the + 25% quantiles of the accuracy while the notches indicate the median and its 95% confidence interval. Whiskers denote 1.5x interquartile range which captures 99.3% the probability mass normal distribution. The spread of the box-plots are large, indicating that the standard deviations the few-shot accuracies is large too. This suggests that progress may be illusory. especially considering that none outperform the simple transductive fine-tuning baseline discussed this paper (rightmost).
As image classification systems begin to tackle more and more classes, the cost of annotating massive number of images and the difficulty of procuring images of rare categories increases. This has fueled interest in few-shot learning, where only few labeled samples per class are available for training. Fig. displays snapshot of the state-of-the-an We estimated this plot by using published
Work done while Amazon Web Services

---

Published as a conference paper at ICLR 2020
numbers for the estimate of the mean accuracy, the 95% confidence interval of this estimate and the number of few-shot episodes. For MAML (Finn et al., 2017) and MetaOpt SVM (Lee et al., 2019), we use the number of episodes in the author's Github implementation.
The field appears to be progressing steadily albeit slowly based Fig. 1. However, the variance of the estimate of the mean accuracy is not the same as the variance of the accuracy. The former can be zero (e.g., asymptotically for an unbiased estimator), yet the latter could be arbitrarily large. The variance of the accuracies is extremely large in Fig. This suggests that progress in the past few years may be less significant than it seems if one only looks at the mean accuracies. To compound the problem, many algorithms report results using different models for different number of ways (classes) and shots (number of labeled samples per class), with aggressive hyper-parameter optimization." Our goal is develop simple baseline for few-shot learning, one that does not require specialized training depending on the number of ways or shots, nor hyper-parameter tuning for different protocols.
The simplest baseline we can think of is to pre-train a model on the meta-training dataset using the standard cross-entropy loss, and then fine-tune on the few-shot dataset. Although this approach is basic and has been considered before (Vinyals et al., 2016; Chen et al., 2018), it has gone unnoticed that it outperforms many sophisticated few-shot algorithms. Indeed, with a small twist of performing fine-tuning transductively, this baseline outperforms all of-thhoftthe--e art algorithms on all standard benchmarks and few-shot protocols (cf. Table 1).
Our contribution is to develop a transductive fine-tuning baseline for few-shot learning, our approach works even for a single labeled example and a single test datum per class. Our baseline outperforms the state-of-the-arr on a variety of benchmark datasets such as Mini-ImageNNt (Vinyals et al., 2016), Tiered-ImageN (Ren et al., 2018). CIFAR-FS (Bertinetto et al., 201 8) and FC-100 (Oreshkin et al., 2018), all with the same hyper-parameters. Current approaches to few-shot learning are hard to scale to large datasets. We report the first few-shot learning results on the ImageNet-21k dataset (Deng et al., 2009) which contains 14.2 million images across 21,814 classes. The rare classes in ImageNet-21k form a natural benchmark for few-shot learning.
The empirical performance of this baseline, should not be understood as us suggesting that this the right way of performing few-shot learning. We believe that sophisticated meta-training, understanding taxonomies and meronomies, transfer learning, and domain adaptation are necessary for effective few-shot learning. The performance of the simple baseline however indicates that we need to interpret existing results? with grain of salt, and be wary of methods that tailor to the benchmark. To facilitate that, we propose metric to quantify the hardness of few-shot episodes and a way to systematically report performance for different few-shot protocols.
2 PROBLEM DEFINITION AND RELATED WORK
We first introduce some notation and formalize the few-shot image classification problem. Let (x. y) denote an image and its ground- -tund-truthhaa label respectively. The training and test datasets are D. {(x,, 1 No and D,- {zz =1) respectively, where y; C2 for some set of classes C,. In the few-shot learning literature, training and test datasets are referred to as support and query datasets respectively, and are collectively called a few-shot episode. The number of ways, or classes, is |C.|- The set (a; y/ k (xi, 34) E D&} is the support of class l and its cardinality is s support shots (this is non-zero and is generally shortened to shots). The number s is small in the few-shot setting. The set {z; y = k, (I,,y) E D2} is the query of class k and its cardinality is q query shots. The goal is to learn a function to exploit the training set D, to predict the label of test datum æ,
For instance, Rusu et al. (2018) tune for different shot protocols, with parameters changing by up to six orders of magnitude; Oreshkin et al. (2018) use different query shot for different few-shot protocols. FFor instance, Vinyals et al. (2016); Ravi Larochelle (2016) use different versions Mini- Nmtgeee Oreshkin et al. (2018) report results for meta- training on the training set while Qiao et al. (2018) use both the training and validation sets; Chen et al. (2018) use full-sized images from the parent ImageNet- 1k dataset (Deng al., 2009); Snell et al. (2017); Finn al. (2017 7); Oreshkin et al. (2018): Rusu et al. (2018) use different model architectures of varying sizes, which makes it difficult disentangle the effect of their algorithmic contributions.
2

---

Published as a conference paper at ICLR 2020
Da. by where (x,
i = F(x; Dz).
(I)
Typical approaches for supervised learning replace D. above with a statistic, 0 = 0- (D.) that is, ideally, sufficient to classify Da. as measured by, say, the cross-entropy loss
0* (Ds) arg min
Σ
llgg po (y/ar)
N
(x.N))
where p-(1 is the probability distribution on C. as predicted by the model in response input When presented with a test datum, the classification rule is typically chosen to be of the form
Fg- (x; Dz) arg max pa- (k:)z)),
where D, is represented by 0" This form of the classifier entails a loss of generality unless 8* is sufficient statistic, po- (y|z) = plylx), which course never the case, especially given few labeled data in D.. However, conveniently separates training and inference phases, never having revisit the training set. This might be desirable in ordinary image classification, but not in few-shot learning. We therefore adopt the more general form of F in (1).
we call the test datum x xx,11* 11. then we can obtain the general form of the classifier by
1 N₂
Fa; D.) arg min min
log pe (vil(v)))).
No. In addition the training set, typically also has meta-training set, Dm {( 31) = where E Cm. with set of classes Cm disjoint from C.. The goal of meta- meta-training is to use Dm to infer the parameters of the few-shot learning model: Ó(Dmi (D,, D₂))
arg ming I (maDe ((y, Fa; (D,, D)))), where meta-training loss l depends on the method. N..
2.1 RELATED WORK
Learning to learn: The meta-training loss is designed to make few-shot training efficient (Utgoff, 1986: Schmidhuber, 1987: Baxter, 1995; Thrun, 1998). This approach partitions the problem into a base-level that performs standard supervised learning and a meta-level that accrues information from the base-level. Two main approaches have emerged to do so.
Gradient-based approaches: These approaches treat the updates of the base-level as learnable mapping (Bengio et al., 1992). This mapping et al., 2001; Ravi & Larochelle, 2016), or one can be learnt using temporal models (Hochreiter the gradients across the base-level updates (Maclaurin et al., 2015; Finn et al., 2017). It is challenging to perform this dual or bi-level optimization, respectively. These approaches have been shown to competitive on large datasets. Recent approaches learn the base-level in closed-form using SVMs (Bertinetto et al., 2018; Lee et al., 2019) which restricts the capacity of the hase-level although it alleviates the optimization problem.
Metric-based approaches: A majority of the tfat--of-the-ar algorithms are metric- based approaches. These approaches learn embedding that be used to compare (Bromley al., 1994; Chopra et al., 2005) or cluster (Vinyals et al., 2016; Snell et al., 2017) query samples. Recent approaches build upon this idea with increasing levels of sophistication in learning the embedding (Vinyals et al., 2016; Gidaris & Komodakis, 2018; Oreshkin et al., 2018), creating exemplars from the support set and picking metric for the embedding (Gidaris & Komodakis, 2018; Allen al., 2018: Ravichandran et al., 2019). There are numerous hyper-paramene parameters involved in implementing these approaches which makes hard to evaluate them systematically (Chen al., 2018).
Transductive learning: This approach is more efficient using few labeled data than supervised learning (Joachims, 1999; Zhou et al., 2004; Vapnik, 2013). The idea is to use information from the test datum x to restrict the hypothesis space while searching for the classifier F(x, D.) at test time. Our approach is closest this line of work. We train model the meta-training set D,,. and
3

---

Published as a conference paper at ICLR 2020
initialize a classifier using the support set Dz. The parameters then fine-tuned to adapt the new test datum æ.
There are recent papers in few-shot learning such as Nichol et al. (2018); Liu et (2018a) that motivated from transductive learning and exploit the unlabeled query samples. The former updates catcc-normalization parameters using query samples while the latter uses label propagation to estimate labels of all query samples at once.
Semi-supervised learning: We penalize the Shannon Entropy of the predictions on the query samples at test time. This is simple technique in the semi-supervised learning literature, closest to Grandvalet & Bengio (2005). Modern augmentation techniques such as Miyato et al. (2015); Sajjadi al. (2016); Dai al. (2017) or graph-based upproaches (Kipf & Welling, 2016) can also be used with our approach; used the entropic penalty for the sake of simplicity.
Semi-supervised few-shot learning is typically formulated as having access to extra unlabeled data during meta-training or few-shot training (Garcia Bruna, 2017; Ren et al., 2018). This different from approach which uses the unlabeled query samples for transductive learning.
Initialization for fine-tuning: We use recent ideas from the deep metric learning literature (Hu et al., 2015; Movshovitz-Attia et al., 2017: Qi et al., 2018; Chen et al., 2018; Gidaris & Komodakis, 2018) to initialize the meta-trained model for fine-tuning. These works connect the softmax cross-entropy loss with cosine distance and are discussed further in Section 3.1.
3 APPROACH
The simplest form of meta-training is pre-training with the cross-entropy loss, which yields
= arg min
Σ - Po + R((0)
(5)
Nm
(2,8)ED
where the second term denotes regularizer, say weight decay R(0) |0112/2. The model predicts logits 2x; 8) for i E Cm and the distribution po(-1x) is computed from these logits using the softmax operator. This loss typically minimized by stochastic gradient descent-based algorithms.
If few-shot training is performed according to the general form in (4), then the optimization is identical to that above and amounts to fine-tuning the pre-trained model. However, the model needs to be modified to account for the new classes. Careful initialization can make this process efficient.
3.1 SUPPORT- BASED INITIALIZATION
Given the pre-trained model (called the "backbone""), Po (dropping the hat from 0), we append new fully-connected "classifier" layer that takes the logits of the backbone input and predicts the labels in Ct. For support sample (a: y). denote the logits of the backbone by 2(2; 0) E R RlCml the weights and biases of the classifier by w E IR [+ and b E RIG1 respectively; and the 4th row of w and 6 by we and 6y respectively. The ReLU non-linearity is denoted by (-)+.
If the classifier's s logits are z' wz(x; 0)+ + b. the first term in the cross-entropy loss: - log ())2z) = (x; 0)+ - by + log Σn ==(=(==;0) +he would be the cosine distance between Wy and 2(x; 0)+ if both were normalized unit &g norm and bias by = O. This suggests
2(x; 0)+
wy = [|<(2; 01+
and by =0
(6)
as a candidate for initializing the classifier, along with normalizing 2(x; 0) to unit & norm. It is easy to see that this maximizes the cosine similarity between features 2(2; 0)+ and weights uy For
multiple support samples per class, take the Euclidean average of features 2<x; 0)+ for each class in Ct, before & normalization in (6). The logits of the classifier are thus given by
2(x; 8)+
IR Cil 9 2(1; = w [= 0)+
+ b.
4

---

Published as a conference paper at ICLR 2020
where θ {0, w, b), the combined parameters of the backbone and the classifier. Note that we have added ReLU non-linearity between the backbone and the classifier, before the l normalization. All the parameters are trainable the fine-tuning phase.
Remark (Relation to weight imprinting). The support-based initialization is motivated from previous papers (Hu et al., 2015; Movshovitz-AAtias et al., 2017; Chen et al., 2018; Gidaris & Komodakis, 2018). In particular, Qi et al. (2018) use similar technique, with minor differences, to expand the size of the final fully-connected layer (classifier) for low-shot continual learning. The authors call their technique "weight imprinting" because Wi can be thought of as template for class k. In case,
only interested in performing well the few-shot classes.
Remark 2 (Using logits of the backbone instead of features as input to the classifier). A natural way to adapt the backbone to predict new classes is to re-initialize its final fully-connected layer (classifier). We instead append new classifier after the logits of the backbone. This is motivated from Frosst et (2019) who show that for trained backbone, outputs of all layers are entangled, without class-specific clusters; but the logits are peaked on the correct class, and are therefore well-clustered. The logits are thus better inputs to the classifier as compared to the features. We explore this choice via an experiment Appendix C.6.
3.2 TRANSDUCTIVE FINE-TUNING
In (4), we assumed that there is a single query sample. However, also process multiple query samples together, and perform the minimization over all unknown query labels. We introduce regularizer, similar to Grandvalet & Bengio (2005), as we seek outputs with a peaked posterior, or low Shannon Entropy HI. So the transductive fine-tuning phase solves for
θ = arg min θ
1 No
Σ
- Pog (v) Nq (x.g)ED,
1
Σ Hipe( (//0(11))) |x)).
(8)
(x.
Note that the data fitting term uses the labeled support samples whereas the regularizer uses the unlabeled query samples. The two terms can be highly imbalanced (due the varying range of values for the two quantities, or due to the variance in their estimates which depend on N, and Ng). To allow finer control on this imbalance, one can use coefficient for the entropic term and/or temperature in the softmax distribution of the query samples. Tuning these syper-parameter. per dataset and few-shot protocol leads to uniform improvements in the results in Section 4 by 1-2%. However, we wish to keep in line with our goal of developing simple baseline and refrain from optimizing these hyper-parameters, and set them equal to for all experiments on benchmark datasets.
EXPERIMENTAL RESULTS
We show results of transductive fine-tuning on benchmark datasets few-shot learning, namely Mini-ImageNet (Vinyals et al., 2016), Tiered-ImageNet (Ren et al., 2018), CIFAR-FS (Bertinetto et al., 2018) and FC-100 (Oreshkin et al., 2018) in Section 4.1. We also show large-scale experiments on the ImageNet-21k dataset (Deng et al., 2009) in Section 4.2. Along with the analysis in Section 4.3, these help us design metric that measures the hardness of an episode in Section 4.4. We sketch key points of the experimental setup here; see Appendix A for details.
Pre-training: We use the WRN-28- 10 (Zagoruyko & Komodakis, 2016) model as the backbone. We pre-train using standard data augmentation, cross-entropy loss with label smoothing (Szegedy et al., 2016) of 1. mixup regularization (Zhang et al., 2017) of =0025 SGD with batch-size of 256, Nesterov's momentum of 0.9, weight-decay of 10- and no dropout. We use batch-normalizate (Ioffe & Szegedy, 2015) but exclude its parameters from weight decay (Jia et al., 2018). We use cyclic learning rates (Smith, 2017) and half- prefi-preiiion distributed training on GPUs (Howard et al., 2018) to reduce training time.
Each dataset has a training, validation and test set consisting of disjoint sets of classes. Some algorithms use only the training set as the meta-training set (Snell et al., 2017: Oreshkin et al., 2018), while others both training and validation sets (Rusu al., 2018). For completeness we report
5

---

Published as a conference paper at ICLR 2020
results using both methodologies; the former is denoted (train) while the latter is denoted (train + val). All experiments in Sections 4.3 and 4.4 use the (train + val) setting.
Fine-tuning: We perform fine-tuning on one GPU in full- cull-precision for 25 epochs and fixed learning rate of 5 x 10-5 with Adam (Kingma & Ba, 2014) without any regularization. We make two weight updates in each epoch: one for the cross-entropy term using support samples and one for the Shannon Entropy term using query samples (cf. (8)).
Hyper-parameters: We used images from ImageNet-IK belonging to the training classes of Mini- ImageNet as the validation set for pre-training the backbone for Mini-ImageNet. We used the validation set of Mini- Aaai-Im to choose are kept constant for experiments on benchmark datasets.
for fine-tuning. All hyper-parameters
Evaluation:
Few-shot episodes contain classes sampled uniformly from classes in the test sets of the respective datasets; support and query samples are further sampled uniformly for each class; the query shot fixed to 15 for all experiments unless noted otherwise. All networks are evaluated over 1,000 few-shot episodes unless noted otherwise. To enable easy comparison with existing literature, report an estimate of the mean accuracy and the 95% confidence interval of this estimate. However, we encourage reporting the standard deviation in light of Section and Fig. 1.
4.1 RESULTS ON BENCHMARK DATASETS
Table 1: Few-shot accuracies on benchmark datasets for 5-way few-shot episodes. The notation cony (64")xa denotes a CNN with 4 layers and 64 channels in the h layer. Best results in cach column are shown in bold. Results where the support-bascd initialization is better than comparable to existing algorithms are denoted by' 1, The notation (train val) indicates that the backbone was pre-trained on both training and validation sets the datasets; the backhone is trained only on the training set otherwise. (Lee et al., 2019) uses a 1.25x wider ResNet-12 which we denote as ResNet-12
Mini-Imaag (543
Therd-Imagg/Ntt
CIFAR-FS
FC-100
Algonihm
Architecture
I-shet (1R)
Matching nerworks (Vinyals et 2016) cone (Ki)x4
46.6
LSTM mete-lemmer (Ravi Larochelle, cone (Ki)xa 43.44 à: 0.77 60.60 a: 0.71 2016)
Prostypical Networks (Snell et al., 2017) cone (Ki)xx 49.42 à: 68.20 0.66
MAML (Finz et al., 2017)
cony (42)
48.70 à: 1.84 63.11 0.92
R2D2 (Beninetio et al., 2018)
conv (06*)x4 Jx
51.8 J: 0.2
68.4 4
65.4 02
79.4 0.2
TADAM (Oresikia et al., 2018)
ResNet 12
59.5 à 0.3 76.7 à
40.1 1.4
56.1 0:44
Transductive Propagation (Lu al., cone (K0)xa 55.51 :: 0.66 69 2018b)
0.65 59.91 096
73,90 0.75
Translative Propartion (Liu et th,, R2sNer-12 2018b)
39.46
75.64
MeuOgt SVM (Lec et al., 2019:
ReiNe:112 12" 62.64 :: 0.61 78.63 0.46
63.99
a 0.72
81.56 + 0.53
72.0
84.2 0.0
41.1 4: 0.6
53.5 à: 0.6
Supeen--sead intiiiioiion (Inin)
WRN-28-10 56.17 = 0.64 73.31 0.53 67.5
= ano7
82.88
0.5st 0.55
70.26 : 0.70 83382 = 0.e51 36.82 = 0.51 40.72 0.55
Fine miing (trais)
WRN 28 10 57.33 à: 0.62 78.17 0.89
66.59
a 0.70
85.55
a 0.48
68.72
a 0.67
86.11
0.47 38.25 = 0.52
57.19
0.57
Transdactvve fine using (eain)
WRN 28 10 65.73 1: 0.68 78.40 1: 0.53
73.34 a 0.71
85.50
0.50
76.58 aa 0.68
35.79 =: 0.50 43.16 =: 0.59 57.57 = 0.55
Activation to Parsecter (Qiio et al., 2018) WRN-28-10 59.60 0.41 73.74 0.19 (Inin + sal)
LFO Rusa al., 2018) (rainn + vall
WRN-28-10 61, + 0.08 77.59 0.12 66.33 0.65
81.44
0.00
MenSOpt SVM (1 al., 2019) (tmin ReaNem- 12 64.9 0.6. 50,00 0.45 65.81 0.34
81.75
0.53
72.8 128-007
65.0++0.5
47.2++006
62.5 0.6
vel)
Support-hooo inhallzzaiion (tmin val)
WRN-78-10 58.47 + 0.66 75.56 0.52 67.34 0.60f 81.30
0.517 23.14 0.601 8521 0.001
45.88 - 0.61
60,05 0.60
Fine-toning (trann val)
WRN-28-10 59.62 0.66 79.93 0.47 66.23 0668
86.08
0,47
7007 a 0.67
87.36
0.45
43,90
0.55
64.40
0.58
Transductive fine terain val:
WRN-28-10 10 48.11 0.69 80.36 0.50
72.87 0.71 86.15 0,50
78.36 6.70 $7.54 0.89 50.44 0,68 65.74 0.60
Table shows the results of transductive fine-tuning on benchmark datasetss for standard few-shot protocols. We see that this simple baseline is uniformly better than state-of-the-art algorithms. We include results for support- initialization, which does no fine-tuning, and for mee-tuning, which involves optimizing only the cross- cross-nntrpyy term in (8) using the labeled support samples.
The support-based initialization is sometimes better than comparable to -f-te-- algorithms (marked f). The few-shot literature has gravitated towards larger backbones (Rusu et al., 2018). Our results indicate that for large backbones even standard cross-entropy pre-training and support-based initialization work well, similar to observation made by Chen et al. (2018).

---

Published as a conference paper at ICLR 2020
For the 1-shot 5-way setting, fine-tuning using only the labeled support examples leads to minor improvement over the initialization, and sometimes marginal degradation. However, for the 5-shot 5-way setting nonttransductive fine-tuning better than the state-of-the-art.
In both (train) and (train val) settings, transductive fine-tuning leads to 2-7% improvement for 1-shot 5-way setting over the state-of of-the-art for all datasets. It results in an increase of 1.5-4% for the 5-shot 5-way setting except for the Mini-Imageti dataset, where the performance is matched. This suggests that the use of the unlabeled query samples is vital for the few-shot setting.
For the Mini-ImageNet, CIFAR-FS and FC-100 datasets, using additional data from the valida- tion set to pre-train the backbone results in 2-8% improvements; the improvement is smaller for hhered-ImageNee This suggests that having more pre-training classes leads to improved few-shot performance consequence of a better embedding. See Appendix C.5 for more experiments.
4.2 LARGE-SCALE FEW-SHOT LEARNING
The ImageNet-21l dataset (Deng et al., 2009) with 14.2M images across 21,814 classes is an ideal large-scale few-shot learning benchmark due the high class imbalance. The simplicity of our approach allows us to present the first few-shot learning results on this large dataset. We use the 7,491 classes having more than 1, 1,000 images each as the meta-training set and the next 13,007 classes with least 10 images each for constructing few-shot episodes. See Appendix B for details.
Table 2: Accuracy (%) on the few-shot data of ImageNet-211k, The confidence intervals are large because we compute statistics only over 80 few-shot episodes so as to test for large number of ways.
Way
Algorithm
Model
Shot
10
20
40
50
160
Support based initalization
WRN-28-10
1 87.20 =: 1.72
7871 + 1,3
69.48 1,30 00.55 à: 103
49.15
0.88
40.57 1: 0.42
Transductive fine-tuning
WRN-28-10
1.56
79.88
1,70
60.72
48.88
40
Support-bure based initialization
WRN WRN-28-10
5
96.73 A: 0.84
91.00 à 1,09
84.77 = L.04
78.10
0.99
7009
0.71
61.93
0.45
Trandhecive fino-tuning
WRN-28-10
95.20 + 0.94
90.1 + 1,03
84,21 + 1.09
77.13
0.82
68
0.75
60.11 10.11:+000 0.48
Table 2 shows the mean accuracy of transductive fine-tuning evaluated over 80 few-shot episodes on ImageNet-21k. The accuracy is extremely high compared to corresponding results in Table I even for large way. E.g., the 1-shot 5-way accuracy on Tiered- Gerred-Imaeeeeii is 72.87 + 0.71% 1% while it is 89 + 1.86% here. This corroborates the results in Section 4.1 and indicates that pre-training with a large number of classes may be an effective strategy to build large-scale few-shot learning systems.
The improvements of transductive fine-tuning are minor for ImageNet-21X because the support-based initialization accuracies are extremely high. We noticed a slight degradation of accuracies due to transductive fine-tuning at high ways because the entropic term in (8) is much larger than the the cross-entropy loss. The experiments for ImageNet-211 therefore scale down the entropic term by log |C.] and forego the ReL.U in (6) and (7). This reduces the difference in accuracies at high ways.
4.3 ANALYSIS
This section presents comprehensive analysis of transductive fine-tuning on the Mini-ImageNet, Tiered- ImageNet and ImageNet-214 2k datasets.
Robustness of transductive fine-tuning to query shot: Fig. 2a shows the effect of changing the query shot on the mean accuracy. For the 1-shot 5-way setting, the entropic penalty in (8) helps as the query shot increases. This effect is minor in the 5-shot 5-way setting as more labeled data is available. Query shot of achieves a relatively high mean accuracy because transductive fine-tuning can adapt to those few queries. One query shot is enough benefit from transductive fine-tuning: for Mini- Imag-Immaeeet, the 1-shot 5-way accuracy with query shot of is 66.94 + 1.55% which is better than non non-transductive fine-tuning (59.62 + 0.66% in Table 1) and higher than other approaches.
Performance for different way and support shot: A few-shot system should be able to robustly handle different fho- scenarios. Figs. 2b and 2c, show the performance of transductive fine-tuning
7

---

Published as a conference paper at ICLR 2020
100
stot F-way Mini-inagg inagerrr
shot
phoo S-way Tiared Imagunet
ahoe
80
60
Noa
sihtt Thered- Inaa--maaaaiee
and
20
10 stot
1ive Imagsten22
20
20 way
way Tereshmaggnd
D3 way
10 anct
1G0 way
Query snot
Support
(a)
(b)
(c)
Figure 2: Mean accuracy of transductive fine-tuning for different query shot, way and support shot. Fig. 2a shows that the mean accuracy improves with query shot if the support shot is low; this effect is minor for Terred-ImageNet. The mean accuracy for query shot of 1 high because transductive fine-tuning those queries. Fig. 2b shows that mean accuracy degrades logarithmically with way for fixed support shot and query shot (15). Fig. 2c suggests that the mean accuracy improves logarithmically with the support shot for fixed way and query shot (15). These trends suggest thumb rules for building few-shot systems.
specialize
with changing way and support shot. The mean accuracy changes logarithmically with the way and support shot which provides thumb rules for building few-shot systems.
Different backbone architectures: We include experiments using conv (64)x4 (Vinyals et al., 2016) and R2 (He et al., 201 Oreshkin et al., 2018) in Table 3, in order to facilitate comparisons for different backbone architectures. The results for transductive fine-tuning are comparable or better than sate--o-ttheaare for a given backbone architecture, except for those in Liu al. (2018b) 8b) who more sophisticated transductive algorithm using graph propagation, with conv (64) x4. In line with our goal for simplicity, we kept the hyper-parameters for pre-training and fine-tuning the same as the ones used for WRN-28- 10 (cf. Sections 3 and 4). These results show that transductive fine-tuning is a sound baseline for variety of backbone architectures.
Computational complexity: There is no free lunch and our advocated baseline has its limitations. performs gradient updates during the fine-tuning phase which makes it slow at inference time. Specifically, transductive fine-tuning is about 300x slower (20.8 vs. 0.07 seconds) for 1-shot 5-way episode with 15 query shot compared to Snell et al. (2017) with the same backbone architecture (prototypical networks (Snell et al., 2017) do not update model parameters at inference time). The latency factor reduces with higher support shot. Interestingly, for a single query shot, the former takes 4 seconds vs. 0.07 seconds. This is more reasonable factor of 50x, especially considering that the mean accuracy of the former is 66.2% compared to about 58% of the latter in our implementation. Experiments in Appendix C.3 suggest that using a smaller backbone architecture partially compensates for the latency with some degradation of accuracy. number of approaches such as Ravi & Larochelle (2016); Finn et al. (2017); Rusu et al. (2018); Lee et al. (2019) also perform additional processing at inference time and are expected to be slow, along with other transductive approaches (Nichol al., 2018; Liu et al., 201 2018b). Additionally, support- initialization has the same inference time Snell et al. (2017).
4.4 A PROPOSAL FOR REPORTING FEW-SHOT CLASSIFICATION PERFORMANCE
As discussed in Section 1, we need better metrics to report the performance of few-shot algorithms. There are two main issues: (i) standard deviation of the few-shot accuracy across different sampled episodes for given algorithm, dataset and few-shot protocol is very high (cf. Fig. 1), and (ii) different models and hyper-pance parameerrssssss for different few-shot protocols makes evaluating algorithmic contributions difficult (cf. Table 1). This section takes step towards resolving these issues.
Hardness of an episode: Classification performance on a few-shot episode determined by the relative location of the features corresponding to labeled and unlabeled samples. If the unlabeled
8

---

Published as a conference paper at ICLR 2020
features close to the labeled features from the same class, the classifier distinguish between the classes easily to obtain a high accuracy. Otherwise, the accuracy would be low. The following definition characterizes this intuition.
For training (support) set D, and test (query) set a. will define the hardness S2p the average log-odds of test datum being classified incorrectly. More precisely,
(Di D2)
Σ Ng (x.y)E22
1
log
1 1 p(y pyy x))
where p(-) æ) is a softmax distribution with logits 2y (x) is the weight matrix constructed using (6) and Dg: and y is the l normalized logits computed using a rich-enough feature generator, say deep network trained for standard image classification. This is clustering loss where the labeled support samples form class-specins cluster centers. The cluster affinities are calculated using cosine-similarities, followed by the softmax operator to get the probability distribution p(-1 r).
Note that Que does not depend on the few-shot learner and gives measure of how difficult the classification problem is for any few-shot episode, using generic feature extractor.
CIFAR-FS
100
FC-100
Tiered-Imagenet
Mini-lmagenet
Imagenet-21k
80
Accuracy (%)
40
20
2
Hardness
Figure 3: Comparing the accuracy of transductive fine-tuning (solid lines) vs. support-based initialization (dotted lines) for different datasets, ways (5, 10, 20, 40, 80 and 160) and support shots (1 and 5), Abscissae are computed using (9) and a Resnet-152 the ImageNet- dataset. Each marker indicates the accuracy of transductive fine-tuning on a few-shot episode; markers for support-based initialization are hidden to avoid clutter. Shape of the markers denotes different ways; ways increase from left right (5, 10, 20, 40, 80 and 160). Size of the markers denotes different support shot (1 and 5): it increases from the bottom to the top. E.g., the ellipse contains accuracies of different 5-shot dataset by combining
al., 2016b) network trained standard image classification on
ImageNet-2Ik, Regression lines are drawn for each algorithm the episodes of all few-shot protocols. This plot akin to precision-recall curve and allows comparing two algorithms for different few-shot scenarios. The areas in the first quadrant under the fitted regression lines are 295 vs. 284 (CIFAR-FS) 167 vs. 149 (FC-100), 208 194 (Mini- ImageNet), 280 270 Tiered-ImageNee 475 vs. 484 (ImageNet-21) for transductive fine-tur tuning support- based initialization.
10-way episodes

---

Published as a conference paper at ICLR 2020
Fig. 3 demonstrates how use the hardness metric. Few-shot accuracy degrades linearly with hardness. Performance for all hardness can thus be estimated by testing for two different ways. We advocate selecting hyper-parameters using the area under the fitted curve as metric instead of tuning them specifically for each few-shot protocol. The advantage of such test methodology is that it predicts the performance of the model across multiple few-shot protocols systematically.
Different algorithms can be compared directly, c.g., transductive fine-tuning (solid lines) and support- based initialization (dotted lines). For instance, the former leads large improvements on easy episodes, the performance is similar for hard episodes, especially for Tiered- and ImageNet-21k.
The high standard deviation of accuracy of few-shot learning algorithms in Fig. can be seen as the spread of the cluster corresponding to each few-shot protocol, c.g., the ellipse in Fig. 3 denotes the 5-shot 10-way protocol for ImageNet-2Ik. It is the nature of few-shot learning that episodes have varying hardness even the way and shot are fixed. However, episodes within the ellipse lie on a different line (with large negative slope) which indicates that given a few-shot protocol, hardness is good indicator of accuracy.
Fig. 3 also shows that due to fewer test classes, CIFAR-FS, FC-100 and Mini- Imag-etteeNet have less diversity in the hardness of episodes while Tiered- IntgeNe and ImageNet-21K 21k allow sampling of both very hard and very easy diverse episodes. For given few-shot protocol, the hardness of episodes in the former three is almost the same as that of the latter two datasets. This indicates that CIFAR-FS, FC-100 and Mini-ImageNet may be good benchmarks for applications with few classes.
The hardness metric in (9) naturally builds upon existing ideas in deep metric learning (Qi et al., 2018). We propose it as means to evaluate few-shot learning algorithms uniformly across different few-shot protocols for different datasets; ascertaining its efficacy and comparisons to other metrics will be part of future work.
DISCUSSION
Our aim is to provide grounding to the practice few-shot learning. The current literature is in the spirit of increasingly sophisticated approaches for modest improvements in mean accuracy using an inadequate evaluation methodology. This is why we set out to establish a baseline, namely transductive fine-tuning, and a systematic evaluation methodology, namely the hardness metric. We would like emphasize that our advocated baseline, namely transductive fine-tuning, is novel and yet performs better than existing algorithms on all standard benchmarks. This is indeed surprising and indicates that we need to take a step back and re-evaluate the status quo in few-shot learning. We hope to use the results in this paper as guidelines for the development of new algorithms.
REFERENCES
Kelsey R Allen, Hanul Shin, Evan Shelhamer, and Josh Tenenbaum. Variadic learning by bayesian nonpara- metric deep emhedding. 2018.
Jonathan Baxter. Learning internal representations. Flinders University of S. Aust., 1995.
Samy Bengio, Yoshua Bengio, Jocelyn Cloutier, and Jan Geesei. On the optimization of synaptic learning rule. In Preprints Conf. Optimality in Artificial and Biological Neural Networks, pp. 6-8. Univ. of Texas, 1992.
Luca Bertinetto, João F Henriques, Philip HS Torr, and Andrea Vedaldi. Meta-lcarning with differentiable closed-form solvers. 18X.v:1805008)6 2018.
Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verification using a" siamese" time delay neural network. In Advances in neural information processing systems, pp. 737-744, 1994.
Wei-Yt Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu- Yu-Chiang Frank Wang, and Jia Bin Huang. A closer look at few-shot classification. 2018.
Sumit Chopra, Raia Hadsell, Yann LeCun, et al. Learning similarity metric discriminatively, with application face verification. In CVPR (1), pp. 539-546, 2005.
10

---

Published as a conference paper at ICLR 2020
Zihang Dai, Zhilin Yang, Fan Yang. William W Cohen, and Ruslan Salakhutdinov. Good semi-supervised learning that requires a bad gan. In Advances in neural information processing systems, pp. 651 6510-655 11-1600,,2266 2017. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248 148-255. Icce, 2009.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Lodellagnostic meta meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Coarning-Volune 70, pp. 1126- 1135. JMLR. org, 2017.
Nicholas Frosst, Nicolas Paperoo,, and Geoffrey Hinton. Analyzing and improving representations with the soft nearest neighbor loss. arXiv: 90889. 2019.
Victor Garcia and Joan Bruna. Few-shot learning with graph neural networks. arXiv: 1.00044,, 2017.
Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition. pp. 4367-4375, 2018.
Yves Grandvalet and Yoshua Bengio. Semi- semi-supervised learning by entropy minimization. In Advances in neural information processing systems, pp. 529-536, 2005.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In The IEEE Conference on Computer Vtsion and Pattern Recognition (CVPR), June 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. arXiv: 3oXiv:6033.030 1603.05077 2016b.
Sepp Hochreiter, A Steven Younger, and Peter R Conwell. Learning to learn using gradient descent. In International Conference on Artificial Neural Nenworks, pp. 87-94. Springer, 2001.
Jeremy Howard al. fastai. https:/ /tgpt/tttttu....o com/ fastai fastai, 2018.
Junlin Hu, Jiwen Lu, and Yap-Peng Tan. Deep transfer metric learning. In Proceedingy of the IEEE conference on computer vision and pattern recognition, pp. 325-333, 2015.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv: 150.001502.3317, 2015.
Xianyan Jia, Shutao Song, Wei He, Yangzihao Wang, Haidong Rong, Feihu Zhou, Liqiang Xie, Zhenyu Guo, Yuanzhou Yang, Liwei Yu, al. Highly scalable deep learning training system with mixed precision: Training imagenet in four minutes. arXi::/807.1100: 2018
Thorsten Joachims. Transductive inference for text classification using support vector machines. In leml, volume 99, pp. 200-209, 1999.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv:1412.6980, 2014.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. 1eXXvv:600.020000 2016.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citescer, 2009.
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. UrXiv:/904.000 037558 2019.
Yanbin I. Liu, Juho Lee, Minseop Park, Sachoon Kim, Funho Yang, Sung Ju Hwang, and Yi Yang. Learning to propagate labels: Transductive propagation network for few-shot learning. 2018a.
Yanbin Liu, Juho Lee, Minscop Park, Sachoon Kim, and Yi Yang. Transductive propagation network for few-shot learning. arXiv:1805. 10002, 2018b.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv: 1608.0308 2016.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE, Journal machine learning research, (NNov):5579-260)) 2008.
Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient-based hyperparameter optimization through reversible learning. In International Conference on Machine Learning, pp. 2113-21222 2122, 2015.
Paulius Micíkevicius, Sharan Narang. Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. arXiv:1710.037400 2017.
11

---

Published as a conference paper at ICLR 2020
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, Ken Nakae, and Shin Ishii Distributional smoothing with virtual adversarial training. arXiv: 2015.
Yair Movshoviiz-Attias, Alexander Toshev, Thomas K Leung, Sergey loffe, and Saurabh Singh. No fuss distance metric learning using proxies. In Proceedings of the IEEE International Conference on Computer Vision, pp- 360-368, 2017.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv: 1eXiv:8003002999 2018.
Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for
improved few-shot learning. In Advances in Neural Information Processing Systems, pp. 719-729, 2018.
Hang Qi, Matthew Brown, and David G Lowe. Low-shot learning with imprinted weights. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5822-5830, 2018.
Siyuan Qiao, Chenxi Liu, Wei Shen, and Alan L Yuille. Few-shot image recognition by predicting parameters from activations. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp.
7229-7238, 2018.
Sachin Ravi and Hugo Larochelle. Optimization model for few-shot learning. 2016.
Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Few-shot learning with embedded class models and shot-free training, 2019.
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenenbaum, Hugo Larochelle, and Richard $ Zemel. Meta-learning for semi-supervised few-shot classification.
arXiv:1803.00670 2018.
Andrei Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. (0ti::8807.0960,, 2018.
Mehdi Sajjadi, Mehran Javanmardi. Tolga Tasdizen. Regularization with stochastic transformations and perturbations for deep semi-supervised learning. In Advances in Neural Information Processing Systems, pp- 1163-1171, 2016.
Jurgen Schmidhuber. Evolutionary principles in self-referential learning. On learning how to learn: The mela-meta... book.) Diploma thesis, Institut.. f. Informatik, Tech. Univ. Munich, 1987.
Leslic N Smith. Cyclical learning rates for training neural networks. In 2017 IEEE Winter Conference on Applications of Computer Vision (WACV). pp. 464-472. IEEE, 2017.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, pp. 4077-4087, 2017.
Christian Szegedy. Vincent Vanhoucke, Sergey loffe. Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2818 2826,, 2016.
Sebastian Thrun Lifelong learning algorithms. Learning to learn, pp. 181-209. Springer. 1998.
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle. Meta-dataset: dataset of datasets for learning to Ieam from few examples. arXiv preprint arXiv: /90.:/033.00 003990 2019.
Paul E Utgoff. Shift of bias for inductive concept learning. Machine learning: An artificial intelligence approach. 2:107-148, 1986.
Vladimir Vapnik. The nature of statistical learning theory. Springer science business media, 2013.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap. Daan Wierstra, al. Matching networks for shot learning. In Advances in neural information procesxing systems, pp. 3630-3638, 2016.
Junyuan Xie, Tong He, Zhi Zhang, Hang Zhang. Zhongyue Zhang, and Mu Li. Bag of tricks for image classification with convolutional neural networks. 182.1:181..01187, 2018.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv:660.007/44 2016.
Hongyi Zhang, Moustapha Cisse. Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv:/710.09412. 2017.
Dengyong Zhou, Olivier Bousquet, Thomas N Lal, Jason Weston, and Bernhard Schölkopf. Learning with local and global consistency. In Advances neural information processing systems, pp. 321-328, 2004.
12

---

Published as a conference paper at ICLR 2020
A SETUP
A.1 DATASETS
We use the following datasets for our benchmarking experiments.
The Mini-ImageNet dataset (Vinyals et al., 2016) which is subset of ImageNet-Ik (Deng et al., 2009) and consists of 84 x 84 sized images with 600 images per class. There are 64 training, 16 validation and 20 test classes. There are multiple versions of this dataset in the literature; we obtained the dataset from the authors of Gidaris & Komodakis (2018)².
The
dataset (Ren al., 2018) is larger subset of ImageNet-Ik Ik with 608 classes split as 351 training, 97 validation and 160 testing classes, each with about 1300 images of size 84 x 84. This dataset ensures that training, validation and test classes do not have semantic overlap and is a potentially harder few-shot learning dataset.
We also consider two smaller CIFAR-100 (Krizhevsky Hinton, 2009) derivatives, both with 32 x 32 sized images and 600 images per class. The first is the CIFAR-FS dataset (Bertinetto et al., 2018) which splits classes randomly into 64 training, 16 validation and 20 test. The second is the FC-100 dataset (Oreshkin et al., 2018) which splits CIFAR-100 into 60 training, 20 validation and 20 test classes with minimal semantic overlap.
Each dataset has training, validation and test set. The set of classes for each of these sets are disjoint from each other. For meta-training, we ran two sets of experiments: the first, where we only use the training set as the meta- veta-training dataset, denoted by (train); the second, where we use both the training and validation sets as the meta-trainie srattttaaning dataset, denoted by (train val). We use the test set to construct feo-shot episodes.
A.2 PRE-TRAINING
We use a wide residual network (Zagoruyko & Komodakis, 2016; Qiao et al., 2018; Rusu al., 2018) with a widening factor of 10 and depth of 28 which we denote as WRN-28-10. The smaller networks: conv (64)x4 (Vinyals et al., 2016; Snell et al., 2017), ResNet-12 (He et al., 2016a; Oreshkin et al., 2018; Lee et al., 2019) and WRN-16-4 (Zagoruyko & Komodakis, 2016), are used for analysis in Appendix C. All networks are trained using SGD with batch-size of 256, Nesterov's momentum set to 0.9, no dropout, weight decay of 10-4. We use aatch-normalization (loffe & Szegedy, 2015). We use two-cycles of learning rate annealing (Smith, 2017), these are 40 and 80 epochs each for all datasets except ImageNet-21k, which uses cycles of 8 and 16 epochs each. The learning rate set to 10- at the beginning of the 2th cycle and decreased to 10- by the end of that cycle with cosine schedule (Loshchilov & Hutter, 2016). We use data parallelism across 8 Nvidia V100 GPUs and half-precision training using techniques from Micikevicius et al. (2017); Howard et al. (2018).
We use the following regularization techniques that have been discovered in the non-few-shot, standard image classification literature (Xie et al., 2018) for pre-training the backbone.
Mixup (Zhang et al., 2017): This augments data by a linear interpolation between input images and their one-hot labels. If (z1, 11), (I2, V2) E D are two samples, mixup creates a new sample (ă, y) where ã Ax_ is the one-hot vector with a non-zero thth entry and À E [0, 1] is sampled from Beta(a, a) for a hyper-parameter a.
A)rz and its label ÿJ = de + (1 A)e here ek
Label smoothing (Szegedy et al., 2016): When using a softmax operator, the logits can increase or decrease in an unbounded manner causing numerical instabilities while training. if k: y and e/(K 1) otherwise, for small constant > 0 and number of classes K. The ratio between the largest and smallest output neuron is thus fixed which helps large-scale training.
Label smoothing sets pa (k|=))
We exclude the batch-normalization parameters from weight-decay (Jia et al., 2018).
13

---

Published as a conference paper at ICLR 2020
We set c=0.1 for label smoothing eross entsoy loss and a=0.25 for mixup regularization for all our experiments.
A.3 FINE-TUNING HYPER-PARAMETTE
We used 1-shot 5-way episodes on the validation set of Mini-ImageNet to manually tune hyper- parameters. Fineituning is done for 25 epochs with a fixed learning rate of 5 x 10-6 with Adam (Kingma & Ba, 2014). Adam is used here as it is more robust to large changes in the magnitude of the loss and gradients which occurs if the number of classes in the few-shot episode (ways) is large. We do not use any regularization (weight-decay, mixup, dropout, or label smoothing) in the fine-tuning phase. These syper--arameters are kept constant on all benchmark datasets, namely Mini-ImageNet, Tiered-Imagelt CIFAR-FS and FC-100.
All fine-tuning and evaluation is performed on single GPU in full-precision. We update the parameters sequentially by computing the gradient of the two terms in (8) independently. This updates both the weights of the model and the nonaainormalization parameters.
A.4 DATA AUGMENTATION
Input images are normalized using the mean and cuandard-devaatoon computed on ImageNet-1k, Our Data augmentation consists of left-right flips with probability of 0.5, padding the image with 4px and adding brightness and contrast changes of + 40%. The augmentation is kept the same for both pre-training and fine-tuning.
We explored augmentation using affine transforms of the images but found that adding this has minor effect with no particular trend on the numerical results.
A.5 EVALUATION PROCEDURE
The few-shot episode contains classes that are uniformly sampled from the test classes of correspond- ing datasets. Support and query samples are further uniformly sampled for each class. The query shot is fixed to 15 for all experiments unless noted otherwise. We evaluate all networks over 1,000 episodes unless noted otherwise. For ease of comparison, report the mean accuracy and the 95% confidence interval of the estimate of the mean accuracy.
B SETUP FOR IMAGENET-2IK
The ImageNet-21k dataset (Deng et al., 2009) has 14.2M images across 21,814 classes. The blue region Fig. 4 denotes our meta-training set with 7,491 classes, each with more than 1,000 images. The green region shows 13,007 classes with at least 10 images each, the set used to construct few-shot episodes, We do not use the red region consisting of 1,343 classes with less than 10 images each. train the same backbone (WRN-28-10) with the same procedure as that in Appendix A on x the other
84 resized images, albeit for only 24 epochs. Since we the same hyper-parameter benchmark datasets, we did not create validation sets for pre-training or the fine-tuning phases. The few-shot episodes are constructed in the same way as Appendix A. We evaluate using fewer few-shot episodes (80) on this dataset because we would like to demonstrate the performance across large number of different ways.
C ADDITIONAL ANALYSIS
This section contains additional experiments and analysis, complementing Section 4.3. All ex- periments use the (train val) setting, pre-training on both the training and validation sets of corresponding datasets, unless noted otherwise.
14

---

Published as a conference paper at ICLR 2020
10 3
images per class
10
10
10
5000
10000
15000
20000
classes
Figure 4: ImageNet-211 is highly imbalanced dataset. The most frequent class has about 3K images while the rarest class has single image.
Figure 5: 1-SNE (Maaten & Hinton, 2008) embedding of the logits for 1-shot 5-way few-shot episode of Mini-ImageNNe.. Colors denote the ground-truth -truh labels; crosses denote the support samples; circles denote the query samples; translucent markers and opaque markers denote embeddings before and after transductive fine-tuning respectively, Even though query samples are far away from their respective supports in the beginning. they move towards the supports hy the end of transductive fine-tuning. Logits of support samples are relatively unchanged which suggests that the support-based initialization is effective.
C. TRANSDUCTIVE FINE-TUNING CHANGES THE EMBEDDING DRAMATICALLY
Fig. demonstrates this effect. The logits for query samples are far from those of their respective support samples and metric-based loss functions, e.g., those for prototypical networks (Snell et al., 2017) would have a high loss on this episode; indeed the accuracy after the support-based initialization 64%. Logits for the query samples change dramatically during transductive fine-tuning and majority of the query samples cluster around their respective supports. The post transductive fine-tuning accuracy of this episode is 73.3%. This suggests that modifying the embedding using the query samples is crucial to obtaining good performance on new classes. This example also demonstrates that the support-based initialization is efficient, logits of the support samples are relatively unchanged during the transductive fine-tuning phase.
15

---

Published as a conference paper at ICLR 2020
C.2 LARGE VS. SMALL BACKBONES
The expressive power of the backbone plays an important role in the efficacy of fine-tuning. We observed that a WRN-16-4 architecture (2.7M parameters) performs worse than WRN-28- 10 (36M parameters). The former obtains 63.28 + 0.68% and 77.39 0.5% accuracy on Mini-Image Mini-ImageNet and 69.04 = 0.69% and 83.55 + 0.51% % accuracy on Tiered-ImageNet on I-shot 5-way and 5-shot 5-way protocols respectively. While these numbers are comparable to those of state-of-the-art algorithms, they are lower than their counterparts for WRN-28- 10 in Table 1. This suggests that larger network is effective in learning richer features from the meta-training classes, and fine-tuning is effective in taking advantage of this to further improve performance on samples belonging to few-shot classes.
C.3 LATENCY WITH A SMALLER BACKBONES
The WRN-16-4 architecture (2.7M parameters) is much smaller than WRN-28-10 10 (36M parameters) and transductive fine-tuning on the former is much faster. As compared to our implementation of Snell et al. (2017) with the same backbone, WRN-16-4 is 20-70x slower (0.87 0.04 seconds for query shot of 1, and 2.85 vs. 0.04 seconds for query shot of 15) for the 1-shot 5-way scenario. Compare this to the computational complexity experiment Section 4.3.
As discussed in Appendix C.2, the accuracy of WRN- 16-4 is 63.28 0.68% and 77.39 0.5% for 1-shot 5- 5-way and 5-shot 5-way on Mini-ImageNNe respectively. As compared to this, our implementation of (Snell et al., 2017) using a WRN- backbone obtains 57.29 + 0.40% and 75.34 + 0.32% accuracies for the same settings respectively; the former number in particular is significantly worse than its transductive fine-tuning counterpart.
C.4 COMPARISONS AGAINST BACKBONES IN THE CURRENT LITERATURE
We include experiments using conv (64) x4 and ResNet-12 in Table 3, in addition to WRN-28 10 in Section 4. in order to facilitate comparisons of the proposed baseline for different backbone architectures. Our results are comparable or better than existing results for a given backbone architecture, except for those in Liu et al. (2018b) who use graph-based transduction algorithm, for conv (64)x4 on Mini-ImageNG In line with goal for simplicity, we kept the hyper-parameters for pre-training and fine-tuning the same as the ones used for WRN-28-10 (cf. Sections 3 and 4). These results suggest that transductive fine-tuning is a sound baseline for a variety of backbone architectures.
C.5 USING MORE META-TRRINIE CLASSES
In Section 4.1 observed that having more pre-training classes improves few-shot performance. But since we append classifier on top of pre-trained backbone and use the logits of the backhone as inputs to the classifier, a backbone pre-trained on more classes would also have more parameters as compared to one pre-trained on fewer classes. However, this difference is not large: WRN-28-11 10 for Mini- -mnie-maaeNNet has 0.03% more parameters for (train val) compared to (train). However, in order to facilitate a fair comparison, we ran an experiment where we use the features of the backbone, instead of the logits, as inputs to the classifier. By doing so, the number of parameters in the pre-trained backbone that are used for few-shot classification remain the same for both the (train) and (train + val) settings. For Mini-ImageNet (train + val) obtains 64.20 + 0.65% and 81.26 + 0.45%, and (train) obtains 62.55 + 0.65% and 78.89 + 0.46%, for 1-shot 5-way and 5-shot 5-way respectively. These results corroborate the original statement that more pre-training classes improves few-shot performance.
C.6 USING FEATURES OF THE BACKBONE AS INPUT TO THE CLASSIFIER
Instead of re-initializing the final fully-connected layer of the backbone to classify new classes, simply append the classifier on top of it. We implemented the former, more common, approach and found that it achieves an accuracy of 64.20 + 0.65% and 81.26 + 0.45% for 1-shot 5-way and 5-shot 5-way respectively on Mini-ImageNer, while the accuracy Tiered-ImageNet is 67.14 +
16

---

Published as a conference paper at ICLR 2020
Table 3: Few-shot accuracies benchmark datasets for 5-way few-shot episodes. The notation conv (644)) x4 denotes a CNN with layers and 64" channels in the 1ahh layer. The rows are grouped by the backbone architectures. Best results in each column and for a given backbone architecture are shown in bold. Results where the support-based initialization is better than or comparable to existing algorithms are denoted by The notation (train + val) indicates that the backbone the datasets; the backbone is trained only on the training set otherwisc. (Lee ct al., 2019) uses a 1.25x wider ResNet-12 12 which we denote as ResNet-11
pre-trained on both training and validation sets of
Mini- ImageNet
Net sbot
CIFAR-TS
FC-100
Algoritim
Aschinettre
shot (8)
shoc
shot(())
sbot
1
sbot (8)
MAML (Pan 2017)
COON (1 48.70 63.11 0.9
Matching networks (Ninyals et al., 2016) comv (64))4
46.6
LSTM meta learner (Rari & Larochelle, comv (64) 43.44 2016)
a 0.77 60 0.71
Prnteypical Networks (Soell al., 2017) 2017) coes (c4)
49.42 = 0.76 68 20 = 0.66
Trasductive Progagation (Lis at al., cuew (64) 2015b)
55.51 = 0.86 69.86 0.65 59.91 0.94
73.50 0.75
Support hased iniiaaiizaiinn (trin))
cozy (64)
30,69 a: 0.63 66.07 a: 0.55 58.42 0.69
73,98 a 0.58 61.77 0.75 76.40 1. 0.54 36,07
0.54
48.72 0.57
Fine using (trin)
cozv (64)
49.43 a 0.62 66.42 à: 0.53
57.45 a 0.68
73.96
1 0.56 19.74 a 0.72 76.37 1. 0.53 35,46 = 0.55
49.43 0.57
Transductive fine uning (cain)
cozv (44)
50,46 a 0.62 66.68 a: 0.52 $8.05 0.65
74.24 a 0.56 61.73 & 0.72 76,92
0.52 36,62 =: 0.55
50.24 1 0.58
R2D2 [Bertinctio et al., 2018)
cusv 196
51.8 0.2
68.4 + 0.2
65.4 4 794
TADAM (Oreshiin 2018)
ResNet 12
58.5 L 0.5
76.7
40.1
56.1 0.4
Trunsductive Propagztion (Lis 2013b)
ResNet 12
59,46
75.64
Supp-st-lueed intiialiattson (tmin)
RosNer-12
$4.21
70.58
0.54
66.39
81.93
0.54
65.99
0.72
79.95
0.51
35.51
48.26 2.56000.44
Fino-tuming
RosNer-12
$6.67
74
051
6..45+0.00
R3.59
0.51
64/6
82.13
0.50 37,52
0.57
Thrrsdactive fine-trning (tain))
RecNet-12
53
0.54
68.11
6.73
0.52
70.76
B1
0.53
MecaOpe et 2019)
ResNer
76.63
0.46
0.72
12.0..077
41.1
5555 1.0.6
Supp-rt-hueed initialization ion (tmin)
WRN-28-11 56.17 + 73:31 + 0.53
67.45+0.00
0.51
70.26
0.70
0.40
36.82 40.72 0.55
Fire-tuning (tmin)
WRN-28-10
$7.73
78.17
0.49
66.58
0.70
68.72
86.1
38.35
57.19 0.57
Trenshetivve fine-tmning (mmin)
WRN-28-10
68.73
0.68 78.40
0.82
73.34 : 6.71
85.50
0,50
76,58
0.68
85.79
0.50
43.16
0.59
57.57
0.55
Support hased intiaaizzaiinn (train vall
(64)xa 52,77 0.64
68.29
0.54
59.05 L 9.70
74.62 1 0.57 64.01 1 0.71 78.46
0,53
40.25
0.56 54.53 0.57
Fine tuning (train
(64) 51.40
68.58
0.52
58.04 L
0.68
74.85
0.56
62.12
77.96
0.52
39,09
0.35
54.85
0.55
Transductive tuning
(64).s 52,30
68.78
0.53
5881
9.69
74.71
0.56
63,89
0.71 78.45
0.52
40.33
0.56
55.60
0.56
Support hased intiializaiion (train vall Fine uuing timin Transductive tuning (tain
ResNet-12
72.96
0.55
67.60 0.71
83.27
58.16 0.57
ResNet-12
76.83
0.50
0.70
68.11
0.57
ResNet 12 64,50
0.55
0.56
McuOOt (Lee et al., 2019) tuain ResNet
64.09 a 80.00 0.45 6581 0.74
81.75 1.53
72.
4..2 62.5 0.6
vall
Activation to Pormmtter Quio 2018) WRN-28-10 59.50 0.41 73.74 0.19 (trai vail
LBO (Rusu et al., 2018) train in val)
WRN 25 10 61,76 = 0.08 77.59 0.12 66.33 0.05
81.44 1 0.09
Support based intiialzaaionn (tuis vall
WRN 10 58.47 =: 0.66 75.56 0.52 67.34 1 0.691
53.32 = 0.511 72.14 0.69 N5.21 0,49
45.08 = 1.61 60005 =: 0.60
Fize-tuiing (srin 4 val)
WRN-228-10
59.62 = 0.66 79.33 : 0.47
66.23
0.68
86.08 :: 0.47 70.07 = 0.67 N7.26
0.45 43.90 =: 0.5% 64-40 0.58
Tranduttive finestuing (cain + val)
WRN--28-10 68.11 = 0,69 80.36 = 0.50
72,87 + 6.71
86.15 :: 0,50 78.36 = 0.70 N7. 0.09 50.44 = 11.6 65.74 =: 0.60
0.74% and 86.67 0.46% for 1-shot 5-way and 5-shot 5-way respectively. These numbers are significantly lower for the 1-shot 5-way protocol on both datasets compared to their counterparts in Table 1. However, the 5-shot 5-way accuracy is marginally higher in this experiment than that in Table As noted in Remark 2, logits of the backbone are well-clustered and that is why they work better for few-shot scenarios.
17

---

Published as a conference paper at ICLR 2020
C.7 FREEZING THE BACKBONE RESTRICTS PERFORMANCE
The previous observation suggests that the network changes a lot in the fine-tuning phase. Freezing the backbone severely restricts the changes the network to only changes to the classifier. As consequence, the accuracy of freezing the backbone is 58.38 0.66 % and 75.46 + 0.52% on Mini-ImageNet and 67.06 + 0.69% and 83.20 + 0.51% % Tiered-ImageNe for 1-shot 5-way and 5-shot 5-way respectively. While the I-shot 5-way accuracies are much lower than their counterparts Table I, the gap in the 5-shot 5-way scenario smaller.
C.8 USING MIXUP DURING PRE-TRAINING
Mixup improves the few-shot accuracy by about 1%; the accuracy for WRN-28-10 trained without mixup is 67.06 + 0.71 % and 79.29 = 0.51 % Mini-ImageNet for 1-shot 5-way and 5-shot 5-way respectively.
C.9 MORE FEW-SHOT EPISODES
Fig. 1 suggests that the standard deviation of the accuracies achieved by few-shot algorithms high. Considering this randomness, evaluations were done over 10,000 few-shot episodes as well. The accuracies on Mini- are 67.77 + 0.21 % and 80.24 + 0.16 % and on 72.36 + 0.23 % and 85.70 = 0.16 for 1-shot 5-way and 5-shot 5-way respectively. The numbers are consistent with the ones for 1,000 few-shot episodes in Table 1, though the confidence intervals decreased the number of episodes sampled increased.
are
C.10 EVALUATION ON META- -DTASET
Table 4: Few-shot accuracies Meta-Dataset: Best results in each row are shown in bold, 600 few-shot episodes used compare the results reported in Triantafillou et al. (2019).
Dataset
Best performance in
Transductive Fine-tuning
Rank for Transductive
Triantafillou et al. (2019)
Fine- Fine-tuning (based on
Triantafillou et al. (2019))
ImageNet- Ik (ILSVRC)
51.01 + 1.05
55.57 + 1.02
Omniglot
63.00 + 1.35
79.59 + 0.98
Aircraft
68.69 + 1.26
67.26 1 0.98
1.5
Birds
68.79 = 1.01
74.26 t 0.82
Textures
69.05 = 0.90
77.35 + 0.74
VGG Flowers
86.86 + 0.75
88.14 + 0.63
1.5
Traffic Signs
66.79 + 1.31
55.98 + 1.32
2
MSCOCO
43.41 + 1.06
40.62 + 0.98
2.5
Average Rank
1.4375
We ran experiments on Meta- Meta-Dataset (Triantafillou et al., 2019), and compared the performance of transductive fine-tuning for meta- veta-training done on ImageNet- 1k (ILSVRC) Table Transductive fine-tuning is better, most times significantly, than state-of oftte-of-hhearr its average rank across all tasks is 1,4375 (calculated using the results reported in Triantafillou et al. (2019)). The Fungi and Quick Draw datasets were not included because of issues with getting the data; the link to access the dataset for the former does not seem work and the latter requires certain legal conditions which we are working obtaining.
6 out of 8 tasks in Meta-Dataset,,
The few-shot episode sampling was done the same way as described Triantafillou et al. (2019); except for the few-shot class sampling for ImageNet-II Ik (ILSVRC) and Omniglot, which was done uniformly over all few-shot classes (Triantafillou al. (2019) use hierarchical sampling technique to sample classes that are far from each other in the hierarchy, and hence easier to distinguish between). used for meta-training and few-shot fine-tuning are kept the same as the ones in Section 4 and are not tuned for these experiments.
The
18

---

Published as a conference paper at ICLR 2020
D FREQUENTLY ASKED QUESTIONS
1. Why has it not been noticed yet that this simple approach works well?
Non-transductive fine-tuning as a baseline has been considered before (Vinyals et al., 2016; Chen al., 2018). The fact that this is comparable to sate-of-the-art has probably gone unnoticed because of the following reasons:
Given that there are only few labeled support samples provided in the few-shot setting, initializing the classifier becomes important. The support-base basee initialization (cf. Section 3. motivated from the deep metric learning literature (Hu et al., 2015; Movshovitz-Attia et al., 2017: Qi et al., 2018: Gidaris Komodakis, 2018) classifies support samples correctly (for support shot of this may not be true for higher support shots). This initialization, as opposed to initializing the weights of the classifier randomly, was critical to performance in our experiments.
In our experience, existing meta-training methods, both gradient-based ones and metric- based ones, are difficult to tune for larger architectures. We speculate that this is the reason large part of the existing literature focuses on smaller backbone architectures. The few-shot learning literature has only recently started to move towards bigger backbone architectures (Oreshkin et al., 2018; Rusu et al., 2018). From Table 3 tuning gets better with a deeper backbone architecture. A similar observation was made by (Chen et al., 2018). The observation that we can use "simple" well-understood training techniques from standard supervised learning that scale up to large backbone architectures for few-shot classification is a key contribution of our paper.
that -rn-tranneetive fine-
Transductive methods have recently started to become popular in the few-shot learning literature (Nichol et al., 2018; Liu et al., 2018a). Because of the scarcity of labeled support samples, it is crucial to make use of the unlabeled query samples in the few-shot regime.
Our advocated baseline makes use of both good initialization and transduction, relatively new in the few-shot learning literature, which makes this simplistic approach go unrecognized till now.
Transductive fine-tuning works better than existing algorithms because of a big backhone architecture. One should compare the same backhone architectures the existing algo- rithms for a fair comparison.
The current literature is in the spirit of increasingly sophisticated approaches for modest perfor- mance gains, often with different architectures (cf. Table 1). This is why we out to establish baseline. Our simple baseline is comparable or better than existing approaches. The backbone we have used common in the recent few-shot learning literature (Rusu et al., 2018; Qiao et al., 2018) (cf. Table 1). Additionally, we have included results on smaller common backbone architec- tures, namely conv (64) x4 and ResNet- 12 in Appendix C.4, and some additional experiments in Appendix C.2. These experiments suggest that transductive fine-tuning is a sound baseline for a variety different backbone architectures. This indicates that we should take results on existing benchmarks with a grain of salt. Also see the response question 1 above.
3. There are missing entries in Tables 1 and 3. Is it still a fair comparison?
Tables 1 and 3 show all relevant published results by the original authors. Re-implementing existing algorithms to fill missing entries without access to original code is impractical and often yields results inferior to those published, which may be judged as unfair. The purpose a benchmark enable others test their method easily. This does not exist today due myriad performance-critica design choices often not detailed in the papers. In fact, missing entries in the table indicate the inadequate state of the current literature. Our work enables benchmarking relative to a simple, systematic baseline.
4. Fine-tuning for few-shot learning is not novel.
We do not claim novelty in this paper. Transductive fine-tuning is our advocated baseline for few-shot classification. It is combination of different techniques that are not novel. Yet, it performs better than existing algorithms on all few-shot protocols with fixed hyper-parameters. We emphasize that this indicates the need to re-interpret existing results on benchmarks and re-evaluate the status quo in the literature.
19

---

Published as a conference paper at ICLR 2020
5. Transductive fine-tuning has very high latency at inference time, this is not practical. Our goal establish a systematic baseline for accuracy, which might help judge the accuracy of few-shot learning algorithms in the future. The question of test-time latency is indeed important but we have not focused on it in this paper. Appendix C.3 provides results using a smaller backbone where we see that the WRN-16-4 network is about 20x slower than metric- besric-based approaches employing the same backbone while having significantly better accuracy. The latencies with WRN-28- 10 are larger (see the computational complexity section in Section 4.3) but with a bigger advantage in terms of accuracy.
There are other transductive methods used for few-shot classification (Nichol al., 2018; Liu et al., 2018a), that are expected to be slow as well.
Transductive fine-tuning does not make sense in the online setting when query samples are shown in a sequence.
Transductive fine-tuning can be performed even with a single test datum. Indeed, the network can specialize itself completely to classify this one datum. We explore a similar scenario in Section 4.3 and Fig. 2a, which discuss the performance transductive fine-tuning with query shot of (this means 5 query samples one from each class for 5-way evaluation). Note that the loss function in (8) leverages multiple query samples when available. It does not require that the query samples be balanced terms their ground-truth classes, particular, the loss function (8) is well-defined even for single test datum. For concerns about latency, see the question 5 above.
Having transductive approaches will incentivize hacking the query set.
There are already published methods that use transductive methods (Nichol et al., 2018; Liu et al., 20) and it a fundamental property of the transductive paradigm be dependent the query set, in addition to the support set. In order to prevent query set hacking, we will make the test episodes public which will enable consistent benchmarking, even for transductive methods.
8. Why is having the same hyper-parameters for different few-shot protocols so important?
practical few-shot learning algorithm should be able to handle any few-shot protocol. Having one model for each different scenario is unreasonable in the real- as the number of different scenarios is, in principle, infinite. Current algorithms not handle this well. single model which can handle any few-shot scenario is thus desirable.
9. Is this over-fitting to the test datum?
No, label the test datum not used in the loss function.
10. Can you give some intuition about the hardness metric? How did you come up with the formula?
The hardness metric the clustering loss where the labeled support samples form the centers of the class caas--pp clusters. The special form, namely, (( log 1-p(yla) (cf. (9)) allows an interpretation of log-odds. We used this form because it is sensitive the number of few-shot classes (cf. Fig. 3). Similar metrics, e.g., (( [- log p(v|=)] can also be used but they come with a few cavcats. Note that is casier for p(y) x)) to be large for small way because the normalization constant in softmax has fewer terms. For large way, p(y|z) could be smaller. This effect is better captured by our metric.
p(ylæ)
11. How does Fig. 3 look for algorithm X, Y, Z?
We compared two algorithms in Fig. 3, namely transductive fine-tuning and support- based initial- ization. Section 4.4 and the caption of Fig. 3 explains how the former algorithm is better. We will consider adding comparisons to other algorithms to this plot in the future.
20


Reviews:
[{"review_id": "rJejGJP_qH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a transductive learning baseline for few-shot image classification. The proposed approach includes a standard cross-entropy loss on the labeled support samples and a Shannon entropy loss on the unlabeled query samples. Despite its simplicity, the experimental results show that it can consistently outperform the state-of-the-art on four public few-shot datasets. In addition, they introduce a large-scale few-shot benchmark with 21K classes of ImageNet21K. Finally, they point out that accuracies from different episodes have high variance and develop another few-shot performance metric based on the hardness of each episode.\n\nPositive comments:\n1. The proposed transductive loss that minimizes entropy of query samples is novel in few-shot learning. Given limited labeled samples, finetuning with unlabeled query samples via proper loss is a good idea to tackle few-shot learning. \n2. The evaluation is thorough. A significant number of few-shot methods are compared on 4 exisiting few-shot benchmarks. An additional large-scale benchmark is also introduced to facilitate  the few-shot learning research. \n3. A novel evaluation metric is proposed to evaluate few-shot learning methods under different difficulties level. Although I am convinced by the importance of such metric, it is interesting to supplement the averaged accuracy because it tells how the methods work under easy and difficult classes. \n\nNegative comments:\n1. The folloing important reference of the Shannon entropy on unlabeled data is missing. In fact, I suggest the authors to extend Section 3.2 a bit more because this is the main technic contribution. \nSemi-supervised Learningby Entropy Minimization.  Grandvalet et al. NIPS 2015\n2. I am not convinced by the necessity of the proposed hardness metric. The main argument of this paper is that accuracies over episodes have high variance. But isn't it expected that different  episode can include samples with different difficulties, leading to high variance of accuracies? I do not think it is realistic to have one algorithm that achieves similar accuracies on both easy and difficult tasks. The authors also fail to evaluate different methods with the proposed metric and show if this metric makes the ranking of algorithms different. Moreover, I find Figure 3 hard to interpret because there are too much information in it, including different colors, a lot of markers and lines. Why is the range of hardness 1-3 for some datasets and 1-5 for other datasets? I believe writting of Section 4.4 could be further improved.\n\nOverall, I think this paper has significant contributions of proposing a novel few-shot baseline that establishes a new state-of-the-art and would recommend weak accept.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "SJlVVzkRFB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a fine-tune-based few-shot classification baseline, which has been validated effectively on several datasets, including Mini-Imagenet, Tiered-Imagenet, CIFAR-FS, FC-100, and Imagenet-21k. In addition to the method, the authors also provide concrete experimental setting and new evaluation proposals.\n\n1. The authors propose to use the logits instead of embedding as the main bridge between the pre-trained model and the meta-learning model. Does it mean we represent novel classes based on the properties of the meta-train classes? If so, does this method requires more meta-train classes to enrich the representation ability? How will the method perform when working on few-shot learning problems with a large distribution shift?\n\n2. To make a fair comparison: instead of citing the values in the published papers directly and comparing different methods with different architectures, the authors should also apply the pre-trained model with the famous baselines, such as Matching Network, Prototypical Network, and MAML. Now there exists a very lap gap between the Matching Network values and the newly proposed one. For example, fine-tune the Matching Network on the pre-trained backbone in both train and train+val settings. Therefore, it is more clear to show the improvement of the proposed baseline models. Q/A 2-3 in appendix D do not fully solve this problem.\n\n3. Considering the randomness of the sampled few-shot tasks, the authors can consider evaluating over more episodes (e.g., 10,000 trials) than 1000 in the paper.\n\n4. It's better for the authors to emphasize and differentiate the transductive fine-tune and the inductive counterpart in the paper. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HJgnmTMTYS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper provided a baseline method for few-shot learning. It utilizes a simple but effective approach via a transductive fine-tuning. The experimental results on several benchmarks show the improvements over state-of-the-art approaches. \n\nIt is a comprehensive study of the methods and datasets in this domain. The motivation, experimental details and result analysis are clear to me. Overall, the paper is well written and the author is very transparent to show what they have. \n\nThe only drawback of this paper is it does not provide insight/explanation. How can the simple baseline work sowell? Is this because of some bias from the datasets? I also suggest that the author can try their method on some new dataset, like Meta-Dataset (Triantafillou et al. 2019). \n\nI agreed with the author that the paper is not novel. However, I think the acceptance of the paper could benefit the community and I encourage the author can try this on some new benchmark. Therefore, I made my recommendation. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a simple baseline for few-shot image classification in the transductive setting, which includes a standard cross-entropy loss on the labeled support samples and a conditional entropy loss on the unlabeled query samples.

Both losses are known in the literature (the seminal work of entropy minimization by Bengio should be cited properly). However, reviewers are positive about this paper, acknowledging the significant contributions of a novel few-shot baseline that establishes a new state-of-the-art on well-known public few-shot datasets as well as on the introduced large-scale benchmark ImageNet21K. The comprehensive study of the methods and datasets in this domain will benefit the research practices in this area.

Therefore, I make an acceptance recommendation.
Decision: Accept (Poster)