A Baseline for Few-Shot Image Classification

1 INTRODUCTION . As image classification systems begin to tackle more and more classes , the cost of annotating a massive number of images and the difficulty of procuring images of rare categories increases . This has fueled interest in few-shot learning , where only few labeled samples per class are available for training . Fig . 1 displays a snapshot of the state-of-the-art . We estimated this plot by using published ∗Work done while at Amazon Web Services numbers for the estimate of the mean accuracy , the 95 % confidence interval of this estimate and the number of few-shot episodes . For MAML ( Finn et al. , 2017 ) and MetaOpt SVM ( Lee et al. , 2019 ) , we use the number of episodes in the author ’ s Github implementation . The field appears to be progressing steadily albeit slowly based on Fig . 1 . However , the variance of the estimate of the mean accuracy is not the same as the variance of the accuracy . The former can be zero ( e.g. , asymptotically for an unbiased estimator ) , yet the latter could be arbitrarily large . The variance of the accuracies is extremely large in Fig . 1 . This suggests that progress in the past few years may be less significant than it seems if one only looks at the mean accuracies . To compound the problem , many algorithms report results using different models for different number of ways ( classes ) and shots ( number of labeled samples per class ) , with aggressive hyper-parameter optimization.1 Our goal is to develop a simple baseline for few-shot learning , one that does not require specialized training depending on the number of ways or shots , nor hyper-parameter tuning for different protocols . The simplest baseline we can think of is to pre-train a model on the meta-training dataset using the standard cross-entropy loss , and then fine-tune on the few-shot dataset . Although this approach is basic and has been considered before ( Vinyals et al. , 2016 ; Chen et al. , 2018 ) , it has gone unnoticed that it outperforms many sophisticated few-shot algorithms . Indeed , with a small twist of performing fine-tuning transductively , this baseline outperforms all state-of-the-art algorithms on all standard benchmarks and few-shot protocols ( cf . Table 1 ) . Our contribution is to develop a transductive fine-tuning baseline for few-shot learning , our approach works even for a single labeled example and a single test datum per class . Our baseline outperforms the state-of-the-art on a variety of benchmark datasets such as Mini-ImageNet ( Vinyals et al. , 2016 ) , Tiered-ImageNet ( Ren et al. , 2018 ) , CIFAR-FS ( Bertinetto et al. , 2018 ) and FC-100 ( Oreshkin et al. , 2018 ) , all with the same hyper-parameters . Current approaches to few-shot learning are hard to scale to large datasets . We report the first few-shot learning results on the ImageNet-21k dataset ( Deng et al. , 2009 ) which contains 14.2 million images across 21,814 classes . The rare classes in ImageNet-21k form a natural benchmark for few-shot learning . The empirical performance of this baseline , should not be understood as us suggesting that this is the right way of performing few-shot learning . We believe that sophisticated meta-training , understanding taxonomies and meronomies , transfer learning , and domain adaptation are necessary for effective few-shot learning . The performance of the simple baseline however indicates that we need to interpret existing results2 with a grain of salt , and be wary of methods that tailor to the benchmark . To facilitate that , we propose a metric to quantify the hardness of few-shot episodes and a way to systematically report performance for different few-shot protocols . 2 PROBLEM DEFINITION AND RELATED WORK . We first introduce some notation and formalize the few-shot image classification problem . Let ( x , y ) denote an image and its ground-truth label respectively . The training and test datasets are Ds = { ( xi , yi ) } Nsi=1 and Dq = { ( xi , yi ) } Nq i=1 respectively , where yi ∈ Ct for some set of classes Ct . In the few-shot learning literature , training and test datasets are referred to as support and query datasets respectively , and are collectively called a few-shot episode . The number of ways , or classes , is |Ct| . The set { xi | yi = k , ( xi , yi ) ∈ Ds } is the support of class k and its cardinality is s support shots ( this is non-zero and is generally shortened to shots ) . The number s is small in the few-shot setting . The set { xi | yi = k , ( xi , yi ) ∈ Dq } is the query of class k and its cardinality is q query shots . The goal is to learn a function F to exploit the training set Ds to predict the label of a test datum x , 1For instance , Rusu et al . ( 2018 ) tune for different few-shot protocols , with parameters changing by up to six orders of magnitude ; Oreshkin et al . ( 2018 ) use a different query shot for different few-shot protocols . 2For instance , Vinyals et al . ( 2016 ) ; Ravi & Larochelle ( 2016 ) use different versions of Mini-ImageNet ; Oreshkin et al . ( 2018 ) report results for meta-training on the training set while Qiao et al . ( 2018 ) use both the training and validation sets ; Chen et al . ( 2018 ) use full-sized images from the parent ImageNet-1k dataset ( Deng et al. , 2009 ) ; Snell et al . ( 2017 ) ; Finn et al . ( 2017 ) ; Oreshkin et al . ( 2018 ) ; Rusu et al . ( 2018 ) use different model architectures of varying sizes , which makes it difficult to disentangle the effect of their algorithmic contributions . where ( x , y ) ∈ Dq , by ŷ = F ( x ; Ds ) . ( 1 ) Typical approaches for supervised learning replace Ds above with a statistic , θ∗ = θ∗ ( Ds ) that is , ideally , sufficient to classify Ds , as measured by , say , the cross-entropy loss θ∗ ( Ds ) = arg min θ 1 Ns ∑ ( x , y ) ∈Ds − log pθ ( y|x ) , ( 2 ) where pθ ( ·|x ) is the probability distribution on Ct as predicted by the model in response to input x . When presented with a test datum , the classification rule is typically chosen to be of the form Fθ∗ ( x ; Ds ) , arg max k pθ∗ ( k|x ) , ( 3 ) where Ds is represented by θ∗ . This form of the classifier entails a loss of generality unless θ∗ is a sufficient statistic , pθ∗ ( y|x ) = p ( y|x ) , which is of course never the case , especially given few labeled data in Ds . However , it conveniently separates training and inference phases , never having to revisit the training set . This might be desirable in ordinary image classification , but not in few-shot learning . We therefore adopt the more general form of F in ( 1 ) . If we call the test datum x = x Ns+1 , then we can obtain the general form of the classifier by ŷ = F ( x ; Ds ) = arg min y Ns+1 min θ 1 Ns + 1 Ns+1∑ i=1 − log pθ ( yi|xi ) . ( 4 ) In addition to the training set , one typically also has a meta-training set , Dm = { ( xi , yi ) } Nmi=1 , where yi ∈ Cm , with set of classes Cm disjoint from Ct . The goal of meta-training is to use Dm to infer the parameters of the few-shot learning model : θ̂ ( Dm ; ( Ds , Dq ) ) = arg minθ 1 Nm ∑ ( x , y ) ∈Dm ` ( y , Fθ ( x ; ( Ds , Dq ) ) ) , where meta-training loss ` depends on the method . 2.1 RELATED WORK . Learning to learn : The meta-training loss is designed to make few-shot training efficient ( Utgoff , 1986 ; Schmidhuber , 1987 ; Baxter , 1995 ; Thrun , 1998 ) . This approach partitions the problem into a base-level that performs standard supervised learning and a meta-level that accrues information from the base-level . Two main approaches have emerged to do so . Gradient-based approaches : These approaches treat the updates of the base-level as a learnable mapping ( Bengio et al. , 1992 ) . This mapping can be learnt using temporal models ( Hochreiter et al. , 2001 ; Ravi & Larochelle , 2016 ) , or one can back-propagate the gradients across the base-level updates ( Maclaurin et al. , 2015 ; Finn et al. , 2017 ) . It is challenging to perform this dual or bi-level optimization , respectively . These approaches have not been shown to be competitive on large datasets . Recent approaches learn the base-level in closed-form using SVMs ( Bertinetto et al. , 2018 ; Lee et al. , 2019 ) which restricts the capacity of the base-level although it alleviates the optimization problem . Metric-based approaches : A majority of the state-of-the-art algorithms are metric-based approaches . These approaches learn an embedding that can be used to compare ( Bromley et al. , 1994 ; Chopra et al. , 2005 ) or cluster ( Vinyals et al. , 2016 ; Snell et al. , 2017 ) query samples . Recent approaches build upon this idea with increasing levels of sophistication in learning the embedding ( Vinyals et al. , 2016 ; Gidaris & Komodakis , 2018 ; Oreshkin et al. , 2018 ) , creating exemplars from the support set and picking a metric for the embedding ( Gidaris & Komodakis , 2018 ; Allen et al. , 2018 ; Ravichandran et al. , 2019 ) . There are numerous hyper-parameters involved in implementing these approaches which makes it hard to evaluate them systematically ( Chen et al. , 2018 ) . Transductive learning : This approach is more efficient at using few labeled data than supervised learning ( Joachims , 1999 ; Zhou et al. , 2004 ; Vapnik , 2013 ) . The idea is to use information from the test datum x to restrict the hypothesis space while searching for the classifier F ( x , Ds ) at test time . Our approach is closest to this line of work . We train a model on the meta-training set Dm and initialize a classifier using the support set Ds . The parameters are then fine-tuned to adapt to the new test datum x . There are recent papers in few-shot learning such as Nichol et al . ( 2018 ) ; Liu et al . ( 2018a ) that are motivated from transductive learning and exploit the unlabeled query samples . The former updates batch-normalization parameters using query samples while the latter uses label propagation to estimate labels of all query samples at once . Semi-supervised learning : We penalize the Shannon Entropy of the predictions on the query samples at test time . This is a simple technique in the semi-supervised learning literature , closest to Grandvalet & Bengio ( 2005 ) . Modern augmentation techniques such as Miyato et al . ( 2015 ) ; Sajjadi et al . ( 2016 ) ; Dai et al . ( 2017 ) or graph-based approaches ( Kipf & Welling , 2016 ) can also be used with our approach ; we used the entropic penalty for the sake of simplicity . Semi-supervised few-shot learning is typically formulated as having access to extra unlabeled data during meta-training or few-shot training ( Garcia & Bruna , 2017 ; Ren et al. , 2018 ) . This is different from our approach which uses the unlabeled query samples for transductive learning . Initialization for fine-tuning : We use recent ideas from the deep metric learning literature ( Hu et al. , 2015 ; Movshovitz-Attias et al. , 2017 ; Qi et al. , 2018 ; Chen et al. , 2018 ; Gidaris & Komodakis , 2018 ) to initialize the meta-trained model for fine-tuning . These works connect the softmax cross-entropy loss with cosine distance and are discussed further in Section 3.1 .

Summary: This paper introduces a transductive learning baseline for few-shot image classification. The proposed approach includes a standard cross-entropy loss on the labeled support samples and a Shannon entropy loss on the unlabeled query samples. Despite its simplicity, the experimental results show that it can consistently outperform the state-of-the-art on four public few-shot datasets. In addition, they introduce a large-scale few-shot benchmark with 21K classes of ImageNet21K. Finally, they point out that accuracies from different episodes have high variance and develop another few-shot performance metric based on the hardness of each episode.