A Closer Look at Few-shot Classification

1 INTRODUCTION . Deep learning models have achieved state-of-the-art performance on visual recognition tasks such as image classification . The strong performance , however , heavily relies on training a network with abundant labeled instances with diverse visual variations ( e.g. , thousands of examples for each new class even with pre-training on large-scale dataset with base classes ) . The human annotation cost as well as the scarcity of data in some classes ( e.g. , rare species ) significantly limit the applicability of current vision systems to learn new visual concepts efficiently . In contrast , the human visual systems can recognize new classes with extremely few labeled examples . It is thus of great interest to learn to generalize to new classes with a limited amount of labeled examples for each novel class . The problem of learning to generalize to unseen classes during training , known as few-shot classification , has attracted considerable attention Vinyals et al . ( 2016 ) ; Snell et al . ( 2017 ) ; Finn et al . ( 2017 ) ; Ravi & Larochelle ( 2017 ) ; Sung et al . ( 2018 ) ; Garcia & Bruna ( 2018 ) ; Qi et al . ( 2018 ) . One promising direction to few-shot classification is the meta-learning paradigm where transferable knowledge is extracted and propagated from a collection of tasks to prevent overfitting and improve generalization . Examples include model initialization based methods Ravi & Larochelle ( 2017 ) ; Finn et al . ( 2017 ) , metric learning methods Vinyals et al . ( 2016 ) ; Snell et al . ( 2017 ) ; Sung et al . ( 2018 ) , and hallucination based methods Antoniou et al . ( 2018 ) ; Hariharan & Girshick ( 2017 ) ; Wang et al . ( 2018 ) . Another line of work Gidaris & Komodakis ( 2018 ) ; Qi et al . ( 2018 ) also demonstrates promising results by directly predicting the weights of the classifiers for novel classes . Limitations . While many few-shot classification algorithms have reported improved performance over the state-of-the-art , there are two main challenges that prevent us from making a fair comparison and measuring the actual progress . First , the discrepancy of the implementation details among multiple few-shot learning algorithms obscures the relative performance gain . The performance of baseline approaches can also be significantly under-estimated ( e.g. , training without data augmentation ) . Second , while the current evaluation focuses on recognizing novel class with limited training examples , these novel classes are sampled from the same dataset . The lack of domain shift between the base and novel classes makes the evaluation scenarios unrealistic . Our work . In this paper , we present a detailed empirical study to shed new light on the few-shot classification problem . First , we conduct consistent comparative experiments to compare several representative few-shot classification methods on common ground . Our results show that using a deep backbone shrinks the performance gap between different methods in the setting of limited domain differences between base and novel classes . Second , by replacing the linear classifier with a distance-based classifier as used in Gidaris & Komodakis ( 2018 ) ; Qi et al . ( 2018 ) , the baseline method is surprisingly competitive to current state-of-art meta-learning algorithms . Third , we introduce a practical evaluation setting where there exists domain shift between base and novel classes ( e.g. , sampling base classes from generic object categories and novel classes from fine-grained categories ) . Our results show that sophisticated few-shot learning algorithms do not provide performance improvement over the baseline under this setting . Through making the source code and model implementations with a consistent evaluation setting publicly available , we hope to foster future progress in the field.1 Our contributions . 1 . We provide a unified testbed for several different few-shot classification algorithms for a fair comparison . Our empirical evaluation results reveal that the use of a shallow backbone commonly used in existing work leads to favorable results for methods that explicitly reduce intra-class variation . Increasing the model capacity of the feature backbone reduces the performance gap between different methods when domain differences are limited . 2 . We show that a baseline method with a distance-based classifier surprisingly achieves competitive performance with the state-of-the-art meta-learning methods on both mini-ImageNet and CUB datasets . 3 . We investigate a practical evaluation setting where base and novel classes are sampled from different domains . We show that current few-shot classification algorithms fail to address such domain shifts and are inferior even to the baseline method , highlighting the importance of learning to adapt to domain differences in few-shot learning . 2 RELATED WORK . Given abundant training examples for the base classes , few-shot learning algorithms aim to learn to recognizing novel classes with a limited amount of labeled examples . Much efforts have been devoted to overcome the data efficiency issue . In the following , we discuss representative few-shot learning algorithms organized into three main categories : initialization based , metric learning based , and hallucination based methods . Initialization based methods tackle the few-shot learning problem by “ learning to fine-tune ” . One approach aims to learn good model initialization ( i.e. , the parameters of a network ) so that the classifiers for novel classes can be learned with a limited number of labeled examples and a small number of gradient update steps Finn et al . ( 2017 ; 2018 ) ; Nichol & Schulman ( 2018 ) ; Rusu et al . ( 2019 ) . Another line of work focuses on learning an optimizer . Examples include the LSTM-based meta-learner for replacing the stochastic gradient decent optimizer Ravi & Larochelle ( 2017 ) and the weight-update mechanism with an external memory Munkhdalai & Yu ( 2017 ) . While these initialization based methods are capable of achieving rapid adaption with a limited number of training examples for novel classes , our experiments show that these methods have difficulty in handling domain shifts between base and novel classes . Distance metric learning based methods address the few-shot classification problem by “ learning to compare ” . The intuition is that if a model can determine the similarity of two images , it can classify an unseen input image with the labeled instances Koch et al . ( 2015 ) . To learn a sophisticated comparison models , meta-learning based methods make their prediction conditioned on distance or 1https : //github.com/wyharveychen/CloserLookFewShot metric to few labeled instances during the training process . Examples of distance metrics include cosine similarity Vinyals et al . ( 2016 ) , Euclidean distance to class-mean representation Snell et al . ( 2017 ) , CNN-based relation module Sung et al . ( 2018 ) , ridge regression Bertinetto et al . ( 2019 ) , and graph neural network Garcia & Bruna ( 2018 ) . In this paper , we compare the performance of three distance metric learning methods . Our results show that a simple baseline method with a distancebased classifier ( without training over a collection of tasks/episodes as in meta-learning ) achieves competitive performance with respect to other sophisticated algorithms . Besides meta-learning methods , both Gidaris & Komodakis ( 2018 ) and Qi et al . ( 2018 ) develop a similar method to our Baseline++ ( described later in Section 3.2 ) . The method in Gidaris & Komodakis ( 2018 ) learns a weight generator to predict the novel class classifier using an attentionbased mechanism ( cosine similarity ) , and the Qi et al . ( 2018 ) directly use novel class features as their weights . Our Baseline++ can be viewed as a simplified architecture of these methods . Our focus , however , is to show that simply reducing intra-class variation in a baseline method using the base class data leads to competitive performance . Hallucination based methods directly deal with data deficiency by “ learning to augment ” . This class of methods learns a generator from data in the base classes and use the learned generator to hallucinate new novel class data for data augmentation . One type of generator aims at transferring appearance variations exhibited in the base classes . These generators either transfer variance in base class data to novel classes Hariharan & Girshick ( 2017 ) , or use GAN models Antoniou et al . ( 2018 ) to transfer the style . Another type of generators does not explicitly specify what to transfer , but directly integrate the generator into a meta-learning algorithm for improving the classification accuracy Wang et al . ( 2018 ) . Since hallucination based methods often work with other few-shot methods together ( e.g . use hallucination based and metric learning based methods together ) and lead to complicated comparison , we do not include these methods in our comparative study and leave it for future work . Domain adaptation techniques aim to reduce the domain shifts between source and target domain Pan et al . ( 2010 ) ; Ganin & Lempitsky ( 2015 ) , as well as novel tasks in a different domain Hsu et al . ( 2018 ) . Similar to domain adaptation , we also investigate the impact of domain difference on fewshot classification algorithms in Section 4.5 . In contrast to most domain adaptation problems where a large amount of data is available in the target domain ( either labeled or unlabeled ) , our problem setting differs because we only have very few examples in the new domain . Very recently , the method in Dong & Xing ( 2018 ) addresses the one-shot novel category domain adaptation problem , where in the testing stage both the domain and the category to classify are changed . Similarly , our work highlights the limitations of existing few-shot classification algorithms problem in handling domain shift . To put these problem settings in context , we provided a detailed comparison of setting difference in the appendix A1 . 3 OVERVIEW OF FEW-SHOT CLASSIFICATION ALGORITHMS . In this section , we first outline the details of the baseline model ( Section 3.1 ) and its variant ( Section 3.2 ) , followed by describing representative meta-learning algorithms ( Section 3.3 ) studied in our experiments . Given abundant base class labeled data Xb and a small amount of novel class labeled data Xn , the goal of few-shot classification algorithms is to train classifiers for novel classes ( unseen during training ) with few labeled examples . 3.1 BASELINE . Our baseline model follows the standard transfer learning procedure of network pre-training and fine-tuning . Figure 1 illustrates the overall procedure . Training stage . We train a feature extractor fθ ( parametrized by the network parameters θ ) and the classifier C ( ·|Wb ) ( parametrized by the weight matrix Wb ∈ Rd×c ) from scratch by minimizing a standard cross-entropy classification loss Lpred using the training examples in the base classes xi ∈ Xb . Here , we denote the dimension of the encoded feature as d and the number of output classes as c. The classifier C ( .|Wb ) consists of a linear layer W > b fθ ( xi ) followed by a softmax function σ . Fine-tuning stage . To adapt the model to recognize novel classes in the fine-tuning stage , we fix the pre-trained network parameter θ in our feature extractor fθ and train a new classifier C ( .|Wn ) ( parametrized by the weight matrix Wn ) by minimizing Lpred using the few labeled of examples ( i.e. , the support set ) in the novel classes Xn . 3.2 BASELINE++ . In addition to the baseline model , we also implement a variant of the baseline model , denoted as Baseline++ , which explicitly reduces intra-class variation among features during training . The importance of reducing intra-class variations of features has been highlighted in deep metric learning Hu et al . ( 2015 ) and few-shot classification methods Gidaris & Komodakis ( 2018 ) . The training procedure of Baseline++ is the same as the original Baseline model except for the classifier design . As shown in Figure 1 , we still have a weight matrix Wb ∈Rd×c of the classifier in the training stage and a Wn in the fine-tuning stage in Baseline++ . The classifier design , however , is different from the linear classifier used in the Baseline . Take the weight matrix Wb as an example . We can write the weight matrix Wb as [ w1 , w2 , ... wc ] , where each class has a d-dimensional weight vector . In the training stage , for an input feature fθ ( xi ) where xi ∈ Xb , we compute its cosine similarity to each weight vector [ w1 , · · · , wc ] and obtain the similarity scores [ si,1 , si,2 , · · · , si , c ] for all classes , where si , j = fθ ( xi ) > w j/‖ fθ ( xi ) ‖‖w j‖ . We can then obtain the prediction probability for each class by normalizing these similarity scores with a softmax function . Here , the classifier makes a prediction based on the cosine distance between the input feature and the learned weight vectors representing each class . Consequently , training the model with this distance-based classifier explicitly reduce intra-class variations . Intuitively , the learned weight vectors [ w1 , · · · , wc ] can be interpreted as prototypes ( similar to Snell et al . ( 2017 ) ; Vinyals et al . ( 2016 ) ) for each class and the classification is based on the distance of the input feature to these learned prototypes . The softmax function prevents the learned weight vectors collapsing to zeros . We clarify that the network design in Baseline++ is not our contribution . The concept of distancebased classification has been extensively studied in Mensink et al . ( 2012 ) and recently has been revisited in the few-shot classification setting Gidaris & Komodakis ( 2018 ) ; Qi et al . ( 2018 ) .

Summary: This paper gives a nice overview of existing works on few-shot learning. It groups them into some intuitive categories and meanwhile distills a common framework (Figure 2) employed by the methods. Moreover, the authors selected four of them, along with two baselines, to experimentally compare their performances under a cleaned experiment protocol.