Fast Task Adaptation for Few-Shot Learning

Few-shot classification is a challenging task due to the scarcity of training examples for each class . The key lies in generalization of prior knowledge learned from large-scale base classes and fast adaptation of the classifier to novel classes . In this paper , we introduce a two-stage framework . In the first stage , we attempt to learn task-agnostic feature on base data with a novel Metric-Softmax loss . The Metric-Softmax loss is trained against the whole label set and learns more discriminative feature than episodic training . Besides , the Metric-Softmax classifier can be applied to base and novel classes in a consistent manner , which is critical for the generalizability of the learned feature . In the second stage , we design a task-adaptive transformation which adapts the classifier to each few-shot setting very fast within a few tuning epochs . Compared with existing fine-tuning scheme , the scarce examples of novel classes are exploited more effectively . Experiments show that our approach outperforms current state-of-the-arts by a large margin on the commonly used mini-ImageNet and CUB-200-2011 benchmarks . 1 INTRODUCTION . In recent years , deep learning models have achieved great success in many visual tasks such as image classification ( Krizhevsky et al. , 2012 ) , object detection ( Girshick , 2015 ) and semantic segmentation ( Long et al. , 2015 ) . In general , training a deep network needs massively labeled instances which leads to expensive manual annotation cost . In contrast , humans can learn novel concepts from only one or a few examples . Analogously , few-shot learning aims to recognize unseen instances by accessing just a small number of labeled images in each class . Unfortunately , naive methods such as re-training or fine-tuning the model on novel data would severely suffer from overfitting and provide poor result ( Finn et al. , 2017 ) . The major difficulty lies in the exceedingly limited data , which can hardly represent the class distribution . In the task of few-shot classification , we are given three datasets , namely training set , support set and query set . The training set ( also known as base dataset ) normally contains large-scale labeled data for learning of prior knowledge . The support set and query set contain a small number of examples which are drawn from unseen classes disjoint with the training set . We are expected to classify the query images , leveraging the prior knowledge from the base training set and the limited cue from the support images . If the support set contains k annotated images for each of m unique classes , then this task is called m-way k-shot classification . Previous works on few-shot learning can be roughly divided into two categories , namely metalearning based and metric learning based . The former typically learns a meta-learner model on base data and generalizes it to novel unseen data . Usually , a recurrent neural network ( Santoro et al. , 2016 ) or long short-term memory network ( Hochreiter & Schmidhuber , 1997 ) is utilized to learn a memory network to store knowledge . The latter attempts to learn a feature embedding where samples of the same class are closer to each other than samples of different classes . To make training on base data and inference on novel data consistent , a so-called episodic training strategy is widely employed ( Vinyals et al. , 2016 ; Snell et al. , 2017 ; Sung et al. , 2018 ) . There are two limitations in metric learning based approaches . One is that episodic training only considers local sample similarity in current data batch . As shown in Snell et al . ( 2017 ) , the limited number of classes pertraining episode is harmful for discriminative feature learning , while simply increasing it leads to better performance . However , in a sampled episode , a large number of classes is infeasible due to limited GPU memory . The other limitation is that most existing methods only utilize the labeled support images in novel classes as templates for feature similarity comparison . We argue that the support images could be better exploited to improve the performance . In this work , we introduce a two-stage framework named Fast Task Adaption ( FTA ) . As illustrated in Figure 1 , the few-shot classification task is decomposed into two steps , i.e . task-agnostic feature learning on base data and adaptation of the classifier to each few-shot learning task . To learn more general and discriminative feature on base data , we propose a novel Metric-Softmax loss , which is an improvement of the Softmax loss . Specifically , we replace the score computing in softmax with a Gaussian kernel-based radial basis function . The Metric-Softmax loss is trained against the full base classes , overcoming the limitation of small number of classes in episodic training . Moreover , the Metric-Softmax classifier can be applied to base and novel classes in the same way . The consistency between training and inference is well preserved . After task-agnostic feature is learned , we design a novel task-adaptive transformation to adapt the classifier to current few-shot learning task . It is essentially an affine transformation applied to features of novel images . Notably , compared with fine-tuning re-initialized weights , it exhibits the desirable characteristics of fast convergence and superior performance . In this way , the aforementioned weaknesses of existing metric learning based methods are well addressed , leading to significant accuracy improvement . Experimental results on the mini-ImageNet and CUB-200-2011 datasets show that our approach consistently outperforms current state-of-the-art methods with respect to various backbone networks . The main contributions of this paper can be summarized as follows . • We propose the Metric-Softmax loss , which learns highly discriminative feature and ensures the consistency between training and inference as well . • We design a novel task-adaptive transformation to adapt the classifier to novel classes efficiently and effectively . • We improve few-shot classification significantly and achieve state-of-the-art performance on two common few-shot learning benchmarks . 2 RELATED WORK . In this section , we briefly review the previous approaches relevant to ours and current state-of-the-art methods we compare with in experiments . Meta-learning based methods aim to learn task-agnostic knowledge from a sequence of similar tasks which have enough training data and apply it to new tasks for fast adaption . Ravi & Larochelle ( 2017 ) utilize an LSTM-based network to control the parameter optimization process , which results in better generalization than SGD when training on a few labeled samples . Qiao et al . ( 2018 ) propose to adapt a pre-trained network to novel classes by directly predicting the parameters from the activations . MAML ( Finn et al. , 2017 ) learns sensitive and general initial parameters , which can fast adapt to a novel task with only one or a few steps of gradient-descent update . MTL ( Sun et al. , 2019 ) learns to adapt a deep neural network for few-shot learning tasks by learning scaling and shifting functions of weights for each task . Cai et al . ( 2018 ) use a contextual learner to predict the parameters of an embedding network for unlabeled data by using memory slots . Mishra et al . ( 2018 ) propose a meta-learner architecture that combines temporal convolutions and soft attention . Metric learning based methods aim to learn a feature embedding that preserves the class neighborhood structure . Specifically , samples of the same class are closer than samples of different classes in the learned embedding space . Matching networks ( Vinyals et al. , 2016 ) combine attention and memory together by learning a network that maps a small labeled support set and an unlabeled example to its label . They also first introduce the episodic training strategy , where the training process mimics the test scenario based on support-query metric learning . Prototypical network ( Snell et al. , 2017 ) uses the mean feature of each class as its corresponding prototype representation to learn a similarity metric space . Relation network ( Sung et al. , 2018 ) learns a deep non-linear distance metric by considering the relation between query images and support images . Satorras & Estrach ( 2018 ) propose a graph neural network to learn the feature similarity between query and support set . Our approach belongs to this category . Compared with existing methods , our major improvements are the Metric-Softmax loss for generic feature learning and the task-adaptive transformation for fast task adaptation . Other methods . Zhang et al . ( 2018 ) propose a GAN-based approach to help few-shot classifiers to learn sharper decision boundary . Si et al . ( 2019 ) propose a progressive cluster purification method for transductive few-shot learning . Li et al . ( 2019 ) propose a deep nearest neighbor neural network for few-shot learning . Chen et al . ( 2019 ) give a consistent comparative analysis of several representative few-shot classification algorithms and show that the depth of backbone network matters . Besides , they attempt to fine-tune a Softmax classifier on the support set . On the contrary , we first apply an affine transformation to features of the support images , and compute the class centroids of the transformed features as the classfier ’ s weights . Concretely , we learn the affine transforming matrix rather than directly fine-tune the randomly initialized weights . Optimizing the transformation is computationally more stable even when one example per-class is available . 3 METHOD . As shown in Figure 1 , the proposed FTA framework is composed of two stages , namely task-agnostic feature learning stage and fast task adaptation stage . In the first stage , we train the network on the base data with the proposed Metric-Softmax loss . After that , we fix the parameters of the network backbone and use it as a feature extractor . In the second stage , we learn a task-adaptive classifier , which is also a Metric-Softmax classifier with its weights replaced with affine transformed features of the support images . The affine transformation g is named Task-Adaptive Transformation ( TAT ) , which is learned on the scarce support set . With proper initialization , training of g converges very fast within a few epochs . After training finishes , g is applied to both support and query images . In this section , we first elaborately describe the proposed Metric-Softmax loss . Since it is an improvement of the Softmax loss , we first give a brief review of the Softmax loss and reveal the issue of discrepancy between training and inference . Then we explain how to address this issue with the Metric-Softmax loss . Finally , we introduce the task-adaptive transformation for fast task adaptation . 3.1 SOFTMAX CLASSIFIER . Feature learning with the Softmax loss has achieved great success in many feature ranking tasks like person re-identification ( Zhang et al. , 2017 ) and face verification ( Wen et al. , 2016 ) . Given an image X of base classes , a backbone network f is applied to project X into feature space h = f ( X ) , h ∈ Rd . Then , the logits z are computed by feeding h into an affine transformation : z = W Th + b , ( 1 ) where W ∈ Rd×c and b ∈ Rc are the transforming weights . c is the number of classes in base data . The Softmax classifier predicts the probability score that the image belongs to class i by si = softmax ( z ) i = exp ( zi ) ∑c j exp ( zj ) ( 2 ) The Softmax loss is an assembly of the Softmax classifier and cross-entropy loss . Suppose the one-hot category label of the image is y ∈ { 0 , 1 } c , the training cross-entropy loss is defined as L = − c∑ j yj log ( sj ) ( 3 ) During inference , if a test image belongs to one of the training classes , its label can be predicted by ŷ = arg max i si ( 4 ) In this case , the scores in both training and inference are computed by an affine transformation and softmax function applied to the feature vector h. However , in the context of few-shot learning , the query image for testing belongs to a novel class unseen in training . A common way to classify the novel query image is to compare its similarities to the few labeled support images , which share the same label space with the query image . Concretely , for m-way k-shot classification , we can extract features of the support images , and compute the mean feature for each class by h̄i = 1 k k∑ j f ( Xij ) ( 5 ) Here , Xij denotes the j-th image of class i in the support set . The label of a query image X can be predicted by ŷ = arg min i ||f ( X ) − h̄i|| ( 6 ) Rather than an inner-product between h and column vectors of the weight matrix ( zi = W T : ,ih+bi ) followed by softmax function in the previous case , the prediction is based on the euclidean distance between the query and support features . We argue that this discrepancy hurts the transferability of features pre-trained on base data to novel data .

Summary: This paper develops a new few-shot image classification algorithm. It has two main contributions. The first one is to use a metric-softmax loss used to train on the meta-training dataset without episodic updates. The second is that the features learnt thereby are further modified using a linear transformation to fit the few-shot training data and the metric soft-max loss is again used for classifying the query samples. The authors provide experimental results for 5-way-1-shot and 5-way-5-shot testing on mini-Imagenet and CUB-200-2011 datasets.