Fast Task Adaptation for Few-Shot Learning

Few-shot classification is a challenging task due to the scarcity of training examples for each class . The key lies in generalization of prior knowledge learned from large-scale base classes and fast adaptation of the classifier to novel classes . In this paper , we introduce a two-stage framework . In the first stage , we attempt to learn task-agnostic feature on base data with a novel Metric-Softmax loss . The Metric-Softmax loss is trained against the whole label set and learns more discriminative feature than episodic training . Besides , the Metric-Softmax classifier can be applied to base and novel classes in a consistent manner , which is critical for the generalizability of the learned feature . In the second stage , we design a task-adaptive transformation which adapts the classifier to each few-shot setting very fast within a few tuning epochs . Compared with existing fine-tuning scheme , the scarce examples of novel classes are exploited more effectively . Experiments show that our approach outperforms current state-of-the-arts by a large margin on the commonly used mini-ImageNet and CUB-200-2011 benchmarks . 1 INTRODUCTION . In recent years , deep learning models have achieved great success in many visual tasks such as image classification ( Krizhevsky et al. , 2012 ) , object detection ( Girshick , 2015 ) and semantic segmentation ( Long et al. , 2015 ) . In general , training a deep network needs massively labeled instances which leads to expensive manual annotation cost . In contrast , humans can learn novel concepts from only one or a few examples . Analogously , few-shot learning aims to recognize unseen instances by accessing just a small number of labeled images in each class . Unfortunately , naive methods such as re-training or fine-tuning the model on novel data would severely suffer from overfitting and provide poor result ( Finn et al. , 2017 ) . The major difficulty lies in the exceedingly limited data , which can hardly represent the class distribution . In the task of few-shot classification , we are given three datasets , namely training set , support set and query set . The training set ( also known as base dataset ) normally contains large-scale labeled data for learning of prior knowledge . The support set and query set contain a small number of examples which are drawn from unseen classes disjoint with the training set . We are expected to classify the query images , leveraging the prior knowledge from the base training set and the limited cue from the support images . If the support set contains k annotated images for each of m unique classes , then this task is called m-way k-shot classification . Previous works on few-shot learning can be roughly divided into two categories , namely metalearning based and metric learning based . The former typically learns a meta-learner model on base data and generalizes it to novel unseen data . Usually , a recurrent neural network ( Santoro et al. , 2016 ) or long short-term memory network ( Hochreiter & Schmidhuber , 1997 ) is utilized to learn a memory network to store knowledge . The latter attempts to learn a feature embedding where samples of the same class are closer to each other than samples of different classes . To make training on base data and inference on novel data consistent , a so-called episodic training strategy is widely employed ( Vinyals et al. , 2016 ; Snell et al. , 2017 ; Sung et al. , 2018 ) . There are two limitations in metric learning based approaches . One is that episodic training only considers local sample similarity in current data batch . As shown in Snell et al . ( 2017 ) , the limited number of classes pertraining episode is harmful for discriminative feature learning , while simply increasing it leads to better performance . However , in a sampled episode , a large number of classes is infeasible due to limited GPU memory . The other limitation is that most existing methods only utilize the labeled support images in novel classes as templates for feature similarity comparison . We argue that the support images could be better exploited to improve the performance . In this work , we introduce a two-stage framework named Fast Task Adaption ( FTA ) . As illustrated in Figure 1 , the few-shot classification task is decomposed into two steps , i.e . task-agnostic feature learning on base data and adaptation of the classifier to each few-shot learning task . To learn more general and discriminative feature on base data , we propose a novel Metric-Softmax loss , which is an improvement of the Softmax loss . Specifically , we replace the score computing in softmax with a Gaussian kernel-based radial basis function . The Metric-Softmax loss is trained against the full base classes , overcoming the limitation of small number of classes in episodic training . Moreover , the Metric-Softmax classifier can be applied to base and novel classes in the same way . The consistency between training and inference is well preserved . After task-agnostic feature is learned , we design a novel task-adaptive transformation to adapt the classifier to current few-shot learning task . It is essentially an affine transformation applied to features of novel images . Notably , compared with fine-tuning re-initialized weights , it exhibits the desirable characteristics of fast convergence and superior performance . In this way , the aforementioned weaknesses of existing metric learning based methods are well addressed , leading to significant accuracy improvement . Experimental results on the mini-ImageNet and CUB-200-2011 datasets show that our approach consistently outperforms current state-of-the-art methods with respect to various backbone networks . The main contributions of this paper can be summarized as follows . • We propose the Metric-Softmax loss , which learns highly discriminative feature and ensures the consistency between training and inference as well . • We design a novel task-adaptive transformation to adapt the classifier to novel classes efficiently and effectively . • We improve few-shot classification significantly and achieve state-of-the-art performance on two common few-shot learning benchmarks . 2 RELATED WORK . In this section , we briefly review the previous approaches relevant to ours and current state-of-the-art methods we compare with in experiments . Meta-learning based methods aim to learn task-agnostic knowledge from a sequence of similar tasks which have enough training data and apply it to new tasks for fast adaption . Ravi & Larochelle ( 2017 ) utilize an LSTM-based network to control the parameter optimization process , which results in better generalization than SGD when training on a few labeled samples . Qiao et al . ( 2018 ) propose to adapt a pre-trained network to novel classes by directly predicting the parameters from the activations . MAML ( Finn et al. , 2017 ) learns sensitive and general initial parameters , which can fast adapt to a novel task with only one or a few steps of gradient-descent update . MTL ( Sun et al. , 2019 ) learns to adapt a deep neural network for few-shot learning tasks by learning scaling and shifting functions of weights for each task . Cai et al . ( 2018 ) use a contextual learner to predict the parameters of an embedding network for unlabeled data by using memory slots . Mishra et al . ( 2018 ) propose a meta-learner architecture that combines temporal convolutions and soft attention . Metric learning based methods aim to learn a feature embedding that preserves the class neighborhood structure . Specifically , samples of the same class are closer than samples of different classes in the learned embedding space . Matching networks ( Vinyals et al. , 2016 ) combine attention and memory together by learning a network that maps a small labeled support set and an unlabeled example to its label . They also first introduce the episodic training strategy , where the training process mimics the test scenario based on support-query metric learning . Prototypical network ( Snell et al. , 2017 ) uses the mean feature of each class as its corresponding prototype representation to learn a similarity metric space . Relation network ( Sung et al. , 2018 ) learns a deep non-linear distance metric by considering the relation between query images and support images . Satorras & Estrach ( 2018 ) propose a graph neural network to learn the feature similarity between query and support set . Our approach belongs to this category . Compared with existing methods , our major improvements are the Metric-Softmax loss for generic feature learning and the task-adaptive transformation for fast task adaptation . Other methods . Zhang et al . ( 2018 ) propose a GAN-based approach to help few-shot classifiers to learn sharper decision boundary . Si et al . ( 2019 ) propose a progressive cluster purification method for transductive few-shot learning . Li et al . ( 2019 ) propose a deep nearest neighbor neural network for few-shot learning . Chen et al . ( 2019 ) give a consistent comparative analysis of several representative few-shot classification algorithms and show that the depth of backbone network matters . Besides , they attempt to fine-tune a Softmax classifier on the support set . On the contrary , we first apply an affine transformation to features of the support images , and compute the class centroids of the transformed features as the classfier ’ s weights . Concretely , we learn the affine transforming matrix rather than directly fine-tune the randomly initialized weights . Optimizing the transformation is computationally more stable even when one example per-class is available . 3 METHOD . As shown in Figure 1 , the proposed FTA framework is composed of two stages , namely task-agnostic feature learning stage and fast task adaptation stage . In the first stage , we train the network on the base data with the proposed Metric-Softmax loss . After that , we fix the parameters of the network backbone and use it as a feature extractor . In the second stage , we learn a task-adaptive classifier , which is also a Metric-Softmax classifier with its weights replaced with affine transformed features of the support images . The affine transformation g is named Task-Adaptive Transformation ( TAT ) , which is learned on the scarce support set . With proper initialization , training of g converges very fast within a few epochs . After training finishes , g is applied to both support and query images . In this section , we first elaborately describe the proposed Metric-Softmax loss . Since it is an improvement of the Softmax loss , we first give a brief review of the Softmax loss and reveal the issue of discrepancy between training and inference . Then we explain how to address this issue with the Metric-Softmax loss . Finally , we introduce the task-adaptive transformation for fast task adaptation . 3.1 SOFTMAX CLASSIFIER . Feature learning with the Softmax loss has achieved great success in many feature ranking tasks like person re-identification ( Zhang et al. , 2017 ) and face verification ( Wen et al. , 2016 ) . Given an image X of base classes , a backbone network f is applied to project X into feature space h = f ( X ) , h ∈ Rd . Then , the logits z are computed by feeding h into an affine transformation : z = W Th + b , ( 1 ) where W ∈ Rd×c and b ∈ Rc are the transforming weights . c is the number of classes in base data . The Softmax classifier predicts the probability score that the image belongs to class i by si = softmax ( z ) i = exp ( zi ) ∑c j exp ( zj ) ( 2 ) The Softmax loss is an assembly of the Softmax classifier and cross-entropy loss . Suppose the one-hot category label of the image is y ∈ { 0 , 1 } c , the training cross-entropy loss is defined as L = − c∑ j yj log ( sj ) ( 3 ) During inference , if a test image belongs to one of the training classes , its label can be predicted by ŷ = arg max i si ( 4 ) In this case , the scores in both training and inference are computed by an affine transformation and softmax function applied to the feature vector h. However , in the context of few-shot learning , the query image for testing belongs to a novel class unseen in training . A common way to classify the novel query image is to compare its similarities to the few labeled support images , which share the same label space with the query image . Concretely , for m-way k-shot classification , we can extract features of the support images , and compute the mean feature for each class by h̄i = 1 k k∑ j f ( Xij ) ( 5 ) Here , Xij denotes the j-th image of class i in the support set . The label of a query image X can be predicted by ŷ = arg min i ||f ( X ) − h̄i|| ( 6 ) Rather than an inner-product between h and column vectors of the weight matrix ( zi = W T : ,ih+bi ) followed by softmax function in the previous case , the prediction is based on the euclidean distance between the query and support features . We argue that this discrepancy hurts the transferability of features pre-trained on base data to novel data .

Summary: This paper deals with few-shot learning from a metric-learning perspective. The authors propose replacing the softmax loss, i.e. softmax + cross-entropy loss, with a so-called "metric-softmax" loss which imitates a Guassian kernel RBF over class templates/weights. This loss is used in both stages of training on base and on novel classes and the authors argue that it helps learning more discriminative feature while preserving consistency between train and test time.