Few-Shot Classification with Task-Adaptive Semantic Feature Learning

Few-shot classification aims to learn a classifier that categorizes objects of unseen classes with limited samples . One general approach is to mine as much information as possible from limited samples . This can be achieved by incorporating data aspects from multiple modals . However , existing multi-modality methods only use additional modality in support samples while adhering to a single modal in query samples . Such approach could lead to information imbalance between support and query samples , which confounds model generalization from support to query samples . Towards this problem , we propose a task-adaptive semantic feature learning mechanism to incorporate semantic features for both support and query samples . The semantic feature learner is trained episodic-wisely by regressing from the feature vectors of the support samples . Then the query samples can obtain the semantic features with this module . Such method maintains a consistent training scheme between support and query samples and enables direct model transfer from support to query datasets , which significantly improves model generalization . We develop two modality combination implementations : feature concatenation and feature fusion , based on the semantic feature learner . Extensive experiments conducted on four benchmarks demonstrate that our method outperforms state-of-the-arts , proving the effectiveness of our method . 1 INTRODUCTION . Deep neural network models have gained huge success in various tasks , some of them even achieved the human-level performances ( He et al. , 2016 ; Krizhevsky et al. , 2012 ; Simonyan & Zisserman , 2014 ) . Nevertheless , such success relies on sufficient annotated training data , which limits the applicability of the models to new concepts with very little supervision . Few shot learning has emerged as an appealing paradigm to solve this problem ( Fei-Fei et al. , 2006 ) . It aims to bridge the sample-efficiency gap between machine learning and human learning in various application fields ( Wang et al. , 2020 ) , such as image classification , object detection and domain adaptation ( Dixit et al. , 2017 ; Motiian et al. , 2017 ; Snell et al. , 2017 ) . Few-shot learning ( FSL ) can classify instances from unseen classes with only limited number of labeled samples . It has witnessed significant advances in few-shot classification . In FSL setting , a series of N-way K-shot few-shot tasks ( or episodes ) are created , where each task ( or episode ) contains a support set with K labeled examples per classes as supervision , and a query set with some unlabeled samples to be predicted . Meta-learning is one popular way of developing few-shot learning strategies , which has been proved effective for few-shot classification . It leverages a series of few-shot tasks generated from base dataset to learn a set of parameters of the desired classifier that can generalize well to novel tasks ( Finn et al. , 2017 ; Ren et al. , 2018 ; Rusu et al. , 2018 ) . Another category of approach is Metric-based method . It focuses on learning a generalizable embedding model to transform all samples into a discriminative metric space , in which , samples of the same classes are clustered together , while distances between samples of the different class are widened . And a non-parametric , distance-based metric is adopted to make classification for the unlabeled samples , which simplifies the classification process in the embedding space ( Gidaris & Komodakis , 2018 ; Li et al. , 2019a ; 2017 ; Sung et al. , 2018 ; Vinyals et al. , 2016 ) . Most of few-shot classification approaches are implemented based on the visual information from images . The classification result greatly depends on the visibility and cleanliness of the target in the images . It is easy to misclassify when the target is small , inconspicuous , or the target is in a similar background , or there is a similar Interference around the target , see in Figure 1 . Regarding to this problem , Yue ( Yue et al. , 2020 ) propose a causal intervention mechanism to eliminate the effect of similar background . Hou ( Hou et al. , 2019 ) proposes a cross attention network to locate the most relevant regions in the pair of labeled and unlabeled samples , which can help to highlight the target object , so as to alleviate the influence of similar Interference . Xing ( Xing et al. , 2019 ) incorporates semantic features with the features for support set samples to help distinguishing samples of different targets with similar appearance , which can modify the position of class centers . However , this method is incapable of adjusting the feature embeddings of samples from query set . Previous works have demonstrated that leveraging multiple modalities in few-shot image classification have great potential in performance improvement ( Xing et al. , 2019 ; Schwartz et al. , 2019 ; Li et al. , 2020 ; Chen et al. , 2019 ) . However , what is missing from the current multi-modal approaches is that the semantic features is only available in the support set but not in query set . Consequently , separating visually similar objects of different classes in the query set , which are in close vicinities of each other in the visual feature space , remains a fairly difficult task . To solve this problem , we propose a task-adaptive semantic feature learning mechanism , to predict semantic features for the query samples which is helpful in distinguishing samples with similar background and interference . The semantic features are the word embeddings of the text labels , which are predicted by the Glove word embedding method . The task adaptive semantic feature learner is trained within each task with the support samples as the supervision . Then , the trained semantic feature learner is used to predict semantic features for the query samples , which can guide the model to pay more attention to the object of the corresponding class . The semantic feature learner is customized to respective task such that the semantic features are most discriminative for a given task . The contributions of this work are three folds . ( 1 ) We propose a task-adaptive semantic feature learner to predict semantic features for the query samples , which improves the discrimination of the query features . ( 2 ) The visual and semantic features are learned separately without collapse into a common feature space . which maintains specific information of different modalities . ( 3 ) we construct some hard tasks with samples having similar background and interference , to prove the efficacy of TasNet in generating discriminative features from hard samples . ( 4 ) We develop two modality combination implementations , feature concatenation and feature fusion ( feature weighted sum ) , both results show the effectiveness of the semantic feature leaner . 2 RELATED WORK . 2.1 FEW SHOT LEARNING . Few-shot classification methods exhibit great diversity . These methods can be roughly divided into two main groups : metric-based and gradient-based approaches . Metric-based approaches aim at learning discriminative representations that minimize intra-class distances while maximizing the inter-class distances with a non-parametric metric . Representative works include Siamese Network ( Koch et al. , 2015 ) , Matching Network ( Vinyals et al. , 2016 ) and Prototypical Network ( Snell et al. , 2017 ) . On this basis , Relation network ( Sung et al. , 2018 ) proposed a learnable deep distance metric , to construct the relation of samples within a task . The unlabeled samples are classified according to the metric scores . Gradient based methods focus on training models that can generalize well to new tasks with only a few fine-tuning updates . MTL ( Sun et al. , 2019 ) approach leverages transfer learning and benefits from referencing neuron knowledge in pre-trained deep nets . MAML ( Finn et al. , 2017 ) aims to learn a good parameter initialization that enables the model easy to fine-tune . 2.2 MULTI-MODALITY FEW SHOT LEARNING . Recently , some researches concerning multi-modality have been proposed , such as Adaptive Modality Mixture Mechanism ( AM3 ) ( Xing et al. , 2019 ) , Multiple-Semantics ( Schwartz et al. , 2019 ) , taskrelevant additive margin loss ( TRAML ) ( Li et al. , 2020 ) and Semantic Feature Augmentation ( Chen et al. , 2019 ) , to leverage extra information of the samples to improve the classification performance . In ( Xing et al. , 2019 ) , Xing et al . observed that some visually-similar images of different classes are distant from each other in the semantic feature space . Thus , they proposed an adaptive feature fusion mechanism to combine features of the visual and semantic modalities to refine the category prototypes of the support set . Similarly , E. Schwartz et al . ( Schwartz et al. , 2019 ) proposed a improved version of AM3 by incorporating multiple and richer semantics information ( category labels , attributes , and natural language descriptions ) to the image visual features , which could result in further performance improvements . This idea came from mimicking the learning process of a human baby . TRAML ( Li et al. , 2020 ) proposed an adaptive margin principle to improve the generalization ability of metric-based meta-learning approaches . The adaptive margins are defined as the semantic similarity between different categories . TriNet ( Chen et al. , 2019 ) solves the few shot classification problem via feature augmentation in the semantic feature space . It first embedded images into a latent semantic space . And then , it augmented semantic features with semantic gaussian and semantic neighborhood approaches . After which , the semantic features are transformed back to the image visual features space , which enriches useful information for the limited samples . A common limitation of these methods is that while additional modalities are used in support samples , there is still a single modality used in the query samples . Our method solves this problem by incorporating visual and semantic features in both support and query samples . 3 METHODOLOGY . We incorporate data of multiple modalities as an attempt to extract additional semantic information from samples . The key challenge , however , is that labels of the query set are obstructed from the classifier . Instead of abandoning the additional modality all together in classifying samples in the query set , we choose to re-construct the missing modality from available information . This requires a translator that predicts values of the missing modality . In the realm of image classification , we train a task-specific image-to-semantic translator , termed the semantic feature learner , from the support set , which is then applied to the query set to generate the semantic features for each image . Since both the visual and the semantic modalities are now present in the query set , we combine these two features via concatenation or fusion operation to produce final feature representations for each sample . In our method , the semantic feature learner is trained separately from the visual feature learner . The key idea of isolating the two training processes is that while visual features might be shared across all categories and many tasks , semantic features , as a supplementing feature set , is used to help distinguish instances of visually similar categories within the current task . Hence the visual feature learner is trained globally and the semantics feature learner is trained locally per each task . Depending on the task setup , the semantic feature learner might focus on different features in images . For example , the semantic feature learner might put different emphases in egg images to help separate eggs from pingpong balls ( in one specific task ) , then from separating eggs from carrots ( in another specific task ) . By limiting the training process of the semantic feature learner to each task , our method allows greater flexibilities adapting to each task and increases the efficacy of using the semantic features to further distinguish visually similar objects in a task . Overall , our method trains a global visual feature learner ( fϕ ) across all tasks during the metatraining phase , while trains a task-specific semantic feature learner ( gψ ) within each task separately . In support set , we compute the visual-feature class centers ( visual prototypes ) and combines them with label-derived semantic features into final prototype feature vectors p ( class centers ) . In query set , we generate both the visual features and the semantic features from fϕ and gψ , respectively , which are then combined into a final feature representation q of the query sample . Finally , the classification of query samples is implemented based on the distance to prototypes associated to each class .

Summary: In few-shot learning for image classification, visual features alone may represent multiple objects in an image. The label text provides additional information that could serve as useful inductive bias, indicating which object in an image the model should pay attention to. There are multiple ways to use such text information to help few-shot classification. The authors propose to build a model that predicts both the image classes and the text label (or its Glove embedding). In other words, an additional loss term is introduced to regulate the learning of visual features. The authors show that the additional "semantic features" (because Glove embedding is used) help improve few-shot image classification.