Year: 2021

Abstract:
Few-shot classification aims to learn a classifier that categorizes objects of unseen classes with limited samples. One general approach is to mine as much information as possible from limited samples. This can be achieved by incorporating data aspects from multiple modals. However, existing multi-modality methods only use additional modality in support samples while adhering to a single modal in query samples. Such approach could lead to information imbalance between support and query samples, which confounds model generalization from support to query samples. Towards this problem, we propose a task-adaptive semantic feature learning mechanism to incorporate semantic features for both support and query samples. The semantic feature learner is trained episodic-wisely by regressing from the feature vectors of the support samples. Then the query samples can obtain the semantic features with this module. Such method maintains a consistent training scheme between support and query samples and enables direct model transfer from support to query datasets, which significantly improves model generalization. We develop two modality combination implementations: feature concatenation and feature fusion, based on the semantic feature learner. Extensive experiments conducted on four benchmarks demonstrate that our method outperforms state-of-the-arts, proving the effectiveness of our method.

text: Few-shot classification aims to learn a classifier that categorizes objects of unseen classes with limited samples. One general approach is to mine as much information as possible from limited samples. This can be achieved by incorporating data aspects from multiple modals. However, existing multi-modality methods only use additional modality in support samples while adhering to a single modal in query samples. Such approach could lead to information imbalance between support and query samples, which confounds model generalization from support to query samples. Towards this problem, we propose a task-adaptive semantic feature learning mechanism to incorporate semantic features for both support and query samples. The semantic feature learner is trained episodic-wisely by regressing from the feature vectors of the support samples. Then the query samples can obtain the semantic features with this module. Such method maintains a consistent training scheme between support and query samples and enables direct model transfer from support to query datasets, which significantly improves model generalization. We develop two modality combination implementations: feature concatenation and feature fusion, based on the semantic feature learner. Extensive experiments conducted on four benchmarks demonstrate that our method outperforms state-of-the-arts, proving the effectiveness of our method.

heading: 1 INTRODUCTION
text: Deep neural network models have gained huge success in various tasks, some of them even achieved the human-level performances (He et al., 2016; Krizhevsky et al., 2012; Simonyan & Zisserman, 2014). Nevertheless, such success relies on sufficient annotated training data, which limits the applicability of the models to new concepts with very little supervision. Few shot learning has emerged as an appealing paradigm to solve this problem (Fei-Fei et al., 2006). It aims to bridge the sample-efficiency gap between machine learning and human learning in various application fields (Wang et al., 2020), such as image classification, object detection and domain adaptation (Dixit et al., 2017; Motiian et al., 2017; Snell et al., 2017).
Few-shot learning (FSL) can classify instances from unseen classes with only limited number of labeled samples. It has witnessed significant advances in few-shot classification. In FSL setting, a series of N-way K-shot few-shot tasks (or episodes) are created, where each task (or episode) contains a support set with K labeled examples per classes as supervision, and a query set with some unlabeled samples to be predicted. Meta-learning is one popular way of developing few-shot learning strategies, which has been proved effective for few-shot classification. It leverages a series of few-shot tasks generated from base dataset to learn a set of parameters of the desired classifier that can generalize well to novel tasks (Finn et al., 2017; Ren et al., 2018; Rusu et al., 2018). Another category of approach is Metric-based method. It focuses on learning a generalizable embedding model to transform all samples into a discriminative metric space, in which, samples of the same classes are clustered together, while distances between samples of the different class are widened. And a non-parametric, distance-based metric is adopted to make classification for the unlabeled samples, which simplifies the classification process in the embedding space (Gidaris & Komodakis, 2018; Li et al., 2019a; 2017; Sung et al., 2018; Vinyals et al., 2016).
Most of few-shot classification approaches are implemented based on the visual information from images. The classification result greatly depends on the visibility and cleanliness of the target in the images. It is easy to misclassify when the target is small, inconspicuous, or the target is in a similar background, or there is a similar Interference around the target, see in Figure 1. Regarding to this problem, Yue (Yue et al., 2020) propose a causal intervention mechanism to eliminate the effect of similar background. Hou (Hou et al., 2019) proposes a cross attention network to locate the most relevant regions in the pair of labeled and unlabeled samples, which can help to highlight the target object, so as to alleviate the influence of similar Interference. Xing (Xing et al., 2019) incorporates semantic features with the features for support set samples to help distinguishing samples of different targets with similar appearance, which can modify the position of class centers. However, this method is incapable of adjusting the feature embeddings of samples from query set. Previous works have demonstrated that leveraging multiple modalities in few-shot image classification have great potential in performance improvement (Xing et al., 2019; Schwartz et al., 2019; Li et al., 2020; Chen et al., 2019).
However, what is missing from the current multi-modal approaches is that the semantic features is only available in the support set but not in query set. Consequently, separating visually similar objects of different classes in the query set, which are in close vicinities of each other in the visual feature space, remains a fairly difficult task. To solve this problem, we propose a task-adaptive semantic feature learning mechanism, to predict semantic features for the query samples which is helpful in distinguishing samples with similar background and interference. The semantic features are the word embeddings of the text labels, which are predicted by the Glove word embedding method. The task adaptive semantic feature learner is trained within each task with the support samples as the supervision. Then, the trained semantic feature learner is used to predict semantic features for the query samples, which can guide the model to pay more attention to the object of the corresponding class. The semantic feature learner is customized to respective task such that the semantic features are most discriminative for a given task. The contributions of this work are three folds. (1) We propose a task-adaptive semantic feature learner to predict semantic features for the query samples, which improves the discrimination of the query features. (2) The visual and semantic features are learned separately without collapse into a common feature space. which maintains specific information of different modalities. (3) we construct some hard tasks with samples having similar background and interference, to prove the efficacy of TasNet in generating discriminative features from hard samples. (4) We develop two modality combination implementations, feature concatenation and feature fusion (feature weighted sum), both results show the effectiveness of the semantic feature leaner.

heading: 2 RELATED WORK

heading: 2.1 FEW SHOT LEARNING
text: Few-shot classification methods exhibit great diversity. These methods can be roughly divided into two main groups: metric-based and gradient-based approaches. Metric-based approaches aim at learning discriminative representations that minimize intra-class distances while maximizing the inter-class distances with a non-parametric metric. Representative works include Siamese Network (Koch et al., 2015), Matching Network (Vinyals et al., 2016) and Prototypical Network (Snell et al., 2017). On this basis, Relation network (Sung et al., 2018) proposed a learnable deep distance metric, to construct the relation of samples within a task. The unlabeled samples are classified according to the metric scores. Gradient based methods focus on training models that can generalize well to new tasks with only a few fine-tuning updates. MTL (Sun et al., 2019) approach leverages transfer
learning and benefits from referencing neuron knowledge in pre-trained deep nets. MAML (Finn et al., 2017) aims to learn a good parameter initialization that enables the model easy to fine-tune.

heading: 2.2 MULTI-MODALITY FEW SHOT LEARNING
text: Recently, some researches concerning multi-modality have been proposed, such as Adaptive Modality Mixture Mechanism (AM3) (Xing et al., 2019), Multiple-Semantics (Schwartz et al., 2019), taskrelevant additive margin loss (TRAML) (Li et al., 2020) and Semantic Feature Augmentation (Chen et al., 2019), to leverage extra information of the samples to improve the classification performance. In (Xing et al., 2019), Xing et al. observed that some visually-similar images of different classes are distant from each other in the semantic feature space. Thus, they proposed an adaptive feature fusion mechanism to combine features of the visual and semantic modalities to refine the category prototypes of the support set. Similarly, E. Schwartz et al. (Schwartz et al., 2019) proposed a improved version of AM3 by incorporating multiple and richer semantics information (category labels, attributes, and natural language descriptions) to the image visual features, which could result in further performance improvements. This idea came from mimicking the learning process of a human baby. TRAML (Li et al., 2020) proposed an adaptive margin principle to improve the generalization ability of metric-based meta-learning approaches. The adaptive margins are defined as the semantic similarity between different categories. TriNet (Chen et al., 2019) solves the few shot classification problem via feature augmentation in the semantic feature space. It first embedded images into a latent semantic space. And then, it augmented semantic features with semantic gaussian and semantic neighborhood approaches. After which, the semantic features are transformed back to the image visual features space, which enriches useful information for the limited samples. A common limitation of these methods is that while additional modalities are used in support samples, there is still a single modality used in the query samples. Our method solves this problem by incorporating visual and semantic features in both support and query samples.

heading: 3 METHODOLOGY
text: We incorporate data of multiple modalities as an attempt to extract additional semantic information from samples . The key challenge, however, is that labels of the query set are obstructed from the classifier. Instead of abandoning the additional modality all together in classifying samples in the query set, we choose to re-construct the missing modality from available information. This requires a translator that predicts values of the missing modality. In the realm of image classification, we train a task-specific image-to-semantic translator, termed the semantic feature learner, from the support set, which is then applied to the query set to generate the semantic features for each image. Since both the visual and the semantic modalities are now present in the query set, we combine these two features via concatenation or fusion operation to produce final feature representations for each sample.
In our method, the semantic feature learner is trained separately from the visual feature learner. The key idea of isolating the two training processes is that while visual features might be shared across all categories and many tasks, semantic features, as a supplementing feature set, is used to help distinguish instances of visually similar categories within the current task. Hence the visual feature learner is trained globally and the semantics feature learner is trained locally per each task. Depending on the task setup, the semantic feature learner might focus on different features in images. For example, the semantic feature learner might put different emphases in egg images to help separate eggs from pingpong balls (in one specific task), then from separating eggs from carrots (in another specific task). By limiting the training process of the semantic feature learner to each task, our method allows greater flexibilities adapting to each task and increases the efficacy of using the semantic features to further distinguish visually similar objects in a task.
Overall, our method trains a global visual feature learner (fϕ) across all tasks during the metatraining phase, while trains a task-specific semantic feature learner (gψ) within each task separately. In support set, we compute the visual-feature class centers (visual prototypes) and combines them with label-derived semantic features into final prototype feature vectors p (class centers). In query set, we generate both the visual features and the semantic features from fϕ and gψ , respectively, which are then combined into a final feature representation q of the query sample. Finally, the
classification of query samples is implemented based on the distance to prototypes associated to each class.

heading: 3.1 PROBLEM DEFINITION
text: In Few Shot Learning setting, There are three datasets: Db, Dv and Dn denoting train (base), validation and test (novel) set respectively, with categories disjoint from each other, i.e. Db ∩ Dv = ∅, which is different from traditional supervised learning. The goal of FSL is to classify samples from unseen classes with few supervised samples. In general, the few shot learning approaches are based on episodic training paradigm. The model is trained across a number of episodes T sampled from Db. A N-way KS-shot task (or episode) is denoted as Ti={S;Q}, containing N categories. Where S denotes the support set consisting of KS samples per category, i.e. S = (xi, yi)N×KSi=1 ; and Q denotes the query set consisting of KQ samples per category, i.e. Q = (xi, yi) N×KQ i=1 . Finally, the performance of the model is evaluated on multiple tasks sampled from the novel dataset Dn.

heading: 3.2 ARCHITECTURE OF THE PROPOSED MODEL
text: Figure 2 illustrates the overall framework of the proposed method. It consists of a pre-training stage, a meta-training stage and a meta-testing stage. Firstly, we pre-train a feature extractor Eθ with C(·|Db) as the classifier from scratch by minimizing a standard cross entropy loss with the base dataset Db as the training data. After pre-training stage, the parameters of the feature extractor are frozen, which is indicated with light blue color. In the meta-training stage, images are first fed into the feature extractor Eθ to produce feature vectors. Then, the feature vectors are used as inputs for the visual feature learner fϕ and the semantic feature learner gψ . Meanwhile, the support images and the corresponding class text labels (IS ,WS) are feed into F (·|S) module to train the task-specific semantic feature leaner. The trained semantic feature learner gψ is represented in light yellow color, denoting the parameters are frozen. Both the final prototypes and the final feature representations of the query samples are generated by F C/F module. The key difference between the support feature representations and the query feature representations is that: in the support set, semantic features are generated from the class text labels, while in the query set, semantic features are predicted with gψ from image feature vectors. Finally, the visual feature learner fϕ is optimized via minimizing the cross-entropy loss between the ground truth labels and the predicted probability vectors. The meta-testing stage is similar with the meta-training stage, except that the parameters of the visual feature learner fϕ can be no longer updated, which is represented in light green color.

heading: 3.3 TASK-ADAPTIVE SEMANTIC FEATURE LEARNING
text: The feature vectors of the support images and the ground truth semantic feature embeddings (it is predicted by Glove word embedding of the class text labels) are paired up as supervision to train the semantic feature learner gψ . The loss function is formulated as the dissimilarity between the
predicted semantic feature embeddings and the ground-truth semantic feature embeddings. See in Eq. 1.
Ldissim (ψ) = N∑ c=1 KS∑ i=1 (1− w̃ci · w T c ||w̃ci || · ||wc|| ) (1)
where w̃ci denotes the predicted semantic feature embedding of the i-th sample belonging to class c, and wc denotes the ground truth semantic feature embedding of class c. We train the semantic feature learner within each task by minimizing Ldissim to produce task-specific parameters.

heading: 3.4 FEATURE GENERATION

heading: 3.4.1 PROTOTYPES GENERATION OF THE SUPPORT SAMPLES
text: Firstly, we extract feature vectors for both support and query samples, as formulated in Eq. 2.
ai = Eθ (xi) , i = 1, . . . , N(KS +KQ) (2)
where xi denotes an image sample, and ai is the feature vector extracted from this image. Then we transform the feature vectors to the visual feature space via the visual feature learner fϕ: v = fϕ (a). And we compute the visual prototypes with Eq. 3
vc = 1
KS ∑ (xj , yj)∈Sc fϕ (aj) c = 1, ..., N (3)
where Sc denotes a subset of S, which contains samples of class c. Finally, we combine the visual prototypes with the ground truth semantic feature embeddings via concatenation or fusion (weighted sum) operation to obtain the final prototypes of the support set. It is formulated as Eq. 4 and Eq. 5, respectively.
pc = C (vc, wc) , c = 1, . . . , N (4)
pc = λvc + (1− λ)wc, c = 1, . . . , N (5)
where C(·, ·) denotes the concatenation operation of the two vectors, wc is the ground truth semantic feature embedding of class c.

heading: 3.4.2 FEATURE GENERATION OF QUERY SAMPLES
text: The feature representations of the query samples are produced in a similar way as support samples, except that the semantic features of the query samples are learned from images. The final query feature representations are generated via fusion approach formulated as Eq. 6 or concatenation approach formulated as Eq. 7
qi = C (fϕ(ai), gψ (ai)) , i = 1, . . . , NKQ (6)
qi = λfϕ(ai) + (1− λ) gψ (ai) , i = 1, . . . , NKQ (7)
where ai is the feature vector of i-th query image sample.
Algorithm 1: Task adaptive semantic feature learning for few shot classification Input: Task set T ={ Ti = {S;Q} |i = 1, ..., NT } Output: visual feature learner fϕ and semantic feature learner gψ 1 Randomly initialize ϕ and ψ; 2 for t in {T1, . . . , TNT } do 3 Compute prototypes p for the support samples: 4 for c in 1, . . . , N do 5 Extract feature vectors of images from S by Eq. 2; 6 Compute visual prototypes vsc by Eq. 3; 7 Combine the semantic and visual features by Eq. 4 or Eq. 5; 8 Compute semantic features of S with gψ; 9 end 10 Optimize parameters of gψ with samples from S by Algorithm 2; 11 Compute features q of query samples: 12 for j in {1, . . . , NKQ} do 13 Extract feature vectors aqj = Eθ ( xqj ) of samples from Q; 14 Compute visual features vqj = fϕ(a q j); 15 Predict semantic features with gψ(a q j); 16 Compute final feature representations of query samples by Eq .6 or Eq .7;

heading: 17 end
text: 18 Compute the cross entropy loss LCE using Eq. 9; 19 Update ϕ by gradient descent∇LCE . 20 end
Algorithm 2: Task-specific semantic feature learner optimization Input: W = { (wc;w ′ ci ) |c = 1, . . . , N, i = 1, . . .KS } . (wc is the ground
truth semantic features and w′ci = gψ (aci) is the predicted semantic features.) Output: semantic feature learner gψ
1 while not convergence do 2 Evaluate Ldissim (ψ) by Eq. 1; 3 Update ψ by gradient descent ∇Ldissim; 4 end

heading: 3.4.3 CLASSIFICATION LOSS OF THE VISUAL FEATURE LEARNER
text: The negative Euclidean distance between the feature representation qi of i-th query and the prototype pc of the c-th support sample is employed as the predicted scores of the query samples, and the distance is transformed into a probability via a Softmax operation, see Eq. 8.
d (qi, pc) = || qi − pc||2 , i = 1, . . . , NKQ p (ŷi = c|qi, ϕ) = e
−d(qi, pc)∑N c=1 e −d(qi, pc) , c = 1, . . . , N
(8)
where ŷi denotes the predicted label for i-th sample in the query set. Then, the loss function is formulated as the negative loglikelihood of the predicted probability scored of each query sample.
LCE (ϕ) = − KQ∑
i=1,xi∈Q yi log (p (ŷi = yi|xi, ϕ)) (9)
We compute the cross-entropy loss between the predicted scores of the query samples and the ground truth labels for all training episodes constructed from the base dataset, see Eq. 9. Parameters of the visual feature learner is optimized by minimizing LCE . We summarize the whole process in Algorithm 1 and Algorithm 2.

heading: 4 EXPERIMENTS
text: We conduct a comparison of our method to state-of-the-art methods in terms of few-shot classification accuracy on four benchmarks, including MiniImageNet (Vinyals et al., 2016), CUB-200-2011 (Wah et al., 2011), CIFAR-FS (Bertinetto et al., 2018) and FC100 (Oreshkin et al., 2018).

heading: 4.1 ARCHITECTIRES
text: For the feature extractor network, we use a ResNet-12 consisting of 4 residual blocks with Dropblock as a regularizer and 640-dimensional output features, the details follow the one proposed in TADAM (Oreshkin et al., 2018). For the visual feature learner and the semantic feature learner, we use a MLP with one layer, outputting 300-dimentional features.

heading: 4.2 RESULTS AND ANALYSIS
text: Table 1 and Table 2 summarize the results of the few-shot classification tasks on MiniImageNet, CIFAR-FS, FC100, and CUB-200-2011, respectively. It is worth noting that our TasNet achieves the state-of-the-art performance for both 1-shot and 5-shot settings on almost all four benchmarks.

heading: 4.3 ABLATION STUDY
text: To further validate the effectiveness of our method, we conduct a series of ablation studies on miniImageNet and CIFAR-FS datasets. We first show some experimental proofs of the task-adaptive semantic feature learner. Then we show some other results on the hyperparameters.

heading: 4.3.1 EFFECTIVENESS OF THE SEMANTIC FEATURE LEARNER
text: The task-adaptive semantic feature learner is designed to alleviate the negative effects of similar background or similar interference, and provide another kind of modality to enrich the information of query samples, which is helpful in producing discriminative feature embeddings. It can also alleviate the few supervision problem to a certain extent. We test the model with and without semantic feature learner. The quantitative and qualitative results are shown in Figure 3(a) and Figure 4.
As shown in Figure 3 (a), the classification accuracy has achieved a reasonable value in the case of only semantic features. In the 1-shot case, the contribution of the semantic feature is especially significant, which proves the effectiveness of the task-adaptive semantic feature learner. We can also observe that, the combination of the semantic and visual features resulting in further performance improvement, which demonstrates that the semantic features and the visual features can complement each other to improve the classification problem together. Moreover, performance of the concatenation approach of features is ahead of the fusion approach.

heading: 4.3.2 T-SNE VISUALIZATION
text: To qualitatively validate the effectiveness of our method, we further provide some t-SNE visualization results in Figure 4. We sample one episode in the test split of miniImageNet and CIFAR-FS datasets under the 5-way 1-shot and 5-way 5-shot settings. And then, we obtain the embeddings of all images using the visual feature learner, the semantic feature learner, and our TasNet, respectively. Across four subfigures, samples with the same color belong to the same class. Evident in these figures, the overlap between different classes of embeddings generated by TasNet is much less than
that with only visual features, which demonstrates the effectiveness of the task-adaptive semantic feature learner in helping learning feature representations with well separated margin between novel classes. From the last two columns of Figure 4 (a)(b), we can further observe that compared with the visualization of the fused feature embeddings, the discriminative performance of the concatenated feature embeddings is a little superior than the former, which is in consistent with the quantitative result in Figure 3(a) .
In Figure 3(b), we vary the value of fusion weight λ from 0 to 1 to observe the performance change. This hyperparameter controls the contribution of the semantic and visual features to the final fused features. As shown in Figure 3(b), we can observe that, the optimal λ for MiniImageNet in 1- shot and 5-shot cases are 0.2 and 0.3, respectively, while for the CIFAR-FS dataset, the optimal λ reaches at 0.1 for both 1-shot and 5-shot cases. It demonstrates that, in order to improve the final classification performance, we should weigh more on the semantic features, while information from the visual side should be relatively low. Since too much attention paid to the visual feature will weak the advantage brought by the semantic features. It is in consistent with the results of the ablation studies in the former sections. This indicates that the selection of λ plays a key role for best complementing the features information of different modalities.
From the quantitative and qualitative results, we can conclude that the task-adaptive semantic feature learning mechanism is the main reason for the performance boosting.

heading: 5 CONCLUSION
text: In this paper, we propose TasNet, a task adaptive semantic feature learning mechanism to incorporate semantic features for the query samples. Since the feature extractor has been initialized after the pre-training stage, we freeze the parameters of the feature extractor in the meta-training and metatesting stages. Therefore, only the semantic feature learner and the visual feature learner should be optimized, which is less time consuming. Our model trains the global classification loss and the task-adaptive semantic feature dissimilarity loss separately, which can avoid collapsing different modalities into a common feature space, so as to preserve the structural heterogeneity of different modalities. Then, we propose two features combination approaches: feature concatenation and feature fusion, to further improve the performance. Experiments show that our method outperforms the state-of-the-arts on four benchmarks. In the future work, we expect to explore semantic-guided attention mechanism to pay more attention to the targets in the images, so as to better capture classspecific information from the images, which can further alleviate the influence of noise and irrelevant information.

heading: A APPENDIX
text: A.1 SETTINGS
We train the feature extractor Eθ with the SGD optimizer using an initial learning rate of 10−1. Adam (Kingma & Ba, 2014) is used to optimize the parameters of the visual and semantic feature learners with an initial learning rate of 10−3. During few-shot learning stage, we train fϕ and gψ by adopting the episodic training procedure with N-way K-shot training tasks as described in (Snell et al., 2017). The model is trained for 100 epochs, with each epoch consisting of 600 randomly sampled episodes for both 1-shot and 5-shot cases on MiniImageNet, CUB-200-2011, CIFAR-FS and FC100 datasets, and is evaluated by averaging metrics over 600 randomly generated episodes from Dn, with each episode having 15 randomly sampled query samples. Parameters of the model except the backbone layers are trained on 5-way, 15 query samples for both 1-shot and 5-shot cases per episode.
A.2 EXPERIMENTS ON HARD TASKS
To further validate the effectiveness of TasNet in discriminating different targets from similar background and interference, we conduct an additional experiment on hard tasks, which are constructed of samples with similar background and interference.
Firstly, we have constructed a sub dataset (we name it hard dataset) from the test split of miniImagenet dataset manually. It contains 8 classes, including lion, golden retriever, goose, coral reef, vase, spider web, ant, house finch. Among samples from all these categories, some of samples from two or more categories have similar background or similar interference, for example, vase and ant categories, flowers are the common interference in the images from these two classes, and for the bird and spider web categories, trees and grassland are common similar backgrounds in the images from these two classes. Then we construct 20 tasks from this hard dataset, and compare our TasNet with ProtoNet and AM3 methods on this dataset. AM3 is a method that combine semantic features for the support samples, while only visual features are extracted from the query samples. The mean accuracy is show in table 3.
In addition to the statistical results, we show the classification results of one hard task in Figure 5, which is carried on 5-way 1-shot case. Five categories of ant, house finch, spider web, vase, coral reef, are included in this task. Each class has 1 support samples and 15 query samples, that is the number of test samples is 5×15=75.
From Figure 5, we can see that there are 33, 36 and 39 samples correctly classified with the ProtoNet, AM3 and our TasNet methods.
A.3 ANALYSIS OF THE ABLATION STUDY RESULTS
From the quantitative and qualitative results in section 4.3.1 and section 4.3.2, we can conclude that the task-adaptive semantic feature learning mechanism is the main reason for the performance
boosting. To make the ablation study results more obvious, we put the accuracy in terms of four kinds of different features in table 4.
The accuracy computed with the semantic features is shown in the second row of table 4 (in red color). When only the semantic features are used in the model, it has achieved a significant accuracy, which demonstrates that the semantic features for the query samples can certainly work.
A.4 TRAINING THE SEMANTIC AND VISUAL FEATURE LEARNER JOINTLY
In order to prove that training the visual feature learner across all tasks and training the semantic feature learner within each task is an effective approach, we have made some experiments to jointly train the visual and semantic features learners. The detailed implementation of the jointly training method is described as follow.
We constructed a cross entropy loss to optimize the visual feature learner, and a word embedding fitting loss, i.e. the cosine dissimilarity loss to optimize the semantic feature learner.
The loss LCE for training visual feature learner is defined as a cross-entropy classification loss averaged across all query-support pairs. It compares the Euclidean distance between the query features and the class mean of the support features to get the predicted score.
d (qi, pc) = || qi − pc||2 , i = 1, . . . , NKQ (10)
p (ŷi = c|qi, ϕ) = e−d(qi, pc)∑N k=1 e −d(qi, pk) , c = 1, . . . , N (11)
where ŷi denotes the predicted label for i-th sample in the query set, and yi is the ground truth label of sample qi.Then, the loss function is formulated as the negative loglikelihood of the predicted probability scored of each query sample.
LCE (ϕ) = − KQ∑
i=1,xi∈Q yi log (p (ŷi = yi|xi, ϕ)) (12)
The word embedding fitting loss Ldissim (ψ) is defined as dissimilarity between the predicted semantic features and the ground-truth semantic features.
Ldissim (ψ) = K∑ c=1 NS∑ i=1 (1− w̃ci · w T c ||w̃ci || · ||wc|| ) (13)
where w̃ci denotes the predicted semantic feature of the i-th sample belonging to class c, and wc denotes the ground truth semantic feature of class c. We train the semantic feature learner within each task to produce task-specific parameters. Parameters of semantic feature learner are optimized by minimizing Ldissim (ψ). Finally, we combine the cross-entropy classification loss and the word embedding fitting loss via weighted sum, λ=0.5, and train across all tasks to minimize the total loss Ltol.
Ltol = λLCE + (1− λ)Ldissim (14)
The experiment results show that the jointly optimization method of the visual and semantic feature learners can not achieve good performance.

Review summaries:
This paper proposes a task-adaptive semantic feature learning mechanism to incorporate semantic features for both support and query samples. Two modality combination implementations, feature concatenation and feature fusion, are devised based on the semantic feature learner. Experimental results are provided on four benchmark datasets that demonstrate that the proposed mechanism outperforms existing methods.

This paper proposes a few-shot learning method that incorporates semantic features (i.e., class label embeddings) for both support and query samples. Although some existing methods use the semantic features for support samples to improve the performance, no methods use them for query samples.  To this end, the proposed method regresses the semantic features for the query samples from the support samples. Experiments show the effectiveness of the proposed method.

This paper describes a few-shot learning approach that takes into account the semantic relatedness of the ground truth labels rather than just using them as binary labels. In the few-shot learning setting, the paper describes a method to train the semantic feature learner (on labels) during the meta-training phase and then at meta-test time use the semantic feature learner to get the said features for the query images. This way both the support and query examples would have semantic features and empirically it has shown to work on many few-shot benchmarks on computer vision datasets.  

In few-shot learning for image classification, visual features alone may represent multiple objects in an image. The label text provides additional information that could serve as useful inductive bias, indicating which object in an image the model should pay attention to. There are multiple ways to use such text information to help few-shot classification. The authors propose to build a model that predicts both the image classes and the text label (or its Glove embedding). In other words, an additional loss term is introduced to regulate the learning of visual features. The authors show that the additional "semantic features" (because Glove embedding is used) help improve few-shot image classification.