Few-Shot Classification with Task-Adaptive Semantic Feature Learning
Authors: Meihong Pan
Chunqiu Xia
Hongyi Xin
Yang Yang
Xiaoyong Pan
Hong-Bin Shen
Venue: ICLR 2022 Submitted (2022)
Keywords: few-shot learning
task-adaptive semantic feature learning
feature concatenation
feature fusion.

Abstract: Few-shot classification aims to learn a classifier that categorizes objects of unseen classes with limited samples. One general approach is to mine as much information as possible from limited samples. This can be achieved by incorporating data aspects from multiple modals. However, existing multi-modality methods only use additional modality in support samples while adhering to a single modal in query samples. Such approach could lead to information imbalance between support and query samples, which confounds model generalization from support to query samples. Towards this problem, we propose a task-adaptive semantic feature learning mechanism to incorporates semantic features for both support and query samples. The semantic feature learner is trained episodic-wisely by regressing from the feature vectors of support samples. Then the query samples can obtain the semantic features with this module. Such method maintains a consistent training scheme between support and query samples and enables direct model transfer from support to query datasets, which significantly improves model generalization. We develop two modality combination implementations: feature concatenation and feature fusion, based on the semantic feature learner. Extensive experiments conducted on four benchmarks demonstrate that our method outperforms state-of-the-arts, proving the effectiveness of our method.

Under review as a conference paper at ICLR 2022
FEW-SHOT CLASSIFICATION WITH TASK-ADAPTIVE SEMANTIC FEATURE LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Few-shot classification aims to learn classifier that categorizes objects of unseen classes with limited samples. One general approach is to mine as much infor- mation as possible from limited samples. This can be achieved by incorporating data aspects from multiple modals. However, existing multi-modality methods only use additional modality in support samples while adhering to single modal in query samples. Such approach could lead to information imbalance between support and query samples, which confounds model generalization from support to query samples. Towards this problem, we propose a task-adaptive semantic feature learning mechanism to incorporate semantic features for both support and query samples. The semantic feature learner is trained episodic-wisely by regress- ing from the feature vectors of the support samples. Then the query samples can obtain the semantic features with this module. Such method maintains a consis- tent training scheme between support and query samples and enables direct model transfer from support to query datasets, which significantly improves model gen- eralization. We develop two modality combination implementations feature con- catenation and feature fusion, based on the semantic feature learner. Extensive experiments conducted on four benchmarks demonstrate that our method outper- forms state-of-ther the--sts-s, proving the effectiveness of our method.
INTRODUCTION
Deep neural network models have gained huge success in various tasks, some of them even achieved the human-level performances (He et al,, 2016) Krizhevsky et al. [012, Simonyan & Zisserman
2014). Nevertheless, such success relies on sufficient annotated training data, which limits the applicability of the models to new concepts with very little supervision. Few shot learning has emerged appealing paradigm to solve this problem (Fei-Fei et al. 2006). It aims to bridge the sample-efficienee gap between machine learning and human learning in various application fields
[Wang al. 2020), such as image classification, object detection and domain adaptation (Diii ct al. 2017 7 Motiian et al. 2017, Snell et al. 2017).
Few-shot learning (FSL) can classify instances from unseen classes with only limited number of labeled samples. It has witnessed significant advances in few-shot classification. In FSL setting, series of N-way K-shot few-shot tasks (or episodes) are created, where each task (or episode) contains a support set with K labeled examples per classes as supervision, and query set with some unlabeled samples to be predicted. Meta-learning is one popular way of developing few-shot learning strategies, which has been proved effective for few-shot classification. It leverages series of few-shot tasks generated from base dataset to learn a set of parameters of the desired classifier that can generalize well to novel tasks (Finn et al. 2017 Ren et al., 2018 Rusu et al.) 2018). Another category of approach Metric-based method. focuses on learning a generalizable embedding model to transform all samples into a discriminative metric space, in which, samples of the same classes are clustered together, while distances between samples of the different class are widened. And a non- non-parametric, distance- based metric is adopted to make classification for the unlabeled samples, which simplifies the classification process in the embedding space (Gidaris & Komodakis, 2018 Li et al. 2019a 2017 Sung al. 2018 Vinyals et al. 2016).

---

Under review as a conference paper at ICLR 2022
chans
robin
Walker hound
lion
worm fince
Ladybird
uxit
cobweb
Vase
harvestiaan
Figure 1: An example of different categories under similar background(left) and similar Interfer- ence(right). When task contains some samples from these categories, is easy to misclassify.
Most of few-shot classification approaches are implemented based on the visual information from images. The classification result greatly depends on the visibility and cleanliness of the target in the images. It is easy to misclassify when the target is small, inconspicuous, or the target is in a similar background, or there a similar Interference around the target, see in Figure [] Regarding to this problem, Yue (Yue et al., (020) propose a causal intervention mechanism to eliminate the effect of similar background. Hou (Hou et al., 2019) proposes a cross attention network to locate the most relevant regions in the pair of labeled and unlabeled samples, which can help to highlight the target object, so as to alleviate the influence of similar Interference. Xing (Xing et al. (019) incorporates semantic features with the features for support set samples to help distinguishing samples of different targets with similar appearance, which can modify the position of class centers. However, this method is incapable of adjusting the feature embeddings of samples from query set. Previous works have demonstrated that leveraging multiple modalities in few-shot image classification have great potential in performance improvement (Xing et al. 2019 Schwartz et al. 2019: Li et al. 2020 Chen et al. 2019).
However, what missing from the current mudti-modal approaches that the semantic features is only available in the support set but not in query set. Consequently, separating visually similar objects of different classes in the query set, which are in close vicinities of each other in the visual feature space, remains a fairly difficult task. To solve this problem, we propose task-adaptive semantic feature learning mechanism, to predict semantic features for the query samples which is helpful in distinguishing samples with similar background and interference. The semantic features are the word embeddings of the text labels, which are predicted by the Glove word embedding method. The task adaptive semantic feature learner is trained within each task with the support samples as the supervision. Then, the trained semantic feature learner is used to predict semantic features for the query samples, which can guide the model to pay more attention the corresponding class. The semantic feature learner is customized to respective task such that the semantic features are most discriminative for given task. The contributions of this work are three folds. (1) We propose a task- adsk-adeetive semantic feature learner to predict semantic features for the query samples, which improves the discrimination of the query features. (2) The visual and semantic features are learned separately without collapse into a common feature space. which maintains specific information of different modalitics. (3) we construct some hard tasks with samples having
the object of
similar background and interference, to prove the efficacy of TasNet in generating discriminative features from hard samples. (4) We develop two modality combination implementations, feature concatenation and feature fusion (feature weighted sum), both results show the effectiveness of the semantic feature leaner.
2
RELATED WORK
2. FEW SHOT LEARNING
Few-shot classification methods exhibit great diversity. These methods can be roughly divided into two main groups: metric-based and gradient-based approaches. Metric-based approaches aim at learning discriminative representations that minimize intra-class distances while maximizing the inter-class distances with a narmmerae metric. Representative works include Siamese Network (Koch et al. 2015), Matching Network (Vinyals et al. 2016) and Prototypical Network (Snell et al.) 2017). On this basis, Relation network (Sung et al. (2018) proposed learnable deep distance metric, to construct the relation of samples within task. The unlabeled samples are classified according to the metric scores. Gradient based methods focus on training models that can generalize well to new tasks with only few fine-tunin finettuning updates. MTL (Sun et al. 2019) approach leverages transfer
2

---

Under review as a conference paper at ICLR 2022
learning and benefits from referencing neuron knowledge in pre-trained deep nets. MAML (Fin ct al. 2017) aims to learn a good parameter initialization that enables the model easy to fine-tune.
2.2 MULTI-MODALITY FEW SHOT LEARNING
Recently, some researches concerning multi-modalin have been proposed, such Adaptive Modal- ity Mixture Mechanism (AM3) (Xing et al. (2019). Multiple-Semantie (Schwartz et al. 2019), task- relevant additive margin loss (TRAML) (Li et al. 2020) and Semantic Feature Augmentation (Chen et al.) 2019), to leverage extra information of the samples to improve the classification performance. In (Xing et al. 2019), Xing et al. observed that some visually-similar images of different classes are distant from each other in the semantic feature space. Thus, they proposed an adaptive feature fusion mechanism to combine features of the visual and semantic modalities to refine the category prototypes of the support set. Similarly, E. Schwartz et al. (Schwartz et al., 2019) proposed im- proved version of AM3 by incorporating multiple and richer semantics information (category labels, attributes, and natural language descriptions) to the image visual features, which could result in fur- ther performance improvements. This idea came from mimicking the learning process of a human baby. TRAMI (Li et al.) 2020) proposed an adaptive margin principle to improve the generalization ability of metric-based meta-learning approaches. The adaptive margins are defined as the semantic similarity between different categories. TriNet (Chen et al.) 2019) solves the few shot classification problem via feature augmentation in the semantic feature space. It first embedded images into a latent semantic space. And then, it augmented semantic features with semantic gaussian and se- mantic neighborhood approaches. After which, the semantic features are transformed back to the image visual features space, which enriches useful information for the limited samples. A common limitation of these methods is that while additional modalities are used in support samples, there is still a single modality used in the query samples. Our method solves this problem by incorporating visual and semantic features in both support and query samples.
METHODOLOGY
We incorporate data of multiple modalities as an attempt extract additional semantic information from samples. The key challenge. however, is that labels of the query set are obstructed from the classifier, Instead of abandoning the additional modality all together in classifying samples in the query set, we choose to re-construct the missing modality from available information. This requires a translator that predicts values of the missing modality. In the realm of image classification, we train a task-specific image-to-an -ogeaati-e translator, termed the semantic feature learner, from the support set, which is then applied to the query set to generate the semantic features for each image, Since both the visual and the semantic modalities are now present in the query set, we combine these two features via concatenation or fusion operation to produce final feature representations for each sample.
In our method, the semantic feature learner is trained separately from the visual feature learner. The key idea of isolating the two training processes is that while visual features might be shared across all categories and many tasks, semantic features, as a supplementing feature set, is used to help distinguish instances of visually similar categories within the current task. Hence the visual feature learner is trained globally and the semantics feature learner is trained locally per each task, Depending on the task setup, the semantic feature learner might focus on different features in im- ages. For example, the semantic feature learner might put different emphases in egg images to help separate eggs from pingpong balls (in one specific task), then from separating eggs from carrots (in another specific task). By limiting the training process of the semantic feature learner to each task, our method allows greater flexibilities adapting to each task and increases the efficacy of using the semantic features to further distinguish visually similar objects in task.
Overall, our method trains a global visual feature learner (fe) across all tasks during the meta- training phase, while trains task-specific semantic feature learner (gv) within each task separately. In support set, we compute the visual-feature class centers (visual prototypes) and combines them with label-derived semantic features into final prototype feature vectors p (class centers). In query set, we generate both the visual features and the semantic features from f and ga. respectively, which then combined into final feature representation q of the query sample. Finally, the
3

---

Under review as a conference paper at ICLR 2022
classification of query samples is implemented based the distance prototypes associated to each class.
3.1 PROBLEM DEFINITION
In Few Shot Learning setting, There three datasets: Di, D, and D, denoting train (base), vali- dation and test (novel) set respectively, with categories disjoint from each other, i.e. D, n2, = 0. which is different from traditional supervised learning. The goal of FSL is classify samples from unseen classes with few supervised samples. In general, the few shot learning approaches are based on episodic training paradigm. The model trained across a number of episodes T sampled from DD. A N-way K g-sshot task (or episode) is denoted as (S= Q). containing N categories. Where s denotes the support set consisting of Ks samples per category, i.e. S (I,, U:); 1:) NxKs, and Q
denotes the query set consisting of Ko samples per category, i.c. Q = (xi, y/ 3=);=1 Nxko Finally, the performance of the model evaluated on multiple tasks sampled from the novel dataset Dn..
3.2 ARCHITECTURE OF THE PROPOSED MODEL
Figure 2 illustrates the overall framework of the proposed method. It consists of pre-training stage, a meta-training stage and a meta-testing stage. Firstly, we pre-train feature extractor Eg with C(+ID,) as the classifier from scratch by minimizing a standard cross entropy loss with the base dataset D, as the training data, After pre-training stage, the parameters the feature extractor are frozen, which is indicated with light blue color. In the meta-training stage, images are first fed into the feature extractor Eg to produce feature vectors. Then, the feature vectors are used as inputs for the visual feature learner fo and the semantic feature learner Iv. Mcanwhile, the support images and the corresponding class text labels (7 Ws) are feed into F(-[S) module to train the task-specific semantic feature leaner. The trained semantic feature learner Pe represented in light yellow color, denoting the parameters are frozen. Both the final prototypes and the final feature representations of the query samples are generated by F C/F module. The key difference between the support feature representations and the query feature representations is that: in the support set, semantic features are generated from the class text labels, while the query set, semantic features are predicted with IΦ from image feature vectors. Finally, the visual feature learner f. is optimized via minimizing the cross-entropy loss between the ground truth labels and the predicted probability vectors. The meta-testing stage is similar with the meta-training stage, except that the parameters of the visual feature learner f can be no longer updated, which is represented in light green color.
Pre training stage
Concatenation Fusion (F CiF)
Semantic feature medule Ft:
Base baches
Feature Concasonation
apps W
[w]lls]]
q [9g]|ell
Lete
Feature Fasion
Bo
Coaite
A)w; AAs
Ciaillent
Gradient
Meta-training stage
Meta-testing stage
W
erange
Hone support net Sy deg
corn
Wy
C2P
Novel suppertset
gultar
Daciddea
Kacildaas diszanee
laker
Fake
Bose quary
Novel
F(115)
8+
PG-15)
Figure 2: The overall architecture of the few shot classification method with task-adaptive semantic feature learning mechanism.
3.3
SEMANTIC FEATURE LEARNING
The feature vectors of the support images and the ground truth semantic feature embeddings (it is predicted by Glove word embedding of the class text labels) are paired up as supervision to train the semantic feature learner 9v- The loss function is formulated as the dissimilarity between the
4

---

Under review as a conference paper at ICLR 2022
predicted semantic feature embeddings and the ground-trun -ruth semantic feature embeddings. See in Eq.
We w
Edissim (1))
W0: 1/
where We, denotes the predicted semantic feature embedding of the i-th sample belonging to class and we denotes the ground truth semantic feature embedding of class c. We train the semantic feature learner within each task by minimizing Ldissim to produce task-specific parameters.
3.4 FEATURE GENERATION
3.4.1
PROTOTYPES GENERATION OF THE SUPPORT SAMPLES
Firstly, we extract feature vectors for both support and query samples, as formulated in Eq.
a;
Eg (x:), i= 1, N(K s + k
(2)
where denotes an image sample, and is the feature vector extracted from this image. Then we transform the feature vectors to the visual feature space via the visual feature learner fę: v = f&p (a). And we compute the visual prototypes with Eq. 3
V.
Ks (x51 95).s.
Σ fo (az)
c= L.
where S. denotes subset of S, which contains samples of class c.
Finally, combine the visual prototypes with the ground truth semantic feature embeddings via concatenation or fusion (weighted sum) operation to obtain the final prototypes of the support set. formulated Eq. and Eq. 5 respectively.
Pe C (Vey We), I,...,
(4)
Pc Xv + (1 -))We; == I,...,N
(5)
where C(.-)) denotes the concatenation operation of the vectors, w. is the ground truth semantic feature embedding of class c.
3.4.2 FEATURE GENERATION OF QUERY SAMPLES
The feature representations of the query samples are produced in a similar way as support samples, except that the semantic features of the query samples are learned from images. The final query feature representations are generated via fusion approach formulated as Eq. 6 or concatenation approach formulated Eq.
4: C Iy (a:)), i i= =1,... ......KKo
a. (a) (1- ON (a:), i 1 NKa
where 0; is the feature vector of i-th query image sample.
5

---

Under review as a conference paper at ICLR 2022
Algorithm 1: Task adaptive semantic feature learning for few shot classification = T Output: visual feature learner fúp and semantic feature learner g Randomly initialize yp and t/;; for r in . -Nrr do Compute prototypes for the support samples: for in I..... N do
Input: Task set T
I..,,..,, Nr}
Extract feature vectors of images from S by Eq.
Compute visual prototypes v. by Eq.
Combine the semantic and visual features by Eq. or Eq.
Compute semantic features of S with ge:
end
Optimize parameters of ge with samples from S by Algorithm 2: Compute features g of query samples:
11
for in {1...., NK) } do
Extract feature vectors a3 = E.g (z.) of samples from Q:
Compute visual features 2 = (a)):
Predict semantic features with (a)):
Compute final feature representations of query samples by Eq or
end
18
Compute the cross entropy loss LCE using Eq. 9
Update sp by gradient descent V VLCE.
end
Algorithm 2: Task-specific semantie feature learner optimization
Input: W {ww, v/,, |c 1, 1,...,,, i 1,... truth semantic features and w', = - (ac,) is the predicted semantic features.) Output: semantic feature learner Os
.. Kg} ). (we is the ground
while not convergence do
2 Evaluate Lasesim (+)) by Eq.
Update v by gradient descent V Ldisssimi
end
3.4.3 CLASSIFICATION LOSS OF THE VISUAL FEATURE LEARNER
The negative Fuclidean distance between the feature representation g of i-th query and the prototype Pe of the c-th support sample employed as the predicted scores of the query samples, and the distance is transformed into probability via Softmax operation, see Eq.
d (gi, Po) = q: Pell2 = L.....,, N
p cla,, EN Mis Pct I...,,NN N
where Ú: denotes the predicted label for i-th sample in the query set. Then, the loss function is formulated as the negative loglikelihood of the predicted probability scored each query sample.
Ko Σ
Lce (1) =
yA log (p (ū: 24 21/21,, sp)
We compute the cross-entropy loss between the predicted scores of the query samples and the ground truth labels for all training episodes constructed from the base dataset, Eq. 9 Parameters of the visual feature learner is optimized by minimizing Log- We summarize the whole process in Algorithm and Algorithm 2.

---

Under review as a conference paper at ICLR 2022
Table 1: The few-shot classification accuracy results on MinilmageNNet and CIFAR-FS datasets.
They present the mean accuracy on 600 novel episodes with 95% confidence interval. The numbered superscripts denote the architecture of the feature extractor: 'ReSNet12 (Franceschi
et al. 2018) ResNett8 (Li et al. 2019b), RResNer25, "WRN-28-10 10 (Zagoruyko & Komodakis 2016
indicates that the model has not run experiments on this dataset.
minilmageNet 5-way
CIFAR-FS 5-way
Model
I-shot
5-shot
I-shot
5-shot
MatchingNet (Vinyals ct al.
2016
63.08 + 0.80
75.99
0.60
ProtoNet (Snell et al. 2017
60.37 + 0.83
78.02 0.57
72.20 = 0.70
83.50
TADAM (Oreshkin (ODehhOrrehkinn
al.
2018
58,50 + 0.30
76.70 + 0.30
MTL. Sun Sun
2019
61.20 1.80
0.80
MetaOptNet' (Lee et al
2019
62.64 0.61
78.63 0.46
7.70
84.20 0.50
FEAT' Ye
2020)
66.78 0.20
82.05
AM3-TADAM (Xing et al. 2019
0.49
0.36
DeepEMD' (Zhang
2020
82.41 0.82
F3BM (Liu al.
20206)
0.90
0.90
ConstellationNet Xu et a
2020
0.23
79.95 #0.17
75.40 = 0.20
86.80 # 0.20
PSST (Chen
1
0.32 0.41
77.02 = 0.38
88.45 0.35
Beg-Cosine- (Liu et al.
2020a)
0.81
0.56
MABAS' Kim et al.
(020)
65.08
0.54
73.51 + 0.92
85.49 0.68
MetaOptNer + ArL." (Zhang et al. 2021
65.21
80.41
TasNet (concatenation)
78.68 + 0.59
85,89
80.80 = 0.70
84.86 = 0.60
TasNer' (fusion)
78.24 + 0.64
85.82 0.49
80.14 + 0.72
84.46 0.58
TADAM +Tasset'
78.57 # 0.56
85.57 1 0.41
80.47 = 0.67
84.57 0.59
MetaOptNet +Tasnet'
78.36 + 0.61
85.78 0.52
80.76 + 0.73
84.69 0.63
4 EXPERIMENTS
We conduct a comparison of our method to state-of-the-arr methods in terms of few-shot classifica- tion accuracy on four benchmarks, including MinilmageNet (Vinyals al. (016). CUB-200-201 Wah et al. 2011) CIFAR-FS (Bertinetto et al. 2018) and FCTO0 (Oreshkin et al. 2018).
4.1
ARCHITECTIRES
For the feature extractor network, we use ResNet- 12 consisting 4 residual blocks with Dropblock as regularizer and dineeninnli output features, the details follow the one proposed in TADAM (Oreshkin et al., (018). For the visual feature learner and the semantic feature learner, we use a MLP with one layer, outputting 300-dimentional features.
4.2 RESULTS AND ANALYSIS
Table and Table 2 summarize the results of the few-shot classification tasks on MinilmageNet, CIFAR-FS, FC100, and CUB-200-2011, 1, respectively. It is worth noting that our TasNet achieves the state-of-the-aat performance for both 1-shot and 5-shot settings on almost all four benchmarks.
ABLATION STUDY
To further validate the effectiveness of our method, we conduct series of ablation studies on mini- ImageNet and CIFAR-FS datasets. We first show some experimental proofs of the task-adaptive semantic feature learner. Then we show some other results on the hyperparameters.
4.3.1 EFFECTIVENESS OF THE SEMANTIC FEATURE LEARNER
The task-adaptive semantic feature learner is designed to alleviate the negative effects of similar background or similar interference, and provide another kind of modality to enrich the information of query samples, which is helpful in producing discriminative feature embeddings. It can also alleviate the few supervision problem to a certain extent. We test the model with and without semantic feature leamner. The quantitative and qualitative results shown in Figure a) and Figure 4

---

Under review as a conference paper at ICLR 2022
Table 2: The few-shot classification accuracy results on FC100 and CUB-200-2011 datasets. They
present the mean accuracy on 600 novel episodes with 95% confidence interval. The numbered superscripts have the same meaning as that in Table
FC100 5-way
CUB 5-way
Model
1-shot
5-shot
1-shot
5-shot
MatchingNet VViyyal al.
2016
72.40 = n/a
83.60 = n/a
ProtoNer Snell
41.50 + 0.70
57.10 0.70
71.90 + n/a
87.40 + n/a
TADAM' Oreshkin ct al
2018
40.10 + 0.40
56.10 0.40
MTL' Sun et al. 2019)
45.10 = 1.80
57.60 0.90
MetaOpiNer* Lee et al
2019
41.10 + 0.60
$5.50
FEAT Ye al. 2020)
68.60 = n/a
83.0 n/a
DeeD (Zhang ct al.
2020
46.50 + 0.80
1.20000.7 63.20
79.30 + 0.30
89.80 0.5
F3BM [Liu et al. 2020bb
45.0 1.30
60.50 + 0.4
ConstellationNet (Xu
2020
43.80 4 0.20
59.70 0.20
Neg- Cosine" Liu et al 2020a
72.66 + 0.85
89.40 = 0.43
MABAS (im al. 2020
42.31 # 0.75
57.56 0.78
SoSN Art. (Zhan et al.
2021
50.62 + n/a
65.87 n/a
TasNet (concatenation)
61.60 + 0.83
68.51 0.69
79.89 + 0.74
85.76 0.63
TasNet (fusion)
60.98 + 0.89
67.66 1 0.84
79.17 0.67
84.99 + 0.66
TADAM +Tasnet'
61.79 1 0.78
68.35 0.65
79.72 0.59
86. 12 0.68
MetaOptNet +Tasnet
61.24 + 0.91
67.98 0.73
79.32 = 0.71
84.75 0.72
Hisimageiiss
Visimugahie
OFAX-S
66.
5:86:
00.02
72-2
Vsal
Figure 3: (a) Quantitative comparison of the effect of different features in terms of the mean ac- curacy 600 novel episodes on 5-way 1-shot and 5-way 5-shot cases for both MinilmageNet and CIFAR-FS datasets, where visual indicates the visual features learned by the visual feature learner Jip semantic indicates the semantic features learned by the semantic feature learner ga- fusion de- notes the weighted sum the visual and semantic feature embeddings, concatenation denotes the concatenated features of the visual and semantic feature embeddings. (b) Hyperparameter analysis for the fusion approach for the semantic and visual features. We study the effectiveness of changing the value of fusion weight À on the mean accuracy of 600 novel episodes in terms of 1-shot and 5-shot cases. Where the red points denote the optimal hyperparameters for the respective cases.
As shown in Figure 3 (a), the classification accuracy has achieved a reasonable value the case of only semantic features. In the 1-shot case, the contribution of the semantic feature is especially significant, which proves the effectiveness of the task-adaptive semantic feature learner. We can also observe that, the combination of the semantic and visual features resulting in further performance improvement, which demonstrates that the semantic features and the visual features can complement each other improve the classification problem together. Moreover, performance of the concatena- tion approach of features is ahead of the fusion approach.
4.3.2 T-SNE VISUALIZATION
To qualitatively validate the effectiveness of our method, we further provide some i-SNE visualiza- tion results in Figure 4 We sample one episode in the test split of minilmageNet and CIFAR-FS datasets under the 5-way 1-shot and 5-way 5-shot settings. And then, we obtain the embeddings of all images using the visual feature learner, the semantic feature learner, and our TasNet, respectively. Across four subfigures, samples with the same color belong to the same class. Evident in these fig- ures, the overlap between different classes of embeddings generated by TasNet is much less than
8

---

Under review as a conference paper at ICLR 2022
that with only visual features, which demonstrates the effectiveness of the task-adaptive semantic feature learner in helping learning feature representations with well separated margin between novel classes. From the last two columns of Figure we can further observe that compared with the visualization of the fused feature embeddings, the discriminative performance of the concatenated feature embeddings is little superior than the former, which is in consistent with the quantitative result in Figure
visual
semmantic
Fusion
concaterattion
visual
semantic
Rasion
concutomation
1shot
Sahoo
(a) Feature embeddings samples from MinilmageNeet
Feature emhoddings of samples from CIFAR-FS
Figure 4: I-SNE visualization of the feature embeddings generated by TasNet in 5-way 1-shot and 5- way 5-shot cases with 40 query samples per class from novel classes on MinilmageNet and CIFAR- FS datasets. the x-axis notations have the same meaning as that in Figure 3(a)
In Figure B/), we vary the value of fusion weight À from 0 to I to observe the performance change. This hyperparameter controls the contribution of the semantic and visual features to the final fused features. As shown in Figure (b)) we can observe that, the optimal À for MinilmageNet in 1- shot and 5-shot cases are 0.2 and 0.3, respectively, while for the CIFAR-FS dataset, the optimal reaches 0.1 for both 1-shot and 5-shot cases. It demonstrates that, in order to improve the final classification performance, we should weigh more on the semantic features, while information from the visual side should be relatively low. Since too much attention paid the visual feature will weak the advantage brought by the semantic features. It is in consistent with the results of the ablation studies the former sections. This indicates that the selection of plays a key role for best complementing the features information of different modalities.
From the quantitative and qualitative results, we can conclude that the task-adaptive semantic feature learning mechanism is the main reason for the performance boosting.
5 CONCLUSION
In this paper, we propose TasNet, a task adaptive semantic feature learning mechanism to incorporate semantic features for the query samples. Since the feature extractor has been initialized after the pre-training stage, we freeze the parameters of the feature extractor in the meta-training and meta- testing stages. Therefore, only the semantic feature learner and the visual feature learner should be optimized, which is less time consuming. Our model trains the global classification loss and the task-adaptive semantic feature dissimilarity loss separately, which can avoid collapsing different modalities into a common feature space, so as to preserve the structural heterogeneity of different modalities. Then, we propose two features combination approaches: feature concatenation and feature fusion, to further improve the performance. Experiments show that our method outperforms the state-of-the--ad on four benchmarks. In the future work, we expect to explore semantic- guided attention mechanism to pay more attention to the targets in the images, so as to better capture class- specific information from the images, which can further alleviate the influence noise and irrelevant information.
REFERENCES
Luca Bertinetto, Joao F Henriques, Philip HS Torr, and Andrea Vedaldi. Meta-learning with differ- entiable closed-form solvers. arXiv preprint
2018.

---

Under review as a conference paper at ICLR 2022
Zhengyu Chen, Jixie Ge, Heshen Zhan, Siteng Huang, and Donglin Wang. Pareto self supervueee training for few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13663-13672, 2021.
Zitian Chen, Yanwei Fu, Yinda Zhang, Yu- Yu-Gang Jiang, Xiangyang Xue, and Leonid Sigal. Multi- level semantic feature augmentation for one-shot learning, IEEE Transactions on Image Process- ing, 28())4594-4605, 2019.
Mandar Dixit, Roland Kwitt, Mare Niethammer, and Nuno Vasconcelos. Aga: Attributegguded aug- mentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7455-7463 2017.
Li Fei-Fei, Rob Fergus, and Pietro Perona. One-shot learning of object categories. IEEE transactions on pattern analysis and machine intelligence, 28(4):594-611 2006.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. International Conference on Machine Learning, pp. 1126-1135. PMLR, 2017.
Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimiliano Pontil. Bilevel programming for hyperparameter optimization and meta-learning. In International Conference on Machine Learning, pp. 1568-1577. PMLR, 2018.
Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4367- 4375, 2018
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Ruibing Hou, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Cross attention network for sew-shot classification. arXiv preprint arXiv: 19000777,, 2019.
Jaekyeom Kim, Hyoungseok Kim, and Gunhee Kim. Model-agnostic boundary-adversarial sam- pling for test-time generalization in few-shot learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 28-28. 2020, Proceedings, Part I 16, pp. 599-617. Springer, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXhv:1412.6980, 2014.
Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. Siamese neural networks for one-shot image recognition. In ICML deep learning workshop, volume 2. Lille, 2015.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. Advances in neural information processing systems, 15:1097- 2012.
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10657-10665, 2019.
Aoxue Li, Tiange Luo, Tao Xiang, Weiran Huang, and Liwei Wang. Few-shot learning with global class representations.
Proceedings of the IEEE/VVF International Conference on Computer Vision, pp. 9715-9724, 2019a.
Aoxue Li, Weiran Huang, Xu Lan, Jiashi Feng, Zhenguo Li, and Liwei Wang. Boosting few-shot learning with adaptive margin loss. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12576-12584, 2020.
Hongyang Li, David Eigen, Samuel Dodge, Matthew Zeiler, and Xiaogang Wang. Finding task- relevant features for few-shot learning by category traversal. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1-10, 2019b.
10

---

Under review as a conference paper at ICLR 2022
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few- shot learning. arXiv preprint arXiv: 2017.
Bin Liu, Yue Cao, Yutong Lin, Li, Zheng Zhang, Mingsheng Long, and Han Hu. Negative margin matters: Understanding margin in few-shot classification. In European Conference on Computer Vision, pp. 438-455. Springer, 2020a.
Yaoyao Liu, Bernt Schiele, and Qianru Sun. An ensemble of epoch-wise wise empirical bayes for few- shot learning. In European Conference on Computer Vision, pp. 404 4211. Springer, 2020b.
Saeid Motiaan, Quinn Jones, Seyed Mehdi Iranmanesh, and Gianfranco Doretto. Few-shot adver- sarial domain adaptation. arXiv preprint 1rXiv:1711.025330 2017.
Boris N Oreshkin, Pau Rodriguez, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. arXiv preprint arXiv: 10123, 2018.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word
representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532 1543, 2014.
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua Tenenbaum, Hugo Larochelle, and Richard S Zemel. Meta-learning for semi-supervised few-shot classifica- tion. arXiv preprint arXiv: 100::88030000 8803000676, 2018.
Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osin- dero, and Raia Hadsell. Meta-learning with latent embedding optimization. arXiv preprint arXiv:1807.05560. 2018.
Eli Schwartz, Leonid Karlinsky, Rogerio Feris, Raja Giryes, and Alex M Bronstein. Baby steps towards few-shot learning with multiple semantics. arXiv preprint 10XX:v:900.0110. 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1409 1556, 2014.
Jake Snell, Kevin Swersky, and Richard S Zemel. Prototypical networks for few-shot learning. arXiv preprint arXiv:1703.0511 2017.
Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. Meta-transfer learning for few-shot learning. Proceedings the the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 403 412, 2019.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pp. 1199-1208, 2018
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. arXiv preprint arXiv:1606.044000 2016.
Catherine Wah, Steve Branson, Peter Welinder, Pictro Perona, and Serge Belongic. The caltech-uesd birds-200-2011 dataset. 2011.
Yaqing Wang, Quanming Yao, James Kwok, and Lionel Ni. Generalizing from few examples: A survey on few-shot learning. ACM Computing Surveys (CSUR), 53(3):1-34, 2020.
Chen Xing, Negar Rostamzadeh, Boris N Oreshkin, and Pedro Pinheiro. Adaptive cross-modal few-shot learning. arXiv preprint arXiv: 1902.090044 2019.
Weijian Xu, Huaijin Wang, Zhuowen Tu, al. Attentional constellation nets for few-shot learning. In International Conference on Learning Representations, 2020.
Hin-Jia Ye, Hexiang Hu, De-Chuan Zhan, and Fei Sha. Few-shot learning via embedding adaptation with set-to-set functions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8808-8817, 2020.

---

Under review as a conference paper at ICLR 2022
Zhongqi Yue, Hanwang Zhang, Qianru Sun, and Xian-Sheng Hua. arXiv preprint arXiv:2009. 130000000,,,0,,, 2020.
Interventional few-shot learning.
Sergey Zagoruyko and Nikos Komodakis. 1605.0:605 2016.
Wide residual networks.
arXiv preprint
Chi Zhang, Yujun Cai, Guosheng Lin, and Chunhua Shen. Deepemd: Few-shot image classifica- tion with differentiable earth mover's s distance and structured classifiers. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12203-12213, 2020.
Hongguang Zhang, Piotr Koniusz, Songlei Jian, Hongdong Li, and Philip HS Torr. Rethinking class relations: Absolute-relatic supervised and unsupervised few-shot learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9432-9441, 2021.
A APPENDIX
A.1 SETTINGS
We train the feature extractor En with the SGD optimizer using an initial learning rate of 10-1 Adam (Kingma & Ba 2014) is used to optimize the parameters of the visual and semantic feature learners with an initial learning rate of 10-3 During few-shot learning stage, we train fo and go by adopting the episodic training procedure with N-way K-shot training tasks as described in (Snell eet al., 2017). The model is trained for 100 epochs, with each epoch consisting of 600 randomly sampled episodes for both I-shot and 5-shot cases on MinilmageNet,, CUB-200-2011, CIFAR-FS and FC100 datasets, and is evaluated by averaging metrics over 600 randomly generated episodes from D,,, with each episode having 15 randomly sampled query samples. Parameters of the model except the backbone layers are trained on 5-way, 15 query samples for both 1-shot and 5-shot cases per episode.
A.2 EXPERIMENTS ON HARD TASKS
To further validate the effectiveness of TasNet in discriminating different targets from similar back- ground and interference, conduct an additional experiment on hard tasks, which are constructed of samples with similar background and interference.
Firstly, have constructed sub dataset (we name it hard dataset) from the test split of minilm- agenet dataset manually. contains 8 classes, including lion, golden retriever, goose, coral reef, vase, spider web, ant, house finch. Among samples from all these categories, some of samples from two or more categories have similar background or similar interference, for example, vase and ant categories, flowers are the common interference in the images from these two classes, and for the bird and spider web categories, trees and grassland are common similar backgrounds in the images from these two classes. Then we construct 20 tasks from this hard dataset, and compare our TasNet with ProtoNet and AM3 methods on this dataset. AM3 is method that combine semantic features for the support samples, while only visual features are extracted from the query samples. The mean accuracy is show in table 3.
Table 3: The fhw-shot classification accuracy results hard datasets sampled from minilmageNet. They present the mean accuracy on 50 novel episodes with a 95% confidence interval.
Hard dataset 5-way
Model
1 1-shot
5-shot
ProtoNet (Vinyals et al.
2016
44.93 0.71
70.67 + 0.58
AM3 (Snell al. 2017)
48.11 0.69
71.24 + 0.61
TasNet
50.97 0.76
72.02 + 0.69
In addition to the statistical results, we show the classification results of one hard task in Figure 5, which is carried on 5-way 1-shot case. Five categories of ant, house finch, spider web, vase, coral reef, are included this task. Each class has support samples and 15 query samples, that is the number of test samples is 5x 15=75.
12

---

Under review as a conference paper at ICLR 2022
reppar
query (house finch)
0
query (ant)
3
2
2
3
2
2
2
query (vase)
2
2
2
3
2
2
4
3
2
2
2
2
3
2
2
2
2
2
2
2
2
2
2
suport
(spider web)
3
3
0
3
3
3
query (coral reef)
4
4
1
2
4
1
4
2
1
2
4
1
4
2
2
2
Figure 5: Detailed classification result of one hard task. where a denotes ProtoNet, denotes AM3, denotes TasNet. The number ID 0-4 below y denotes the class labels. The red number means the sample is miss classified.
From Figure 5, we can see that there are 33, 36 and 39 samples correctly classified with the ProtoNet, AM3 and our TasNet methods.
ProtoNet
AM3
TasNet
Figure 6: TSNE visualization of the query samples from one hard task.
A.3 ANALYSTS OF THE ABLATION STUDY RESULTS
From the quantitative and qualitative results in section 4.3.1 and section 4.3.2, we can conclude that the task adaptive semantic feature learning mechanism is the main reason for the performance
13

---

Under review as a conference paper at ICLR 2022
boosting. To make the ablation study results more obvious, we put the accuracy in terms of four kinds different features in table 4.
Table 4: Ablation study results
Features
minilmageNet 5-way
CIFAR-FS 5-way
visual
semantic
fused
concatenated
I-shot
5-shot
1-shot
5-shot
60.37
78.02
12.2
83.5
75.57
83.23
79.97
84.13
78.24
85.82
80.14
84.46
78.68
85.89
80.8
84.86
The accuracy computed with the semantic features shown in the second row of table 4 (in red color). When only the semantic features are used in the model, it has achieved a significant accuracy, which demonstrates that the semantic features for the query samples can certainly work.
A.4 TRAINING THE SEMANTIC AND VISUAL FEATURE LEARNER JOINTLY
In order to prove that training the visual feature learner across all tasks and training the semantic feature learner within each task an effective approach, we have made some experiments to jointly train the visual and semantic features learners. The detailed implementation of the jointly training method is described as follow.
We constructed cross entropy loss to optimize the visual feature learner, and word embedding fitting loss, i.c. the cosine dissimilarity loss to optimize the semantic feature learner.
The loss Lcg for training visual feature learner is defined averaged across all query-support pairs. It compares the Euclidean distance between the query features and the class mean of the support features to get the predicted score.
cross-entropy classification loss
d (gi, Pe) q: Pellz
i= 1=1....,NKK NKę
(10)
e dla. Pc)
(ύ: = clus (0) =
C I......, N
(11)
Σk d(g,, pk))
where i denotes the predicted label for i-th sample in the query set, and is the ground truth label of sample g: Then, the loss function is formulated as the negative loglikelihood of the predicted
probability scored of each query sample.
Kg
LCE (4p)
Σ (=1,0E
yA log (p (ū: y1 [z,, sp)
(12)
The word embedding fitting loss Latissim (vb) defined dissimilarity between the predicted se- mantic features and the ground-truth semantic features.
K
Ns
Wc
w
Cdissim (1)
(13)
Wo:
||
1=1
where We; denotes the predicted semantic feature of the i-th sample belonging to class c, and we denotes the ground truth semantic feature of class We train the semantic feature learner within each task to produce task-specific parameters. Parameters of semantic feature learner are optimized by minimizing Ldissim (1)). Finally, combine the cross-entropy classification loss and the word embedding fitting loss via weighted sum, A=055, and train across all tasks to minimize the total loss
List-
Liot XLOE (1 D)
(14)
14

---

Under review as a conference paper at ICLR 2022
Table The mean accuracy the joint training method (of the semantic and visual feature learners) on 600 novel episodes with a 95% confidence interval on minilmage.Net and CIFAR-FS datasets.
minilmageNet 5-way
CIFAR FS 5 way
Model
1-shot
5-shot
1-shot
5-shot
TasNet (concatenation)
78.68 0,59
85,89 + 0,47
80,80 0.70
84.86 1 0.60
TasNet((fsionn)
78.24 0.64
85.82 0.49
80.14 0.72
84.46 0.58
Train jointly
63.48 1: 0.80
76.75 + 0.61
64.72 0.77
77.70 +: 0.57
The experiment results show that the jointly optimization method of the visual and semantic feature learners can not achieve good performance.
15


Reviews:
[{"review_id": "iGUbP1ireSl", "reviewer": "Reviewer_7a88", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ A task-adaptive semantic feature learner is proposed to incorporate semantic features for both support and query samples.\n+ Two modality combination implementations, feature concatenation and feature fusion, are proposed.\n+ The visual and semantic features are learned separately, thus preserving the structural heterogeneity of different modalities.\n+ Experimental results are provided on four benchmark datasets that demonstrate that the proposed approach outperforms existing approaches.\n\nWeaknesses:\n- A clear definition of N-way K-shot few-shot tasks is not provided.\n- What does interference mean in the context of Figure 1?\n- What is the definition of semantic features? Are they text-based features?\n- How are visual-feature class centers and label-derived semantic features computed?\n- What is the difference between D_b, D_v, and D_n datasets?\n- Effort has to be put in Section 3 to improve its clarity and readability. It is highly recommended to clearly define the problem to be addressed and the approach to be outlined along with any notation in the beginning of the section.\n- No discussion of the baselines and their settings is provided. Also, what is the meaning of dashes in Tables 1 and 2?\n- It is not clear why jointly learning visual and semantic features will not improve performance?\n\nSome minor comments follow:\n- Abstract: \"to incorporates\" -> \"to incorporate\"\n- pg. 1: Please define acronyms before they are first used, e.g., FSL\n- pg. 2: \"Few-shot classification methods exhibits\" -> \"Few-shot classification methods exhibit\"\n            \"learning a discriminative representations\" -> \"learning discriminative representations\"\n- pg. 3: \"the these methods is\" -> \"these methods is\"\n- Table 2 caption: \"in Tabel1\" -> \"in Table 1\"\n- pg. 9: \"except\" -> \"expect\"\n- pg. 6: \"We computes\" -> \"We compute\"", "summary_of_the_paper": "This paper proposes a task-adaptive semantic feature learning mechanism to incorporate semantic features for both support and query samples. Two modality combination implementations, feature concatenation and feature fusion, are devised based on the semantic feature learner. Experimental results are provided on four benchmark datasets that demonstrate that the proposed mechanism outperforms existing methods.", "summary_of_the_review": "Overall, I recommend the rejection of the paper. My major concern is that the main contribution of the paper, i.e., separately learning the visual and semantic features, is not very intuitive and well justified. Furthermore, the proposed modality combination approaches are well-known methods.\n\n----After rebuttal----\n\nI would like to thank the authors for their effort to answer my questions. Even though the technical novelty of the paper is limited, as also supported by the new experiments added, the responses to my questions are satisfying. Thus, I increase my score from 3 to 5.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FcXbUfvWZPM", "reviewer": "Reviewer_QNok", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- Using additional semantic information such as class label embeddings is interesting in few-shot learning problems.\n\nWeaknesses\n- The proposed method seems incremental to me because the only difference from the existing method is that it predicts the semantic features for the query set from the support set and uses it for the few-shot classification.\n- There is room for improvement in the clarity of the paper. For example, I didn't know what \"semantic features\" meant until I read section 3.2.\nI recommend describing concrete examples of it at the early stage of this paper such as Introduction, which also clarifies the problem setting tackled in this paper.\nIn addition, the explanation of the proposed method (Section 3, 3-1, and 3-2) seems unnecessarily redundant. \n- The experiments are unconvincing. The effect of using pre-trained feature extractor $E_{\\theta}$ is unclear. Did comparison methods use this pre-trained extractor in the experiments? Although the proposed method seems to work well in Tables 1 and 2, I don't know if the semantic features for query samples introduced in the proposed method are working because the description of the comparison methods is insufficient. Did you confirm that the effect of learning the semantics feature learner locally per each task? \n\nMinor Comments\n- index $c_i$ is a bit confusing in eq. (2) since $c_i \\in$ { $1,\\dots, N$ } . ${\\cal S} _c$ is undefined in eq. (3).\n\n", "summary_of_the_paper": "This paper proposes a few-shot learning method that incorporates semantic features (i.e., class label embeddings) for both support and query samples.\nAlthough some existing methods use the semantic features for support samples to improve the performance, no methods use them for query samples. \nTo this end, the proposed method regresses the semantic features for the query samples from the support samples.\nExperiments show the effectiveness of the proposed method.", "summary_of_the_review": "I have some concerns described in Main Review. Therefore, I recommended \"3: reject, not good enough\".\n\n----After rebuttal----\n\nThanks for the response. I still have some doubts about the technical novelty of the proposed method, but\ngiven the other reviews and response, and the strength of the experimental results, I increase my score from 3 to 5.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9rqm4PkIROu", "reviewer": "Reviewer_LGLG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposed a reasonable model to use the text labels in image classification. The experimental results look convincing. With that said, the proposed solution looks overly complicated.  A simple baseline should be considered: a multi-task learning model with two outputs, one to predict the image class,  the other to fit the Glove embedding. Compared with this simple baseline, the extra complicity of the proposed model should be better motivated. Also it would be good to have an empirical comparison with the simple two-outputs baseline.\n\n", "summary_of_the_paper": "In few-shot learning for image classification, visual features alone may represent multiple objects in an image. The label text provides additional information that could serve as useful inductive bias, indicating which object in an image the model should pay attention to. There are multiple ways to use such text information to help few-shot classification. The authors propose to build a model that predicts both the image classes and the text label (or its Glove embedding). In other words, an additional loss term is introduced to regulate the learning of visual features. The authors show that the additional \"semantic features\" (because Glove embedding is used) help improve few-shot image classification.", "summary_of_the_review": "The authors use additional information in the training labels to improve few-shot image classification. The solution is a combination of different existing tricks. Without proper ablation experiments, it is difficult to infer generalisable knowledge from the results. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5ZcsWYZSng", "reviewer": "Reviewer_wrVH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\nThe paper is clearly written and easy to follow. \nEmpirically they show that using such semantic features clearly shows improvements over the visual features. \n\nQuestions:\n(i) The method (TASNet) is built over Protonet (Snell et al) method. Does it show similar performance gain with other popular architectures such as MetaOptNet, etc.?\n(ii) Details on the architectures of visual feature learner (f) and semantic feature learner (g) are missing?\n\nWeakness:\nThe motivation for the paper involves robust few-shot learning with different categories under similar background, interference. The results are only on the full test set. Results on a few episodes where such conditions exist would cement the robustness of the method.", "summary_of_the_paper": "This paper describes a few-shot learning approach that takes into account the semantic relatedness of the ground truth labels rather than just using them as binary labels. In the few-shot learning setting, the paper describes a method to train the semantic feature learner (on labels) during the meta-training phase and then at meta-test time use the semantic feature learner to get the said features for the query images. This way both the support and query examples would have semantic features and empirically it has shown to work on many few-shot benchmarks on computer vision datasets.  ", "summary_of_the_review": "Overall the paper suggests a method that makes an improvement over an existing state-of-the-art, but the efficacy of their enhancements are not fully tested. In principal adding textual data that is semantically less ambiguous than image data helps. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a few-shot learning method that learns task-adaptive semantic features that can incorporate for both of the support and query sets. Two approaches for modality combination are developed. The additional experiments in the author response addressed some concerns of the reviewers. However, the technical novelty of the proposed method is high enough since the proposed method uses existing techniques.
Decision: Reject