Fast Task Adaptation for Few-Shot Learning
Authors: Yingying Zhang
Qiaoyong Zhong
Di Xie
Shiliang Pu
Venue:  (2020)
Keywords: Few-Shot Learning
Metric-Softmax Loss
Fast Task Adaptation

Abstract: Few-shot classification is a challenging task due to the scarcity of training examples for each class. The key lies in generalization of prior knowledge learned from large-scale base classes and fast adaptation of the classifier to novel classes. In this paper, we introduce a two-stage framework. In the first stage, we attempt to learn task-agnostic feature on base data with a novel Metric-Softmax loss. The Metric-Softmax loss is trained against the whole label set and learns more discriminative feature than episodic training. Besides, the Metric-Softmax classifier can be applied to base and novel classes in a consistent manner, which is critical for the generalizability of the learned feature. In the second stage, we design a task-adaptive transformation which adapts the classifier to each few-shot setting very fast within a few tuning epochs. Compared with existing fine-tuning scheme, the scarce examples of novel classes are exploited more effectively. Experiments show that our approach outperforms current state-of-the-arts by a large margin on the commonly used mini-ImageNet and CUB-200-2011 benchmarks.

Under review as a conference paper at ICLR 2020
FAST TASK ADAPTATION FOR FEW-SHOT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Few-shot classification is challenging task due to the scarcity of training ex- amples for each class. The key lies in generalization of prior knowledge learned from lcage base classes and fast adaptation of the classifier to novel classes. In this paper, we introduce two-stage framework. In the first stage, we attempt learn task-agnostic feature on base data with a novel Metric-Softmax loss. The Metric-Softmax loss is trained against the whole label set and learns more dis- criminative feature than episodic training. Besides, the MetrrieSofftmx classifier can be applied to base and novel classes in a consistent manner, which is critical for the generalizability of the learned feature. In the second stage, we design a task-adaptive transformation which adapts the classifier to each few-shot setting very fast within few tuning epochs. Compared with existing fine-tuning scheme, the scarce examples of novel classes are exploited more effectively. Experiments show that our approach outperforms current state-of-ther a-tttrr-rrs by a large margin on the commonly used mini- Amni-lmaeeeet and 200-2011 benchmarks.
INTRODUCTION
In recent years, deep learning models have achieved great success in many visual tasks such as image classification (Krizhevsky et al., 2012), object detection (Girshick, 2015) and semantic seg- mentation (Long et al., 2015). In general, training deep network needs massively labeled instances which leads to expensive manual annotation cost. In contrast, humans can learn novel concepts from only one or a few examples. Analogously, few-shot learning aims to recognize unseen instances by accessing just small number of labeled images in each class. Unfortunately, naive methods such as re-training or fine-tuning the model on novel data would severely suffer from overfitting and provide poor result (Finn et al., 2017). The major difficulty lies in the exceedingly limited data, which can hardly represent the class distribution.
In the task of few-shot classification, we are given three datasets, namely training set, support set and query set. The training set (also known as base dataset) normally contains large-scale labeled data for learning of prior knowledge. The support set and query set contain small number of examples which are drawn from unseen classes disjoint with the training set. We are expected to classify the query images, leveraging the prior knowledge from the base training set and the limited cue from the support images. If the support set contains k annotated images for each of m unique classes, then this task is called m-way k-shot classification.
Previous works on few-shot learning can be roughly divided into two categories, namely meta- learning based and metric learning based. The former typically learns meta-learner model on base data and generalizes it to novel unseen data. Usually, a recurrent neural network (Santoro al., 2016) or long short-term memory network (Hochreiter Schmidhuber, 1997) is utilized to learn a memory network to store knowledge. The latter attempts to learn feature embedding where samples of the same class are closer to each other than samples of different classes. To make training on base data and inference on novel data consistent, a so-called episodic training strategy is widely employed (Vinyals et al., 2016; Snell et al., 2017: Sung et al., 2018). There are two limitations in metric learning based approaches. One is that episodic training only considers local sample similarity in current data batch. As shown in Snell et al. (2017), the limited number of classes per- training episode is harmful for discriminative feature learning, while simply increasing it leads to better performance. However, in a sampled episode, a large number of classes is infeasible due to limited GPU memory. The other limitation is that most existing methods only utilize the labeled

---

Under review as a conference paper at ICLR 2020
probability
label
Metric-Softmax classifier Ut dxe
backbone network fe
loss
y
W
base class images
Task Aanks-gnotiic Feature Learning Stage
probability score
label
W (mrin)
loss
fixed f
y
class In
W
R xm
novel class images support set
Fast Task Adaptation Stage
Figure 1: Overview of the proposed FTA framework, which is composed of two stages.
support images in novel classes templates for feature similarity comparison. We argue that the support images could be better exploited to improve the performance.
In this work, introduce two-stage framework named Fast Task Adaption (FTA). As illustrated in Figure 1, the few-shot classification task is decomposed into two steps, i.e. task-agnostic feature learning base data and adaptation of the classifier each few-shot learning task. To learn more general and discriminative feature on base data, we propose a novel Metric-Softmax loss, which is an improvement of the Softmax loss. Specifically. we replace the score computing softmax with Gaussian kemel--bsse baanel radial basis function. The Metric-Softmat loss is trained against the full base classes, overcoming the limitation of small number of classes episodic training. Moreover, the Metric-Softmax classifier can be applied to base and novel classes in the same way. The consistency between training and inference well preserved. After task-agnostic feature is learned, we design novel taskaadaptive transformation to adapt the classifier to current few-shot learning task. It is essentially an affine transformation applied to features novel images. Notably, compared with fine-tuning re- re-initialized weights, it exhibits the desirable characteristics of fast convergence and superior performance. In this way, the aforemmentooned weaknesses of existing metric learning based methods well addressed, leading significant accuracy improvement. Experimental results the mini-ImageNet and CUB-200-2011 datasets show that our approach consistently outperforms current state-of-the-art methods with respect various backbone networks.
The main contributions of this paper can be summarized as follows.
We propose the Metric-Softmal loss, which learns highly discriminative feature and en- sures the consistency between training and inference as well.
We design a novel task adaptive transformation adapt the classifier novel classes effi- ciently and effectively.
We improve few-shot classification significantly and achieve state-of-the-art performance on common few-shot learning benchmarks.
RELATED WORK
In this section, we briefly review the previous approaches relevant to ours and current state-of-the-art methods we compare with in experiments.
Meta-learning based methods aim to learn task-agnostic knowledge from a sequence of similar tasks which have enough training data and apply it to tasks for fast adaption. Ravi & Larochelle
2

---

Under review as a conference paper at ICLR 2020
(2017) utilize an LSTMM-based network to control the parameter optimization process, which results in better generalization than SGD when training on a few labeled samples. Qiao et al. (2018) pro- pose adapt pre-trained network to novel classes by directly predicting the parameters from the activations. MAML (Finn et al., 2017) learns sensitive and general initial parameters, which can fast adapt a novel task with only one few steps of gradient-descent update. MTL (Sun al., 2019) learns to adapt deep neural network for few-shot learning tasks by learning scaling and shifting functions of weights for each task. Cai al. (2018) use a contextual learner to predict the parameters of an embedding network for unlabeled data by using memory slots. Mishra et al. (2018 18) propose a meta-learner architecture that combines temporal convolutions and soft attention.
Metric learning based methods aim to learn feature embedding that preserves the class neigh- borhood structure. Specifically, samples of the same class are closer than samples of different classes in the learned embedding space. Matching networks (Vinyals et al., 2016) combine attention and memory together by learning a network that maps a small labeled support set and an unlabeled ex- ample to its label. They also first introduce the episodic training strategy. where the training process mimics the test scenario based on support-query metric learning. Prototypical network (Snell et al., 2017) uses the mean feature of each class its corresponding prototype representation to learn similarity metric space. Relation network (Sung et al., 2018) learns a deep non-linear distance met- ric by considering the relation between query images and support images. Satorras & Estrach (2018) propose a graph neural network to learn the feature similarity between query and support set. Our approach belongs this category. Compared with existing methods, our major improvements are the Metric-Softmax loss for generic feature learning and the task-adaptive transformation for fast task adaptation.
Other methods. Zhang et al. (2018) propose a GAN-based approach to help few-shot classifiers to learn sharper decision boundary. Si al. (2019) propose a progressive cluster purification method for transductive few-shot learning. Li et al. (2019) propose a deep nearest neighbor neural network for few-shot learning. Chen et al. (2019) give a consistent comparative analysis of several repre- sentative few-shot classification algorithms and show that the depth of backbone network matters. Besides, they attempt to fine-tune a Softmax classifier on the support set. On the contrary. we first apply an affine transformation to features of the support images, and compute the class centroids of the transformed features as the classfier's weights. Concretely, we learn the affine transforming matrix rather than directly fine-tune the randomly initialized weights. Optimizing the transformation is computationally more stable even when one example per-class is available.
3 METHOD
As shown in Figure 1, the proposed FTA framework is composed of two stages, namely task- feature learning stage and fast task adaptation stage. In the first stage, we train the network on the base data with the proposed Metric-Softmax loss. After that, we fix the parameters of the network backbone and use it as a feature extractor. In the second stage, we learn task-adaptive classifier, which is also a Metric-Softmax classifier with its weights replaced with affine transformed features of the support images. The affine transformation g is named Adaptivee Transformation (TAT), which is learned on the scarce support set. With proper initialization, training of g converges very fast within few epochs. After training finishes, y is applied to both support and query images.
In this section, we first elaborately describe the proposed Metric-Softmax loss. Since it is an im- provement of the Softmax loss, we first give brief review of the Softmax loss and reveal the issue of discrepancy between training and inference. Then we explain how to address this issue with the Metric-Softmax loss, Finally, we introduce the task-adaptive transformation for fast task adaptation.
3.1 SOFTMAX CLASSIFIER
Feature learning with the Softmax loss has achieved great success in many feature ranking tasks like person re-identification (Zhang et al., 2017) and face verification (Wen et al., 2016). Given an image X of base classes, backbone network f is applied to project X into feature space h = f(X), h E Rd. Then, the logits are computed by feeding h into an affine transformation:
z W h + b.
3

---

Under review as a conference paper at ICLR 2020
where W E R and b R are the transforming weights. c the number of classes in base data. The Softmax classifier predicts the probability score that the image belongs to class by
exp(zi)
- soltmax(2)) = Ej exp(2;)
The Softmax loss is an assembly of the Softmax classifier and cross-entropy loss. Suppose the one-hot category label of the image is y E 1,, the training cross-entropy loss is defined as
==
Σ yo log(s;)
During inference, if test image belongs to one of the training classes, its label can be predicted by
ÿ arg max S;
(4)
In this case, the scores in both training and inference are computed by an affine transformation and softmax function applied to the feature vector h. However, in the context of few-shot learning, the query image for testing belongs to novel class unseen in training. A common way to classify the novel query image is compare its similarities to the few labeled support images, which share the same label space with the query image. Concretely, for m-way k-shot classification, we can extract features of the support images, and compute the mean feature for each class by
k 1 Te E(())
h:
(5)
Here, X denotes the j-th image of class i in the support set. The label of query image X can be predicted by
ÿ = arg min (I hall
(6)
Rather than an inner- mener-prouuct between and column vectors the weight matrix (z, = W + +b)) followed by softmax function in the previous case, the prediction is based the euclidean distance between the query and support features. We argue that this discrepancy hurts the transferability of features pre-trained on base data novel data.
3.2 METRIC-SOFTMAX CLASSIFIER
To eliminate this discrepancy, we improve the Softmax classifier by redefining the probability score calculating function. Specifically, we replace the exp(2;) term in Equation 2 with Gaussian kernel- radial basis function, whose center is the column vector of a learnable weight matrix W.,, W E gdxe, Then the score can be computed by
exp(-ai l W.,l)
8i cxp(ppar
W.
where a R. a 0 hyper-parameter for scaling and h is an 12-normalized feature vector, In this way, it can be easily derived that arg max s; is equal to arg min || W. Here W., can
be interpreted as the centroid of class in the learned embedding space. That means the training procedure is essentially optimizing the euclidean distance-based similarity between images to their corresponding class centroids. During training, the cross-entropy loss adopted as the Softmax loss.
During inference, we replace W., with the mean feature (see Equation 5) of class i in the support set. Then classification of a query image is conducted according to Equation 4. Like episodic training, the consistency between training and inference is well preserved. And we easily enjoy the benefit of more general and discriminative feature learned with the cross-entropy loss. It is worth noting that the number of classes during inference on novel data is independent from the base dataset, which makes it flexible enough to adapt to different few-shot classification tasks.
4

---

Under review as a conference paper at ICLR 2020
3.3 FAST TASK ADAPTATION
Although the feature learned on the large-scale base data is highly general and discriminative, unlikely to perfectly fit arbitrary novel classes. A proper adaptation each few-shot task is beneficial. To leverage the valuable cue provided by the support images, straightforward choice fine-tune a specific Softmax classifier as in Chen et al. (2019). However, even with the feature extractor being fixed, it is still difficult to train the classifier due to easy overfitting small dataset. In contrast, we design a parameterized transformation function g to adapt the general feature to current few-shot learning task,
h' g())
Concretely, y is simply zero-offset affine transformation: g(h) = h M E Rddd For way k-shot classification, we reconstruct an m-class Metric-Softmax classifier whose weight matrix W Raxm is derived from the transformed features of the support images
k W.,
1 Σ (
Here ((.)) denotes the feature of j-th image in class i. It is worth mentioning that both the trans- formed feature vector ( and each column of the weight matrix W should be L2-normalized. And now, the computation of probability score is defined as
exp(- 8, Σj exp((aallg(h Wllll
W.ill)
(10)
The transformation g is trained on all labeled images in the support with the cross-entropy loss. To case optimization, the transforming matrix M is initialized with an identity matrix. After training of finishes, it is applied to both support and query images, And the final prediction of query image X is computed by
9 = arg min [lx((( - W.ill
(11)
4 EXPERIMENTS
4.1 DATASETS
We conduct few-shot classification experiments on two common benchmarks, namely mini- ImageNet (Vinyals et al., 2016) and CUB-200-201 (Wah et al., 2011).
mini-ImageNee is first proposed by Vinyals et al. (2016) for few-shot classification evaluation. consists of 100 classes and 600 images per class, which subset sampled from the ImageNet dataset (Deng et al., 2009). In our experiments, we follow the dataset splits proposed in Ravi & Larochelle (2017), which takes 64, 16 and 20 classes for training, validation and testing respectively.
CUB-200-2011 is initially proposed for fine- fine-grained classification of birds. It contains 200 cate- gories and 11,788 images in total. We follow the evaluation protocol used in Hilliard et al. (2018), which randomly splits the dataset into 100, 50 and 50 classes for training, validation and testing.
4.2 IMPLEMENTATION DETAILS
To make a fair comparison with existing works, we evaluate our method for three commonly used backbone networks, namely Conv-4 (Vinyals al., 2016). ResNet-10 (Chen et al., 2019) and ResNet- (Oreshkin et al., 2018) Besides difference in depth and architecture, Conv-4 and ResNet- 12 expect an input size of 84 x 84, while ResNet-10 takes 224 x 224 images as input, following mainly focus the 5-way I-shot and 5-way
the setting in existing works. In our experiments, 5-shot classification settings which are the most commonly used in existing works. The test episode contains 5 classes and each class contains (or 5) support image(s) and 15 query images. For all experiments, we report the mean accuracy over 1200 randomly sampled episodes and the 95% confidence interval.
5

---

Under review as a conference paper at ICLR 2020
Few-shot classification results on the test set the mini ImageNet dataset. ported by Chen et al. (2019).
Table 1:
*Results re-
Method
Backbone
5-way 1-shot
5-way 5-shot
NatchingNet" (Vinyals et al., 2016)
48.14410..8
63.48+00666
ProtoNet (Snell et al., 2017)
49,42=10788
68,20:10.66
MAML (Finn et al., 2017)
48.70=1.84
63.11=0.92
RelationNet (Sung et al., 2018)
50,44=0082
65.32++070
Conv-4
PABN (Huang et al., 2019)
51.87=00.45
65.37=0068
Baseline++ (Chen et al., 2019)
48.24=10.75
66.43+00.3
DN4 (Li et al., 2019)
51.24:1:0.74
71.02:10.64
FTA (Ours)
52.13
80.35
MetaGAN (Zhang al., 2018)
52.71-10.64
68.63++0.67
SNAIL. (Mishra et al., 2018)
55.71++0.99
68.88+00.2
AdaResNet (Munkhdalai al., 2018)
56.88=10.62
71.94:0.57
TADAM (Oreshkin et al., 2018)
58.5
76,7=10.3
ResNet-12
MTL (Sun al., 2019)
61.2
75.5
Ravichandran et al. (2019)
59.00
77.46
MetaOptNet-SVN (Lee et al., 2019)
62.6410.61
08.63:40446
FTA (Ours)
58.03
80.73:10.44
Our implementation is based on PyTorch (Paszke et al., 2017). We apply the same data augmentation Chen al. (2019), including random cropping, horizontal flipping and color jittering in both feature learning and fast task adaptation stages. The Adam (Kingma & Ba, 2015) optimizer with 0.9 and 32 0.999 is used. In the feature learning stage, the backbone network is trained from scratch for 300 epochs in total with a batch size of 32. We decay the learning rate by factor of 0.1 every 75 epochs. The scaling factor used in Metric-Softmax is set to 15 for mini-Imagell and for -UBB-200-201. 1. In the task adaptation stage, we use the same optimizer with different learning rate. We set the learning rate to 0.005 for the 1-shot setting and 0.05 for the 5-shot setting. is set to 0.25 and 2 for the 1-shot and 5-shot settings respectively, We fine-tune the weights of the transformation y for 20 epochs. Note that all the hyper-parameters are determined by the performance the validation set of each dataset.
10-3
4.3 COMPARISONS WITH THE STATE-OF-THE-ARTS
Results on mini-ImageNNe. Table shows a comparison of our method to current state-of-the-an on the mini- -mai-Im dataset. Classification accuracies reported for two backbone networks, i... Conv-4 and ResNet-12 12. For Conv-4 backbone, we achieve the best accuracy both 5-way 1-shot and 5-way 5-shot settings. Our method boosts the performance for the 5-shot setting, surpassing the second best by 9.33% and 2.10% for the Conv-4 and ResNet-12 12 backbones respectively. closer analysis on the larger improvement for the 5-shot setting than 1-shot setting is given in the discussion section.
On this dataset, we find that there is work reporting the few-shot performance for the ResNet-12 backbone. For fair comparison, we report the accuracies of the Conv-4 and ResNet-10 backbones. As shown in Table 2, similar to mini-ImageNet, outperforms the second best method by large margin (87.92% vs. 81.90% for Conv-4 and 92.89% vs. 87.45% for ResNet-10) in the 5-way 5-shot setting. In the 1-shot setting, we achieve the best performance for ResNet-10 and comparable accuracy to PABN (65.11% vs. 66.71%) for Conv-4. is worth mentioning that in PABN, the max- max-pooling layers of the backbone network are replaced with bilinear pooling, which is specially optimized for fine- mee-grrined classification tasks. We expect further accuracy improvement by incorporating such techniques in our method.
Results
method
Results under Domain Shift. To evaluate the generalizability of the aask-agnostic feature learned with the proposed Metric-Softmax loss, perform the cross domain experiments. In the cross

---

Under review as a conference paper at ICLR 2020
Table 2: Few-shot classification results the test set of the CUB-200-201] dataset.
Method
Backbone
5-way 1-shot
5-way 5-shot
MatchingNet (Vinyals al., 2016)
61.16110.9
72.86110.70
ProtoNet (Snell et al., 2017)
51.31
70.7710.69
MAML (Finn et al., 2017)
55.92=
72.09=0.76
RelationNet (Sung et al., 2018)
62.45+0.98
76.11=0.69
PABN (Huang al., 2019)
Conv-4
6..71++0.3
76.90++0.21
Baseline++ (Chen et al., 2019)
6..53+0.83
79.34+-0.61
DN4 (Li et al.. 2019)
53.1540.84
81.90=0.60
FTA (Ours)
65.11
87.92
Baseline++ (Chen et al., 2019)
69.55+0.89
85.17
SubspaceNet (Devos Grossglauser, 2019) ResNet-10
72.92:10.00
87.45
FTA (Ours)
73.1910.62
92.8911022
Table Identical domain and cross domain results for 5-way 5-shot classification using the ResNet- 10 backbone. The dataset names (mini-lmageNer and CUB-200-201 are abbreviated to mini and CUB respectively for brevity.
Identical Domain
Cross Domain
Method
mini-ImageeNet
CUB-200-2011
mini -> CUB
CUB mini
MatchingNet
69.14:10.69
83.7510060
52.59:10.71
48.95:10.77
ProtoNet
73.77=00.64
85.70=0.52
59.22:0.74
53.58=00.33
RelationNet
69.97=10.68
82.67=0.61
54.36:10.71
45.2710.66
SubspaceNet
74.03++0.68
87.45+00.48
62.71+0071
56.66+0068
FTA (Ours)
86.44-10.36
92.89
85.38
58.35
domain scenario, swap the training sets of the two datasets. Specifically, in the mini- MungeNet->CUB--000002 setting, the training set is comprised the 64 base classes from mini- ImageNet, while the validation and test data come from 200-2011--miii Imaie-ee setting, we use the 100-class base data CUB-200-201 for general feature learning and evaluate its performance under the same setting as Devos & Grossglauser (2019) in Table 3. Compared with existing meth- ods, our approach achieves the highest accuracy in all settings. Notably, improve SubspaceNet by 22.67% in the mini- ImageNet--CCU-200 200-2011 setting. The strong cross domain performance mainly attributed the task-adaptive transformation g of our FTA. If it is disabled. the accuracy drops drastically from 85.38% to 67.22%. Another observation is that the performance degrades more in CUB-200-2011 1-mini --mini-ImageNNer than mini-ImageNet--CCUB -C---CCUUB--0001111 200-2011. This is because mini-ImageNet contains more abundant classes than CUB-200-2011 and the network can learn more general and discriminative feature.
Similarly, in the CUB-
the mini-ImageeNt dataset. We report the results
DISCUSSION
Ablation Study of FTA. As shown in Table 4, compared with the Softmax classifier, the proposed Metric-Softmax classifier improves the accuracy by 4.77% and 3.41% in the 5-way 1-shot and 5- way 5-shot settings respectively. This improvement is attributed elimination of the discrepancy between training and inference brought by our Metric-Softmax classifier. By incorporating the task- adaptive transformation, the performance further boosted, especially the 5-shot setting.
TAT ys. Fine-tuning. To leverage the cue from the supported set, an alternative method to fine-tune re-initialized Metrie-Softmax classifier similar to Chen et al. (2019). Figure 2 shows comparison of the proposed task task-adaptive transformation to the alternative fine-tuning scheme. In terms of convergence speed, TAT takes only 5 epochs get appreciable performance. However,
7

---

Under review as a conference paper at ICLR 2020
Table 4: Ablation study of FTA on the mini-ImageNer dataset using the ResNet-11 backbone.
Method
Metric-Softmax
TAT
5-way 1-shot
5-way 5-shot
Softmax Classifier
X
X
08.62-10.56
69.99-10.57
Metric-Softmax Classifier
53.39
73.40=+0.47
FTA
58.03=0.58
80.73=+0.44
S-way
S-wwy S-thot performance
30000
AUCHNC
204
PHA
11.22%
3037%
D000
801
Foetusiin
Tim turing
TAT
15
3
19
15
Fine spochs
Rine tuning cpachs
Figure 2: Performance comparison of TAT to direct fine-tuning on the mini- Muni-ImageNet dataset using the ResNet- 12 backbone. Accuracies on the test set different training epochs are plotted.
(8)
(b)
Figure 3: Schematic illustration of different feature transformations by TAT in the 5-way 1-shot (a) and 5-way 5-shot (b) settings. Circles of different colors represent the features of different classes. In (b) the black circles indicate the centroid of each class.
direct fine-tuning takes over 40 epochs to converge for the 5-way 5-shot setting, For the 5-way 1- shot setting, barely improves the accuracy. terms of final performance, TAT far more superior over direct fine-tuning.
Why does FTA improve the 5-shot setting more than 1-shot setting? As shown in Section 4.3, FTA outperforms existing methods more significantly in the 5-shot setting than 1-shot setting. can be explained by the mechanism the task-adaptive transformation. In the 1-shot setting, only one image is available in each class. Then Equation degenerates to W., = g(h(..))) For image with label y,v 21, 3,4,5),, the numerator of Equation 10 becomes a constant: exp(-al Ig ( - ((((),))))))))
1. Now minimizing the cross-entropy means minimizing the denominator of Equation 10, which is equivalent maximizing the total distance of this feature (hh,, As illustrated in Figure 3(a), task-adaptive transformation only enlarges the distance among different classes in the 1-shot setting. However, for the 5-shot setting (see Figure 3(b)), the transformation not only enlarges inter-class distance but also shrinks the intra-class distance, which helps to improve the classifier.
to others: Σ3 (l)(,,,)))
8

---

Under review as a conference paper at ICLR 2020
6 CONCLUSIONS
In this paper, we propose the Metric-Softmax loss which can adequately explore the feature similar- ities in the whole base dataset with the premise of keeping the classifier consistent between training and inference. In addition, we take full advantage of the few labeled novel data by introducing a pa- rameterized transformation adapt the learned general feature to each few-shot classification task. Our approach is particularly helpful for the 5-shot setting. We leave further improvement for the 1-shot setting as our future work.
REFERENCES
Qi Cai, Yingwei Pan, Ting Yao, Chenggang Yan, and Tao Mei. Memory matching networks for one-shot image recognition. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4080-4088, 2018.
Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. closer look at few-shot classificatton. In ICLR 2019 7th International Conference on Leurning Repre- sentations, 2019.
Jun Deng, Wenjie Dong, Richard Socher, Li-Jia Li, Kehui Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pp.
2009.
Arnout Devos and Matthias Grossglauser. Subspace networks for few-shot classification. arXiv preprint 1905: 13613, 2019.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. In IC Proceedings of the 34th International Conference on Machine Learning Volume 70, pp. 112--1135, 2017.
Ross B. Girshick. Fast r-cnn. In 2015 IEEE International Conference on Computer Vision (ICCV), pp. 1440-1448, 2015.
Nathan Hilliard, Lawrence Phillips, Scott Howland, Artëm Yankov, Courtney D. Corley, and Nathan O. Hodas. Few-shot learning with metric-agnostic conditional embeddings. arXiv preprint OrXiv:/802.044 2018.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8): 1735-1780, 1997.
Huaxi Huang, Junjie Zhang, Jian Zhang, Qiang Wu, and Jingsong Xu. Compare more nuanced: Pair- wise alignment bilinear network for few-shot fine-grained learning. In 2019 IEEE International Conference on Multimedia and Expo (ICME), pp. 91-96, 2019.
Diederik Kingma and Jimmy Lei Ba. Adam: method for stochastic optimization. In ICLR 2015 International Conference on Learning Representations 2015, 2015.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convo- lutional neural networks. neural information processing systems, 19975):0077-115, 2012.
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 10657-10665, 2019.
Wenbin Li, Lei Wang. Jinglin Xu, Jing Huo, Yang Gao, and Jiebo Luo. Revisiting local descriptor based image-to-class measure for few-shot learning. arXiv preprint arXiv:1903.12299 2019.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In 2015 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 3431-3440, 2015.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta- learner. In ICLR 2018: International Conference Learning Representations 2018, 2018.

---

Under review as a conference paper at ICLR 2020
Tsendsuren Munkhdalai, Xingdi Yuan, Soroush Mehri, and Adam Trischler. Rapid adaptation with conditionally shifted neurons. In JCML 2018: Thirty-fifth International Conference on Machine Learning. pp. 3661-3670, 2018.
Boris N. Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. In NIPS 2018: The 32nd Annual Conference on Neural Information Processing Systems. pp. 721-731, 2018.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop, 2017.
Siyuan Qiao, Chenxi Liu, Wei Shen, and Alan L. Yuille. Few-shot image recognition by predicting parameters from activations. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7229-7238, 2018.
Sachin Ravi and Hugo Larochelle. Optimization as model for few-shot learning. In ICLR 2017 International Conference on Learning Representations 2017, 2017.
Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Few-shot learning with embedded class models and shot-free meta training. In The IEEE International Conference on Computer Vision (ICCV), October 2019.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy P. Lillicrap- Meta-leaming with memory-augmented neural networks. In ICML'16 Proceedings of the 33rd International Conference on International Conference on Machine Learning Volume 48, pp. 1842-1850, 2016.
Victor Garcia Satorras and Joan Bruna Estrach. Few-shot learning with graph neural networks. In ICLR 2018: : International Conference on Learning Representations 2018, 2018.
Chenyang Si, Wentao Chen, Wei Wang, Liang Wang, and Tieniu Tan. Progressive cluster purifica- tion for transductive sho-shot learning. arXiv preprint arXiv: 0084773477 2019.
Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning, In Advances in Neural Information Processing Systems, pp. 4077-4087, 2017
Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. Meta-transfer learning for few-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 403-412, 2019.
Flood Sung, Yongxin Yang. Li Zhang, Tao Xiang, Philip H.S. Torr, and Timothy M. Hospedales. Learning compare: Relation network for few-shot learning. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1209-1208, 2018
Oriol Vinyals, Charles Blundell, Timothy P. Lillicrap. Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In NIPS Proceedings of the 30th International Conference on Neural Information Processing Systems, pp. 3637-3645, 2016.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge J. Belongie. The caltech- uesd birds-200-2011 dataset. Advances in Water Resources, 2011.
Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. A discriminative feature learning approach for deep face recognition. In European Conference on Computer Vision, pp. 499-515, 2016.
Ruixiang Zhang, Tong Che, Zoubin Ghahramani, Yoshua Bengio, and Yangqiu Song. Metagan: An adversarial approach to few-shot learning. In NIPS 2018: The 32nd Annual Conference on Neural Information Processing Systems, pp. 2371-2380 2018.
Xuan Zhang, Hao Luo, Xing Fan, Weilai Xiang, Yixiao Sun, Qiqi Xiao, Wei Jiang, Chi Zhang, and Jian Sun. Alignedreid: Surpassing human-level performance in person re-identification. arXiv preprint arxiv: 1711.001844 2017.
10


Reviews:
[{"review_id": "r1ewpj31cB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n\n# Summary\n\nThis paper deals with few-shot learning from a metric-learning perspective. The authors propose replacing the softmax loss, i.e. softmax + cross-entropy loss, with a so-called \"metric-softmax\" loss which imitates a Guassian kernel RBF over class templates/weights. This loss is used in both stages of training on base and on novel classes and the authors argue that it helps learning more discriminative feature while preserving consistency between train and test time.\nSecondly the authors advance a task-adaptive transformation for stage 2 that maps the features from the previously learned feature extractor to a space which is easier to learn. The contributions are evaluated on the standard mini-ImageNet benchmark and on CUB-200-2011 individually and in domain shift mode.\n\n# Rating\nAlthough some of the results in the paper might look impressive, my rating for this work is reject for the following reasons (which will be detailed below):\n1) the main contribution, metric-softmax loss, is not novel. It has been used and described in multiple works in the past 1-2 years.\n2) a part of the evaluations and comparisons do not follow the usual protocol and are not fair\n3) the second contribution, Fast Task Adaptation (FTA), is not well described and it's unclear in what it actually consists, how does it work and how was it trained exactly.\n\n# Strong points\n- This paper deals with a highly interesting and relevant topic for ICLR.\n\n# Weak points\n\n## Contributions\n- This work ignores a large body of research in few-shot learning and metric learning aiming to improve the efficiency per training sample and feature discrimination. \nThe proposed loss can be traced back to Goldberger et al. [i] in  NCA (Neighborhood Component Analysis). Prototypical Networks are derived from this work and hence similar with the metric-softmax loss. \nQi et al. [ii] point out that when h and W are l2-normalized (using the notations from this submission, eq. 7) maximizing their inner-product or cosine-similarity is equivalent to the minimization of the squared Euclidean distance between them. This leads to the loss from [ii], [iii], also known as Cosine Classifier and which also is accompanied by a scaling factor or temperature as here. \nOther related works on improving softmax and selecting the representative weights for a class include: center loss [iv], ring loss[v], L-GM loss[vi].\nIn this light, the metric-softmax is actually not novel and can be found in several other contributions from last year.\n\n- In my opinion, it is difficult from the paper to understand what is actually the fast adatpation module. The authors describe $g$ as \"simply a zero-offset affine transformation\" $g(h)= M^T h$. In the implementation details we do not find out more about this module and we don't have more insights on what is it doing inside, other than a toy hand-drawn example in Figure 3. I find it difficult to assess.\n\n## Experiments\n- The authors evaluate 3 backbone architectures, Conv-4, ResNet-10 and ResNet-12. For the former they use 84 x 84 images, while for the latter they use 224 x 224 images.  The larger images are not standard in the few-shot ImageNet evaluation protocol. Data augmentation (jittering, flipping, etc.) is used here, while in most works it is not. Chen et al. are the first ones to introduce larger images and data augmentation and acknowledge that the large scores are due to this. \nTesting out new configuration is not a problem as long as the baselines are evaluated in the same conditions. However, in this case they are not and this is not visible in the captions of the tables and descriptions in the paper. Training a network with data augmented images and/or higher resolution images and comparing to baselines without data augmentation and images with 7 times less pixels, for sure does not allow seeing the true impact of the proposed method. I would advise to either evaluate in the usual mini-ImageNet settings, either implement a few representative and easy to train baselines, e.g. ProtoNets, Cosine Classifier[iii] in the same conditions as here and compare against. This should provide a better idea on the effectiveness of the proposed methods. \n\n\n## Other comments\n- the scores for baseline methods are seemingly taken from the paper of Chen et al. who trained them themselves. This should be mentioned in the paper and in the caption\n\n# Suggestions for improving the paper:\n1) Review the experimental section and make sure at least some of the baselines are trained in similar conditions as the proposed method or alternatively evaluate the proposed methods in standard mini-ImageNet settings\n\n2) Provide additional insights, experiments and implementation details for FTA to make it easier to understand, there are some examples in the references below.\n\n\n\n# References \n[i] J. Goldberger et al., Neighbourhood components analysis, NIPS 2005\n[ii] H. Qi et al., Low-Shot Learning with Imprinted Weights, CVPR 2018\n[iii] S. Gidaris and N. Komodakis, Dynamic Few-Shot Visual Learning without Forgetting, CVPR 2018\n[iv] W. Wen et al., A Discriminative Feature Learning Approach\nfor Deep Face Recognition, ECCV 2016\n[v] Y. Zeng et al., Ring loss: Convex Feature Normalization for Face Recognition, CVPR 2018\n[wi] W. Wan et al., Rethinking Feature Distribution for Loss Functions in Image Classification, CVPR 2018", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HylyzIxXtB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Authors propose a new method for adaptation in a few-shot learning setting. Their method comprises two different steps; first they propose a new metric-softmax loss, which aims at improving the transferability of features pre-trained on base data to novel data. They achieve this via redefining the probability score calculating function, which in practice means they replace the exponent term found in the softmax loss with a Gaussian kernel-based radial basis function. This first step improves the feature learning process at large scale but does not solve the problems found when trying to fit arbitrary novel classes. At this point comes step two, where a fast task adaptation process is proposed, i.e. Task-Adaptive Transformation based on affine transformation. This method converges fast and is learnt from the support set, vs step 1 which is trained on the training/base set. Post-training the affine transformation is applied to both support and query image sets.\nAuthors have been fairly detailed in their experimental processes and used different backbone models, routinely found in papers focusing on similar issue. They compare against 4 other state-of-the art methods showing a significant improvement across all of them. They also demonstrate the superiority of the metric-softmax classifier vs softmax-classifier and finally the overall superiority of the whole method proposed.\nSimilar results are obtained on domain sift settings as well as when comparing the step 2 of this method with a fine-tuning approach. \nThe proposed method makes a good contribution towards reducing the problem of overfitting when very few examples of a new problem are available.\nI have read the rebuttal and think the paper could be a good addition to the programme.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HylaQFF8cH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper develops a new few-shot image classification algorithm. It has two main contributions. The first one is to use a metric-softmax loss used to train on the meta-training dataset without episodic updates. The second is that the features learnt thereby are further modified using a linear transformation to fit the few-shot training data and the metric soft-max loss is again used for classifying the query samples. The authors provide experimental results for 5-way-1-shot and 5-way-5-shot testing on mini-Imagenet and CUB-200-2011 datasets.\n\nI think this paper is below the acceptance threshold. The reasons are:\n\n1. The contributions of this paper are marginal: both learning centroids for each meta-training class and projecting the few-shot features have been used before in published work (https://arxiv.org/abs/1905.04398). The empirical results are weaker than existing work (see for instance, https://arxiv.org/abs/1904.03758, https://arxiv.org/abs/1909.02729 etc.); also see #3 below.\n2. The authors should provide experimental results on other few-shot learning datasets like tiered-Imagenet.\n3. The image-size used here for Resnet-12 is 224x224, the authors should report results using 84x84 image size so that one can compare against existing literature fairly. Are the results for Resnet-12 so good because of the larger image size?\n4. The training procedure is task-agnostic, why do you train a different model for the 1-shot and the 5-shot case?\n\nI will consider increasing my score if some of the concerns above are addressed. I am listing some more comments below which I would like the authors to consider.\n\n\n1. Contributions: “consistency between training and inference”, do you instead mean consistency between meta-training and few-shot training? There are no weight updates at inference time.\n2. How essential is the metric-softmax loss? Training on the meta-training dataset without episodic updates has also been done in https://arxiv.org/abs/1909.02729. These authors seem to use standard soft-max training and perform standard fine-tuning, they report empirical performance that is significantly better than that in Table 4 and Figure 2. I am very skeptical as to why the accuracy of fine-tuning is only 21% in Figure 2.\n3. Section 3.2 does not motivate or explain the metric-softmax loss. Why should one have the network learn the centroids of the meta-training dataset? Can you draw a TSNE of the centroids learnt during meta-training? The features of the support samples (or their transformations) can be the centroids of the few-shot classes in the prototypical loss so inference phase does not need these centroids.\n4. I am not sure whether the matrix M is changed non-trivially during few-shot training. The weights W are already initialized to be the centroid of the features (eqn. 9). So the metric-softmax loss in eqn. 10 is expected to be small for the support samples after initialization. Why should the additional expression power afforded by M matter? There is no incentive for the network to change the matrix M. Can you show results on how much M changes from the identity?\n5. I believe the reported numerical results for LEO (Rusu et al. 2019) are for a WRN-28-10 architecture, not ResNet-12.\n6. The accuracy using Resnet-12 seem extremely high. I believe this is because the results reported in the literature, e.g., https://arxiv.org/abs/1904.03758, use images of size 84x84, not 224x224 as the authors here have used. Can you report results using 84x84 sized images?\n7. I don’t understand the explanation at the end of Section 5. Since the prototypical loss is being used to classify the query datum, it should not matter whether the cluster is shrunk in the 5-shot case, or whether simply the distances between the clusters are increased as in the 1-shot case.\n8. Table 1 is quite incomplete, the authors should mention other existing few-shot classification results are similar to the performance of this paper, e.g., https://arxiv.org/abs/1805.10123, among the ones listed above.\n9. The entries in Table 1 and 2 are not made bold appropriately. All entries with overlapping standard error should be bold.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper develops a new few-shot image classification algorithm by using a metric-softmax loss for non-episodic training and a linear transformation to modify the model towards few-shot training data for task-agnostic adaptation.

Reviewers acknowledge that some of the results in the paper are impressive especially on domain sift settings as well as with a fine-tuning approach. However, they also raise very detailed and constructive concerns on the 1) lack of novelty, 2) improper claim of contribution, 3) inconsistent evaluation protocol with de facto ones in existing work. Author's rebuttal failed to convince the reviewers in regards to a majority of the critiques.

Hence I recommend rejection.
Decision: Reject