Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper develops a new few-shot image classification algorithm. It has two main contributions. The first one is to use a metric-softmax loss used to train on the meta-training dataset without episodic updates. The second is that the features learnt thereby are further modified using a linear transformation to fit the few-shot training data and the metric soft-max loss is again used for classifying the query samples. The authors provide experimental results for 5-way-1-shot and 5-way-5-shot testing on mini-Imagenet and CUB-200-2011 datasets.

I think this paper is below the acceptance threshold. The reasons are:

1. The contributions of this paper are marginal: both learning centroids for each meta-training class and projecting the few-shot features have been used before in published work (https://arxiv.org/abs/1905.04398). The empirical results are weaker than existing work (see for instance, https://arxiv.org/abs/1904.03758, https://arxiv.org/abs/1909.02729 etc.); also see #3 below.
2. The authors should provide experimental results on other few-shot learning datasets like tiered-Imagenet.
3. The image-size used here for Resnet-12 is 224x224, the authors should report results using 84x84 image size so that one can compare against existing literature fairly. Are the results for Resnet-12 so good because of the larger image size?
4. The training procedure is task-agnostic, why do you train a different model for the 1-shot and the 5-shot case?

I will consider increasing my score if some of the concerns above are addressed. I am listing some more comments below which I would like the authors to consider.


1. Contributions: “consistency between training and inference”, do you instead mean consistency between meta-training and few-shot training? There are no weight updates at inference time.
2. How essential is the metric-softmax loss? Training on the meta-training dataset without episodic updates has also been done in https://arxiv.org/abs/1909.02729. These authors seem to use standard soft-max training and perform standard fine-tuning, they report empirical performance that is significantly better than that in Table 4 and Figure 2. I am very skeptical as to why the accuracy of fine-tuning is only 21% in Figure 2.
3. Section 3.2 does not motivate or explain the metric-softmax loss. Why should one have the network learn the centroids of the meta-training dataset? Can you draw a TSNE of the centroids learnt during meta-training? The features of the support samples (or their transformations) can be the centroids of the few-shot classes in the prototypical loss so inference phase does not need these centroids.
4. I am not sure whether the matrix M is changed non-trivially during few-shot training. The weights W are already initialized to be the centroid of the features (eqn. 9). So the metric-softmax loss in eqn. 10 is expected to be small for the support samples after initialization. Why should the additional expression power afforded by M matter? There is no incentive for the network to change the matrix M. Can you show results on how much M changes from the identity?
5. I believe the reported numerical results for LEO (Rusu et al. 2019) are for a WRN-28-10 architecture, not ResNet-12.
6. The accuracy using Resnet-12 seem extremely high. I believe this is because the results reported in the literature, e.g., https://arxiv.org/abs/1904.03758, use images of size 84x84, not 224x224 as the authors here have used. Can you report results using 84x84 sized images?
7. I don’t understand the explanation at the end of Section 5. Since the prototypical loss is being used to classify the query datum, it should not matter whether the cluster is shrunk in the 5-shot case, or whether simply the distances between the clusters are increased as in the 1-shot case.
8. Table 1 is quite incomplete, the authors should mention other existing few-shot classification results are similar to the performance of this paper, e.g., https://arxiv.org/abs/1805.10123, among the ones listed above.
9. The entries in Table 1 and 2 are not made bold appropriately. All entries with overlapping standard error should be bold.
review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: 

# Summary

This paper deals with few-shot learning from a metric-learning perspective. The authors propose replacing the softmax loss, i.e. softmax + cross-entropy loss, with a so-called "metric-softmax" loss which imitates a Guassian kernel RBF over class templates/weights. This loss is used in both stages of training on base and on novel classes and the authors argue that it helps learning more discriminative feature while preserving consistency between train and test time.
Secondly the authors advance a task-adaptive transformation for stage 2 that maps the features from the previously learned feature extractor to a space which is easier to learn. The contributions are evaluated on the standard mini-ImageNet benchmark and on CUB-200-2011 individually and in domain shift mode.

# Rating
Although some of the results in the paper might look impressive, my rating for this work is reject for the following reasons (which will be detailed below):
1) the main contribution, metric-softmax loss, is not novel. It has been used and described in multiple works in the past 1-2 years.
2) a part of the evaluations and comparisons do not follow the usual protocol and are not fair
3) the second contribution, Fast Task Adaptation (FTA), is not well described and it's unclear in what it actually consists, how does it work and how was it trained exactly.

# Strong points
- This paper deals with a highly interesting and relevant topic for ICLR.

# Weak points

## Contributions
- This work ignores a large body of research in few-shot learning and metric learning aiming to improve the efficiency per training sample and feature discrimination. 
The proposed loss can be traced back to Goldberger et al. [i] in  NCA (Neighborhood Component Analysis). Prototypical Networks are derived from this work and hence similar with the metric-softmax loss. 
Qi et al. [ii] point out that when h and W are l2-normalized (using the notations from this submission, eq. 7) maximizing their inner-product or cosine-similarity is equivalent to the minimization of the squared Euclidean distance between them. This leads to the loss from [ii], [iii], also known as Cosine Classifier and which also is accompanied by a scaling factor or temperature as here. 
Other related works on improving softmax and selecting the representative weights for a class include: center loss [iv], ring loss[v], L-GM loss[vi].
In this light, the metric-softmax is actually not novel and can be found in several other contributions from last year.

- In my opinion, it is difficult from the paper to understand what is actually the fast adatpation module. The authors describe $g$ as "simply a zero-offset affine transformation" $g(h)= M^T h$. In the implementation details we do not find out more about this module and we don't have more insights on what is it doing inside, other than a toy hand-drawn example in Figure 3. I find it difficult to assess.

## Experiments
- The authors evaluate 3 backbone architectures, Conv-4, ResNet-10 and ResNet-12. For the former they use 84 x 84 images, while for the latter they use 224 x 224 images.  The larger images are not standard in the few-shot ImageNet evaluation protocol. Data augmentation (jittering, flipping, etc.) is used here, while in most works it is not. Chen et al. are the first ones to introduce larger images and data augmentation and acknowledge that the large scores are due to this. 
Testing out new configuration is not a problem as long as the baselines are evaluated in the same conditions. However, in this case they are not and this is not visible in the captions of the tables and descriptions in the paper. Training a network with data augmented images and/or higher resolution images and comparing to baselines without data augmentation and images with 7 times less pixels, for sure does not allow seeing the true impact of the proposed method. I would advise to either evaluate in the usual mini-ImageNet settings, either implement a few representative and easy to train baselines, e.g. ProtoNets, Cosine Classifier[iii] in the same conditions as here and compare against. This should provide a better idea on the effectiveness of the proposed methods. 


## Other comments
- the scores for baseline methods are seemingly taken from the paper of Chen et al. who trained them themselves. This should be mentioned in the paper and in the caption

# Suggestions for improving the paper:
1) Review the experimental section and make sure at least some of the baselines are trained in similar conditions as the proposed method or alternatively evaluate the proposed methods in standard mini-ImageNet settings

2) Provide additional insights, experiments and implementation details for FTA to make it easier to understand, there are some examples in the references below.



# References 
[i] J. Goldberger et al., Neighbourhood components analysis, NIPS 2005
[ii] H. Qi et al., Low-Shot Learning with Imprinted Weights, CVPR 2018
[iii] S. Gidaris and N. Komodakis, Dynamic Few-Shot Visual Learning without Forgetting, CVPR 2018
[iv] W. Wen et al., A Discriminative Feature Learning Approach
for Deep Face Recognition, ECCV 2016
[v] Y. Zeng et al., Ring loss: Convex Feature Normalization for Face Recognition, CVPR 2018
[wi] W. Wan et al., Rethinking Feature Distribution for Loss Functions in Image Classification, CVPR 2018
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: Authors propose a new method for adaptation in a few-shot learning setting. Their method comprises two different steps; first they propose a new metric-softmax loss, which aims at improving the transferability of features pre-trained on base data to novel data. They achieve this via redefining the probability score calculating function, which in practice means they replace the exponent term found in the softmax loss with a Gaussian kernel-based radial basis function. This first step improves the feature learning process at large scale but does not solve the problems found when trying to fit arbitrary novel classes. At this point comes step two, where a fast task adaptation process is proposed, i.e. Task-Adaptive Transformation based on affine transformation. This method converges fast and is learnt from the support set, vs step 1 which is trained on the training/base set. Post-training the affine transformation is applied to both support and query image sets.
Authors have been fairly detailed in their experimental processes and used different backbone models, routinely found in papers focusing on similar issue. They compare against 4 other state-of-the art methods showing a significant improvement across all of them. They also demonstrate the superiority of the metric-softmax classifier vs softmax-classifier and finally the overall superiority of the whole method proposed.
Similar results are obtained on domain sift settings as well as when comparing the step 2 of this method with a fine-tuning approach. 
The proposed method makes a good contribution towards reducing the problem of overfitting when very few examples of a new problem are available.
I have read the rebuttal and think the paper could be a good addition to the programme.
review_title: Official Blind Review #1

Meta-review: This paper develops a new few-shot image classification algorithm by using a metric-softmax loss for non-episodic training and a linear transformation to modify the model towards few-shot training data for task-agnostic adaptation.

Reviewers acknowledge that some of the results in the paper are impressive especially on domain sift settings as well as with a fine-tuning approach. However, they also raise very detailed and constructive concerns on the 1) lack of novelty, 2) improper claim of contribution, 3) inconsistent evaluation protocol with de facto ones in existing work. Author's rebuttal failed to convince the reviewers in regards to a majority of the critiques.

Hence I recommend rejection.
Decision: Reject