Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary

The paper proposes a method to calibrate the underlying distribution of a few samples in the few-shot classification scenario. The idea is to estimate a feature distribution of a few samples of a novel class from base class distributions. The authors assume that every dimension in the feature vector follows a Gaussian distribution. Based on the observation that the mean and variance of the distribution with respect to each class are correlated to the semantic similarity of each class, base class distribution can be transferred to the novel class distribution. After distribution calibration, features can be generated from the calibrated distribution and the generated features are used to train classifiers. SVM and logistic regression classifier are used to verify the approach on the mini-imagenet and CUB datasets.

Pros
-   The idea can be applied to any types of feature extractors or classifiers when the Gaussian distribution assumption holds.
-   The proposed approach shows competitive performance on two benchmarks, mini-imagenet and CUB.
-   Extensive ablation studies verify hyper-parameter settings and their sensitivities.

Cons
- Many hyper-parameters are involved in the approach. ( Turkey’s transformation parameter lambda, top-k base distributions, dispersion parameter alpha, number of generated features)
-   Some hyper-parameter setting should be different depending on the dataset. (dispersion parameter alpha)
-   The paper uses the previous work (Mangla et al. 2020) as a baseline and applies their approach on top of it. While theoretically the approach can be applied to any types of feature extractors, only one baseline improvement is shown in the paper.


Rating

I give marginally below the threshold. The paper shows good performance and also claims the approach can be applied on top of any classifiers or feature extractor. The general applicability and effectiveness are the strength of the paper. However, the proposed approach is only verified on one baseline approach (Mangla et al. 2020). More empirical data would strengthen the claim of the paper.

Questions

The approach outputs calibrated distribution, i.e., a mixture of Gaussian distributions. These distributions can be directly used to classify query samples by calculating the likelihood of the sample on each distribution. How does likelihood classification perform compared to the retrained classifiers (SVM or RL)?
The optimal values for top-k parameter and alpha parameter are different depending on the dataset. Are optimal values for lambda and the number of generated features same or different depending on the dataset? Did the authors use the same lambda and number of generated features for both mini-imagenet and CUB experiments?
The authors only applied the proposed method on one baseline approach. Is the approach effective for more variety of backbone networks and losses? 

Feedback

- Table1 shows that there is a correlation between feature distributions and semantic similarities. It would be interesting to see how the distribution calibration performs on each class depending on the similarity level of top-k base distributions.
- The empirical evidence is not sufficient to claim the approach applies to general network architectures and few-shot learning approaches. Experiments with more baseline approach to show the effectiveness of the idea is recommended.

review_title: Marginally below threshold.

final_score: rating: 7: Good paper, accept
review_content: Summary:

This paper identifies the problem of biased distributions in few-shot learning and proposes to fix it. In few-shot learning, only a few samples per class are available; this makes estimating the class distribution difficult. The paper proposes a distribution calibration algorithm that makes use of the meta-train class distributions to calibrate the few-shot class distributions. Once calibrated, more samples are drawn from this distribution to learn a classifier that generalizes better. This approach does not require additional learnable parameters and can be (potentially) built on-top of any pre-trained feature extractor. Empirical results show that this approach achieves state-of-the-art results on Mini-ImageNet and CUB.

Pros:
1. This paper identifies and tries to tackle an important problem in few-shot learning - estimation of the class distribution. Due to the limited number of samples, this problem is difficult, but important for few-shot learning. The proposed algorithm is simple and effective in tacking this problem.
2. As opposed to other related works, the proposed algorithm does not have any learnable parameters. It makes use of the features obtained for the meta-train and few-shot samples.

Cons:
1. One of the claims of the paper is that the proposed algorithm is pre-trained feature extractor agnostic. However, there are no experiments to validate this claim. Consider adding feature extractors trained in different ways.
2. Some of the popular few-shot datasets were not included in the experimental section, namely Tiered-ImageNet [1] and Meta-Dataset [2]. The former is a larger dataset than the ones used, and the latter provides cross-domain results for the approach. The proposed algorithm assumes that statistics from the meta-train classes would transfer to the few-shot classes, which would be tested more thoroughly in the cross-domain setting.

Clarifications:
1. In the feature space, it is intuitive that the means of similar classes would be correlated. Is there a justification for why this is true for the variances?
2. Tukey's Ladder of Powers transformation is used only on the few-shot samples and not the meta-train samples. Is there a reason for that? If the pre-trained feature extractor is trained using a certain metric (cosine distances for example), I would imagine transforming all the features would be beneficial.
3. The backbone used in the experiments is trained using a supervised and self-supervised loss. What are the results without the self-supervised loss?
4. How many samples are drawn from the calibrated distribution for the numbers in Table 2?

Notes:
1. The performance of few-shot learning algorithms has been traditionally evaluated using the averaged accuracy over multiple tasks, but that is not the only way to do it. Look at [3] for details.

[1] Mengye Ren et al. Meta-Learning for Semi-Supervised Few-Shot Classification.
[2] Eleni Triantafillou et al. Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples.
[3] Guneet S. Dhillon et al. A Baseline for Few-Shot Image Classification.
review_title: Simple and effective method for calibrating the few-shot class distribution

final_score: rating: 7: Good paper, accept
review_content: This paper presents a simple and intuitive data augmentation method for few shot image classification. The proposed method assumes the feature distribution of a single class to be gaussian. Based on this assumption, the proposed method samples features from a gaussian distribution that is constructed using the statistics of the similar base classes. Combined with logistic regression, the proposed method achieves strong performance on two standard few-shot image classification benchmarks. This submission also carries out an ablation study on several design choices, which I really appreciate.

The submission is well-written and clear. The proposed method is novel and can inspire future augmentation-based methods in few-shot image classification.

I have a few more requests/ablations that I am curious to see. 

- Looking at Figure 4, the 1-shot accuracy with only 1 retrieved class is already very strong. Instead of sampling from the "calibrated" distribution, can we simply retrieve examples from the nearest class? Namely, find the nearest class, randomly sample some examples, and use their features to augment the novel classes. This ablation should make clear what additional value the sampling procedure adds.
- In equation (6), how important is it to include the novel class feature into the mean? Can we simply do \mu_prime = \sum_{i \in S_N} \mu_i /  k?
- This method makes an important assumption that the feature distribution is gaussian. How well does this assumption hold in practice? Can the authors provide some analysis of the feature distribution? Ideally both before and after Tukey’s Ladder of Powers transformation. 


review_title: A simple data augmentation for few-shot image classification

Meta-review: This paper proposes a novel and powerful data augmentation strategy for few-shot learning, producing convincing improvements over current approaches. The request by the reviewers to include additional ablations, more backbones, and an additional dataset have been satisfactorily  resolved, with the results remaining strong. The reviewers are all unanimous in their recommendation that the paper be accepted for publication.

Decision: Accept (Oral)