Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: <Paper summary>
The authors argue that the popular benchmark datasets, Omniglot and miniImageNet, are too simple to evaluate supervised few-shot classification methods due to their insufficient variety of class semantics. To validate this, the authors proposed clustering-based meta-learning method, called Centroid Network. Although it does not utilize supervision information during meta-evaluation, it can achieve high accuracies on Omniglot and miniImageNet. The authors also proposed a new metric to quantify the difficulty of meta-learning for few-shot classification.

<Review summary>
Although the main claim of this paper seems correct, it is not sufficiently supported by theory or experiments. My score is actually on the border line, but I currently vote for ``weak reject, because some points in the paper are ambiguous yet. Given clarifications in an author response, I would be willing to increase the score.

<Details>
* Strength
 + The paper is well-organized. Especially, the examples shown in the introduction greatly help understanding of what the authors argue in this paper.
 + A novel study on quantifying the difficulty of meta-learning.
 + The proposed CentroidNet performs well in the experiments.
 
* Weakness and concerns
 - Does CentroidNet really work without labels during ``meta-validation"? As far as I understand, ground truth clusters of the support set defined by the labels are required to compute the accuracies. Therefore, the labels seem to be required to validate the performance of the model. I think it should be ``meta-test."
 - The authors state ``The most intuitive way to train ..., we did not have much success with this approach" in 5.3, but it is counter-intuitive. If the class semantics are similar among episodes, ``the most intuitive way" should work, because it can learn the common semantics via meta-training. Further discussion about why it does not work is required.
 - The high performance of CentroidNet does not support the claim on the insufficient variety of the class semantics. According to ablation study, adopting Sinkhorn K-means is the most important factor to improve the performance. It means that adopting weighted average like in [R1] can also improve the performance of ProtoNet, which results in substantial difference in the performance between ProtoNet and CentroidNet that can deny the claim. 
 - The definition of CSCC is not convincing. First, I could not get the meaning of ``unsupervised Bayes accuracy" (supervised Bayes accuracy means 1 - Bayes error rate, right?). Second, CSCC seems to mainly quantify the importance of the supervision information during meta-learning, which is not directly related to the difficulty of few-shot learning problem. Intuitively, difficult few-shot learning problems should lead to lower supervised Bayes accuracy, which does not necessarily decrease CSCC. Third, what we can induce via comparing CSCC is not clarified in theory. The discussion in 6.3 is too subjective and specific for the case of training with ILSVRC/all datasets.
 - This paper lacks citing some closely related works [R1, R2].

[R1] ``Infinite Mixture Prototypes for Few-Shot Learning," ICML2019
[R2] ``A Closer Look at Few-shot Classification," ICLR2019

* Minor concerns that do not have an impact on the score
 - Another arXiv paper related to this work: ``Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML"


review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The paper is concerned with few-shot classification, both its benchmarks and method used to tackle it. The scope of the few-shot classification problem can be set relatively widely, depending on what data is available at what stage. In general few-shot classification is an important ability of intelligent systems and arguably an area in which biological systems outperform current AI systems the most.

The paper makes a number of contributions. (1) It suggests an approach to do a specific type of clustering and compares it favorably to the existing literature. In a specific sense the approach does not use supervised labels (“without labels at meta-evaluation time”). (2) It applies that approach to currently existing datasets and achieves “surprisingly high accuracies” in that setting, with the implication that this shows a weakness in these datasets when used for benchmarking (“too easy”). (3) It further suggests a metric, dubbed “class semantics consistency criterion”, that aims to quantify this shortcoming of current benchmarks on these datasets. (4) It assesses a specific meta-dataset using that metric, confirming it is harder in this sense, at least in specific settings.

My assessment of the paper is mildly negative; however this is an assessment with low confidence given that I am no expert on few-shot classification or related areas.

While the authors first example (the “Mongolian” alphabet of the Omniglot dataset and geometric shapes falling into different categories) illustrates the problem space well and is indeed quite intuitive, the same cannot be said about either the specific setting they consider nor the metric they propose. It’s not immediately clear that the other approaches from the literature they compare their method to were conceived for the setting considered here, or indeed optimized for it. The authors do show good accuracy on clustering Omniglot characters without using labels and thus indeed demonstrate a high amount of class semantics consistency for that dataset. The results on miniImageNet are less clear-cut, and the results of the evaluation of the meta-dataset appear to depend on the specific setting considered. This makes it unclear to what extent the proposed metric is general and predictive. To their credit, the authors state that in future work they are looking to make their metric “more interpretable and less dependent on the backbone architectures”.

I believe the paper might benefit from being given additional attention. A streamlined and more accessible version might well be an important contribution in the future.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper introduces a new method for learning to cluster without labels at meta-evaluation time and show that this method does as well as supervised methods on benchmarks with consistent class semantics. The authors propose a new metric for measuring the simplicity of a few-shot learning benchmark and demonstrate that it is possible to achieve high performance on Omniglot and miniImageNet with their unsupervised method, resulting in a high value of this criterion, whereas the Meta-Dataset is much more difficult.

The paper is well written and generally very clear. I appreciate that the authors have highlighted the limitations of both their clustering method (that it requires more assumptions than CCNs) and their benchmark. The centroid method itself seems to draw heavily on pre-existing work, but uses a new similarity metric that improves performance beyond the current state-of-the-art on few-shot clustering tasks.

The authors acknowledge that the approximate CSCC metric they define is not consistent across architectures and hyperparameters. It is also a fairly simple metric, but nonetheless represents a novel contribution.

Overall I feel that the paper introduces a well-defined problem and makes a step toward quantifying and resolving it. The experiments do a thorough job supporting the arguments of the authors.


I have only minor issues that could help with the clarity of this paper: 

I wasn’t sure what was meant by “relabeling” the query set predictions in the text below Figure 2. 

I would have appreciated some discussion as to why Sinkhorn distance might be expected to improve performance .
review_title: Official Blind Review #2

Meta-review: The paper is interested in assessing the difficulty of popular few-shot classification benchmarks (Omniglot and miniImageNet). A clustering-based meta-learning method is proposed (called Centroid Network), on which a metric is built (gap between the performance of Prototypical Networks and Centroid Networks). As noted by several reviewers, the proposed metric (critical for the paper) is however not motivated enough, nor convincing enough - after discussion, the logic in the metric reasoning seems to remain flawed.

Decision: Reject