Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper introduces a transductive learning baseline for few-shot image classification. The proposed approach includes a standard cross-entropy loss on the labeled support samples and a Shannon entropy loss on the unlabeled query samples. Despite its simplicity, the experimental results show that it can consistently outperform the state-of-the-art on four public few-shot datasets. In addition, they introduce a large-scale few-shot benchmark with 21K classes of ImageNet21K. Finally, they point out that accuracies from different episodes have high variance and develop another few-shot performance metric based on the hardness of each episode.

Positive comments:
1. The proposed transductive loss that minimizes entropy of query samples is novel in few-shot learning. Given limited labeled samples, finetuning with unlabeled query samples via proper loss is a good idea to tackle few-shot learning. 
2. The evaluation is thorough. A significant number of few-shot methods are compared on 4 exisiting few-shot benchmarks. An additional large-scale benchmark is also introduced to facilitate  the few-shot learning research. 
3. A novel evaluation metric is proposed to evaluate few-shot learning methods under different difficulties level. Although I am convinced by the importance of such metric, it is interesting to supplement the averaged accuracy because it tells how the methods work under easy and difficult classes. 

Negative comments:
1. The folloing important reference of the Shannon entropy on unlabeled data is missing. In fact, I suggest the authors to extend Section 3.2 a bit more because this is the main technic contribution. 
Semi-supervised Learningby Entropy Minimization.  Grandvalet et al. NIPS 2015
2. I am not convinced by the necessity of the proposed hardness metric. The main argument of this paper is that accuracies over episodes have high variance. But isn't it expected that different  episode can include samples with different difficulties, leading to high variance of accuracies? I do not think it is realistic to have one algorithm that achieves similar accuracies on both easy and difficult tasks. The authors also fail to evaluate different methods with the proposed metric and show if this metric makes the ranking of algorithms different. Moreover, I find Figure 3 hard to interpret because there are too much information in it, including different colors, a lot of markers and lines. Why is the range of hardness 1-3 for some datasets and 1-5 for other datasets? I believe writting of Section 4.4 could be further improved.

Overall, I think this paper has significant contributions of proposing a novel few-shot baseline that establishes a new state-of-the-art and would recommend weak accept.


review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The authors propose a fine-tune-based few-shot classification baseline, which has been validated effectively on several datasets, including Mini-Imagenet, Tiered-Imagenet, CIFAR-FS, FC-100, and Imagenet-21k. In addition to the method, the authors also provide concrete experimental setting and new evaluation proposals.

1. The authors propose to use the logits instead of embedding as the main bridge between the pre-trained model and the meta-learning model. Does it mean we represent novel classes based on the properties of the meta-train classes? If so, does this method requires more meta-train classes to enrich the representation ability? How will the method perform when working on few-shot learning problems with a large distribution shift?

2. To make a fair comparison: instead of citing the values in the published papers directly and comparing different methods with different architectures, the authors should also apply the pre-trained model with the famous baselines, such as Matching Network, Prototypical Network, and MAML. Now there exists a very lap gap between the Matching Network values and the newly proposed one. For example, fine-tune the Matching Network on the pre-trained backbone in both train and train+val settings. Therefore, it is more clear to show the improvement of the proposed baseline models. Q/A 2-3 in appendix D do not fully solve this problem.

3. Considering the randomness of the sampled few-shot tasks, the authors can consider evaluating over more episodes (e.g., 10,000 trials) than 1000 in the paper.

4. It's better for the authors to emphasize and differentiate the transductive fine-tune and the inductive counterpart in the paper. 
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper provided a baseline method for few-shot learning. It utilizes a simple but effective approach via a transductive fine-tuning. The experimental results on several benchmarks show the improvements over state-of-the-art approaches. 

It is a comprehensive study of the methods and datasets in this domain. The motivation, experimental details and result analysis are clear to me. Overall, the paper is well written and the author is very transparent to show what they have. 

The only drawback of this paper is it does not provide insight/explanation. How can the simple baseline work sowell? Is this because of some bias from the datasets? I also suggest that the author can try their method on some new dataset, like Meta-Dataset (Triantafillou et al. 2019). 

I agreed with the author that the paper is not novel. However, I think the acceptance of the paper could benefit the community and I encourage the author can try this on some new benchmark. Therefore, I made my recommendation. 
review_title: Official Blind Review #2

Meta-review: This paper introduces a simple baseline for few-shot image classification in the transductive setting, which includes a standard cross-entropy loss on the labeled support samples and a conditional entropy loss on the unlabeled query samples.

Both losses are known in the literature (the seminal work of entropy minimization by Bengio should be cited properly). However, reviewers are positive about this paper, acknowledging the significant contributions of a novel few-shot baseline that establishes a new state-of-the-art on well-known public few-shot datasets as well as on the introduced large-scale benchmark ImageNet21K. The comprehensive study of the methods and datasets in this domain will benefit the research practices in this area.

Therefore, I make an acceptance recommendation.
Decision: Accept (Poster)