Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The authors of this paper construct a new few-shot learning dataset. The whole dataset consists of several data from different sources. The authors test several representative meta-learning models (e.g., matching network, Prototype network, MAML) on this dataset and give the analysis. Furthermore, the authors combine MAML and Prototype network, which achieves the best performance on this new dataset.

Pros:
+ Compared with previous datasets (e.g., miniimagenet, omniglot), the constructed meta-dataset is larger and more realistic, which contains several datasets collected from different sources
+ Several competitive baselines are compared on this dataset under different scenarios (e.g., different number of shot) with reasonable analysis.

Cons:
- I am familiar with meta-learning, however, it is my first time to review a paper whose main contribution is proposing a new benchmark. The proposed dataset may useful in further meta-learning research. However, I do not feel the construction way is quite difficult. The authors only propose several rules to construct the data set (see 3.2). 
- It would be better if the authors can explain more about Proto-MAML. My understanding of Proto-MAML is to apply the prototype on the last layer and keep the other layers.

-------------------------------------------------------------------------------------------------------------------------------------------------
After rebuttal:

After reading the response, I think constructing a new benchmark is important and useful. However, considering the technical contributions of this paper, I finally decide to keep my score.

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper proposed a really interesting direction for few-shot and meta-learning, the concept of a 'meta-dataset', which can be used for more realistic evaluation of algorithms. The main contributions are:

1) A more realistic environment for training and testing few-shot learners. 
2) Experimental evaluation of popular models
3) Analyses of whether different models benefit from more data,
4) A new meta-learner

I think this work is an interesting empirical paper which should be supported by solid experimental results. My concern about this paper in its current form is that the layout/structure of the paper needs to be improved, for example:

Considering putting some of the key results in the appendix section in the main text
Removal of repeating results from the main text by shortening the early sections
review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: The paper presents Meta-Dataset, a benchmark for few-shot classification that combines various image classification data sets, allows the number of classes and examples per class to vary, and considers the relationships between classes. It performs an empirical evaluation of six algorithms from the literature, k-NN, FineTune, Prototypical Networks, Matching Networks, Relation Networks, and MAML. A new approach combining Prototypical Networks and first-order MAML is shown to outperform those algorithms, but there is substantial room for improvement overall.

Many papers in the literature have observed that current algorithms achieve very high performance on existing few-shot classification data sets such as Omniglot, making it difficult to compare them. This paper fills the need for a more challenging data set. This data set is also more realistic in that
1. The training and test data may come from different data sets. In the real world, we often encounter locations, weather conditions, etc. that are unseen during training, and agents must be able to adapt quickly to these new situations.
2. Classes are selected based on their semantic structure. This allows control of the difficulty of the task, as it is easier to adapt to a semantically similar class, and thus enables a more nuanced comparison of few-shot classification algorithms.

Suggestions for clarifying the writing:
1. Section 3 should also discuss choosing the data set to sample from.
2. In figures 1c and 1d, it would be helpful to include standard error regions, like in the rest of figure 1.
3. Maybe the paragraph on Proto-MAML should be moved to section 3, as it is not from previous literature. In addition, steps 2b and 2c in section 3.2 overlap in content, and so it may be clearer to combine them.
review_title: Official Blind Review #2

Meta-review: While the reviewers have some outstanding issues regarding the organization and clarity of the paper, the overall consensus is that the proposed evaluation methods is a useful improvement over current standards for meta-learning.
Decision: Accept (Poster)