Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This study extensively introduces the distinction between ID and OOD scenarios in a meta-learning framework. This points out that model selection and performance comparison in OOD scenarios have some problems. To resolve this issue, the authors introduce protocols for FSL benchmarks in ID and OOD evaluation.
main_review: This work points out the importance of the distinction between ID and OOD scenarios in a meta-learning framework. The current evaluation protocol, especially in FSL, focuses on an OOD scenario rather than ID. However, I still believe that validating with an OOD scenario makes more sense than the one with an ID setting, because the objective of meta-learning (compared to inductive learning) is to learn a model (or an agent) that is quickly capable of acquiring a new skill in a different environment. 

### Strengths
* This work points out the current FSL benchmark aims to validate meta-learning models in an out-of-distribution (OOD) setting, rather than an in-distribution (ID) scenario. In addition, the authors presented several examples in ID scenarios to highlight the difference between two scenarios. 
* I’d like to see the challenges in OOD scenarios. Specifically, I agree that model selection is problematic in this setting. 

### Weaknesses
Clarity should be improved. In the submission history above, the one of critical reviews on an earlier version of this work is the lack of clarity. However, the current version is not that clear in many places. 


### Detailed comments
In my understanding, FSL benchmarks have been deliberately designed in OOD scenarios to validate meta-learning algorithms on how to quickly adapt new skills in different environments. Therefore, I believe that the current evaluation protocol isn’t invalid. 

I couldn’t follow the logic behind that randomized class partitions don’t imply randomized task partitions. Specifically, I’m not sure why a task is represented by a class tuple, rather than a class. 

Some attempts have been proposed to mitigate the negative effects of domain mismatch between the meta-training and meta-testing. I think that the papers below are related to the analysis presented here.
* TAEML: Task-Adaptive Ensemble of Meta-Learners, NeurIPS workshop on MetaLearn, 2018. 
* Domain-Agnostic Few-Shot Classification by Learning Disparate Modulators, NeurIPS workshop on MetaLearn, 2019. 
limitations_and_societal_impact: This work is purely theoretical, so this doesn't bring any negative societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper studies the evaluation of meta-learning algorithms on in-distribution and out-of distribution learning tasks. Through an argument based on a proof by contradiction, the paper shows how most of the current evaluation benchmarks for few-shot learning are based on out-of-distribution evaluations. The paper outlines the difference between in-distribution and out-of-distribition evaluations, provide examples for the necessity of in-distribution evaluations, highlight concerns about model selection strategies and algorithm comparison when using OOD evaluation benchmarks, and propose suggestions to remedy these concerns by introducing new IID benchmarks and modifications to the existing OOD evaluations.
main_review: ##  Originality

- The categorization of meta-learning evaluation into in-distribution and out-of-distribution is an important distinction for meta-learning researchers and practitioners to consider. 

- The suggested insights about the subtleties in out-of-distribution evaluation are informative for the meta-learning community. 

- The related work section is adequate and covers recent related work in this area.

## Quality

-  The paper is technically sound and the claims in the paper are supported by empirical evaluations, however, almost all the experiments were performed on computer vision tasks, and other applications for meta-learning in language and speech were not considered.  
- The paper could also benefit from a more in depth theoretical analysis. 


## Clarity

- The paper is clear, well organized, and easy to follow, except for the distinction between “Snapshot selection” vs “Hyper-parameter selection”. It’d be useful if the authors could provide a concrete example to distinguish between the two concepts.

## Significance

- The work is significant and presents new insights on the evaluation of meta-learning methods, model selection, and comparison between different approaches.

- The experimental analysis is thorough, however, except for the proof by contradiction argument for out-of-distribution evaluation of the current benchmarks, the paper lacks the theoretical insights.

- Except for a small section for “Actionable Suggestions”, the paper lacks more insights to mitigate the inconsistencies of the current few-shot learning benchmarks, and issues regarding model selection and comparison of different approaches.
 
 
limitations_and_societal_impact: - Results are mostly empirical, and in many cases lack theoretical analysis.
- The papers highlights significant inconsistencies and issues regarding the current meta-learning evaluation benchmark, this is very useful for the meta-learning community to know, however, I still think the suggestions provided in the paper doesn’t address all of these issues, for instance, the answers for these questions are not clear to me:

1) Out of the four meta-learning approaches used in the paper (proto-nets, meta-opt-net, foMAML, and R2D2), which approach consistently works best for in-distribution and out-of-distribution learning tasks?
2) It’s possible that a practitioner would be interested in developing meta-learning algorithms that work well for both in-distribution and out-of-distribution learning tasks. For instance, in “Example 1” for federated learning, maybe we’d also be interested in deploying a model for a new user writing in a totally different language script, what would be the best practice recommendations for model selection and comparison in this setting?
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this paper, the authors focus on certain limitations of popular few-shot learning benchmarks (e.g. MiniImageNet or TieredImageNet) that are used to evaluate meta-learning algorithms. The authors argue that these benchmarks, although generally known to be testing in-domain generalization, actually test out-of-domain generalization and therefore the meta-learning algorithms that are benchmarked using these datasets over optimize for out-of-domain generalization at the cost of in-domain generalization. The authors also shed some lights on some other pitfalls of meta-learning training and evaluation procedures like how to select the best model during training and how increasing the amount of novel classes for meta-test can make the test-time performance evaluation more robust.
main_review: In this paper, the authors challenge an existing assumption in common few-shot learning benchmarks (e.g. MiniImageNet or TieredImageNet) that are used to evaluate meta-learning algorithms that these benchmarks actually test out-of-domain generalization although the benchmarks claim to be in-domain (as the classes are split from the same dataset). Empirically, the authors show that by evaluating meta-learning algorithms using these benchmarks, those are incentivized to do better on out-of-domain generalization at the cost of in-domain generalization. Apart from this, the authors also provide analysis on some other aspect of meta-training like how to pick the best model during training and how to design the test set better to improve the test time evaluation procedure. Overall I found the paper to be easy to read and found the notations to be clear. I also like some of the areas that the authors have focussed on like 1)  how to do model selection during meta-training 2) including additional samples from the base classes to do in-domain evaluation and 3) include more novel classes in the meta-test segment to make the evaluation more robust. 

However, for me, there are certain major drawbacks of the paper that I'd like to point out:
*  The main claim that existing benchmarks focus on out-of-domain generalization and the meta-learning methods optimize for that are not solidly justified via the experiment(s). From Table 1, we can see that the methods perform pretty much identically on both in-domain and out-of-domain (as per the notion of this paper) rather than the claim that the trend is reversed between these two scenarios.   PN, Ridge and SVM all perform within $1$ percent or less on all the three benchmarks (especially when considering the confidence intervals, the gap is insignificant). This, in my opinion, is not sufficient to justify this whole claim of in-domain vs out-of-domain and that the methods optimize one vs the other. To me, it looks like they all perform quite similarly on both in-domain and out-of-domain.
* Secondly, I am not convinced with the rationale of why the authors consider the classes belonging to meta-test as out-of-domain. I would have liked to see a more rigorous analysis of domain-gap between the tasks sampled from the base and novel classes to arrive at this conclusion. For example, the authors can try using an algorithm like [Task2Vec](https://arxiv.org/abs/1902.03545) to embed the tasks from these two splits and then compare their mutual distances to verify their claim.
* On the topic of including additional images from the base classes, this idea is not new and has already been used in [this](https://arxiv.org/abs/1804.09458) paper by Gidaris et al.
*  I agree with the author's claim that using meta-validation set might not be the right way to figure out the best model during training. However, the authors do not provide a clear alternative and suggests also considering using the best model that gets the lowest loss on the meta-train set. As a practitioner, it is not clear to me which method should I choose.
* Although the paper is easy-to-read, the organization of the topics is not great.  In my opinion, the paper tries to address too many points without a very clear suggestion or solution for most of those. 

Overall, I think the paper has some merit when it comes to addressing a few concerns with meta-learning training and evaluation, but in its current form, it does not have enough novelty nor the claims are sufficiently backed by the experimental results. 
limitations_and_societal_impact: There are no potential negative societal impact of their work as far as I understand.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work differentiates two evaluation procedures for meta-learning approaches: *in-distribution (ID)* and *out-of-distribution (OOD)*. In the former, the *tasks* used in the meta-training and meta-testing stage come from the *same distribution* whereas in the latter, the tasks come from *different distributions*.
The authors give a formal proof to show that current Few-Shot Learning (FSL) benchmarks only give OOD performance and not ID performance. They propose updated versions of these benchmarks to reflect ID performance of popular FSL models. They show that the performance can vastly differ between OOD and ID, and that the ranking in terms of performance of the models is almost inverted. Furthermore, they also verify if selecting the best model based on validation performance is really correlated with the best performance in an ID setting, and show that it is not always the case. Finally, they investigate the inconsistencies in performance between multiple version of the same models in relation to the number of classes in the novel set used for meta-testing. 
main_review: __**Originality:**__  This paper challenges current meta-learning benchmarks and provide formal justifications to consider them as OOD even when evaluating on the same dataset, rather than ID. As mentioned by the authors, some prior works in related topics already considered a comparison of performance on *base classes* and *novel classes*, such as for transfer learning, incremental or continual learning methods. Meta-learning methods, however, were always compared using a novel set of classes in the literature. This is an interesting and novel way to evaluate meta-learning methods.  

__**Quality:**__  Their main argument about the OOD meta-learning benchmarks is supported by a formal proof-by-contradiction. The authors also proved it empirically by modifying the meta-learning benchmarks and showing that performance rankings on the current OOD benchmarks does not translate to their proposed ID-version of the benchmarks. However, they do not give explanations as to why some methods can have drastically different performance rankings between ID and OOD (like PN), while others do not (like FOMAML).
Finally, the authors also highlight the problem of model selection with current OOD benchmarks. However, I wonder if the inconsistencies found in the performance comparisons are linked to the high variance of performance between tasks, since only mean accuracies are compared. This variance is known to be higher than the $\epsilon = 0.5\%$ chosen by the authors on certain benchmarks.

__**Clarity:**__  A lot of important details only appear in the appendix. In the main paper, the section 3 and particularly the part explaining the formal result, which is the starting point of the paper, are quickly explained. I had to read the appendix to better understand the proof and the result. All the experimental settings are also relegated to the appendix, even though I think they are important to assess if the experiments and comparisons are sounded or not. Other than that, the paper is well written, and its objectives are clear. 

__**Significance:**__ I think the paper is valuable to meta-learning researchers and practitioners, even more so for applications which would benefit from true ID benchmarks as detailed in the paper. There are still important problems, such as the model selection, but the paper offers a novel view to current meta-learning benchmarks.

### Minor Remarks

- Some minor typos I noticed:
  - l.247-248: "is in some sense fail~~s~~*ing*"
  - l.392: "set ~~set~~"
limitations_and_societal_impact: The authors did not discuss potential negative impacts of their work. Even though the work presented here is not fully theoretical, it is high upstream of the applications. To study the possible negative impacts of this kind of work is to study the negative impacts of the whole field, since it is challenging the way we are evaluating meta-learning models. As any other field in machine learning, meta-learning could also be used for harmful applications, such as quick racial profiling of a user.
review_title: null

Meta-review: This paper examines few-shot meta-learning from the perspective of in-distribution vs out-of-distribution task splits, and demonstrates that optimising for one criterion may lead to worse performance on the other. This distinction has not been very carefully considered by the community before. While there are some concerns, reviewers unanimously agree that the theoretical and empirical analysis will be of value to the meta-learning and few-shot learning community. I agree and recommend accept.
Decision: Accept (Poster)