Two Sides of Meta-Learning Evaluation: In vs. Out of Distribution
Authors: Amrith Setlur
Oscar Li
Virginia Smith
Venue: NeurIPS 2021 Poster (2021)
Keywords: meta-learning
meta-learning evaluation
few-shot learning
out of distribution

Abstract: We categorize meta-learning evaluation into two settings: $\textit{in-distribution}$ [ID], in which the train and test tasks are sampled $\textit{iid}$ from the same underlying task distribution, and $\textit{out-of-distribution}$ [OOD], in which they are not. While most meta-learning theory and some FSL applications follow the ID setting, we identify that most existing few-shot classification benchmarks instead reflect OOD evaluation, as they use disjoint sets of train (base) and test (novel) classes for task generation. This discrepancy is problematic because -- as we show on numerous benchmarks -- meta-learning methods that perform better on existing OOD datasets may perform significantly worse in the ID setting. In addition, in the OOD setting, even though current FSL benchmarks seem befitting, our study highlights concerns in 1) reliably performing model selection for a given meta-learning method, and 2) consistently comparing the performance of different methods. To address these concerns, we provide suggestions on how to construct FSL benchmarks to allow for ID evaluation as well as more reliable OOD evaluation. Our work aims to inform the meta-learning community about the importance and distinction of ID vs. OOD evaluation, as well as the subtleties of OOD evaluation with current benchmarks.

Two Sides of Meta-Learning Evaluation: In vs. Out of Distribution
Amrith Setlur'
Oscar Li oscarliecmu, edu
Virginia Smith²
asetlurčcs cmu, edu "Language Technologies Institute
smi thvūcmu. edu Machhne Learning Department School of Computer Science, Carnegie Mellon University
Abstract
We categorize meta-learning evaluation into two settings: in-distribution [ID], in which the train and test tasks are sampled iid from the same underlying task distribution, and out-of- offiiiiiibbtii [OOD], in which they are not. While most meta- learning theory and some FSL applications follow the ID setting, we identify that most existing few-shot classification benchmarks instead reflect OOD evaluation, as they use disjoint sets train (base) and test (novel) classes for task generation. This discrepancy is problematic because-as we show on numerous benchmarks- meta-learning methods that perform better on existing OOD datasets may perform significantly worse in the ID setting. In addition, in the OOD setting, even though current FSL benchmarks seem befitting, our study highlights concerns in I) reliably performing model selection for given meta-learning method, and 2) consistently comparing the performance of different methods. To address these concerns, provide suggestions on how to construct FSL benchmarks to allow for ID evaluation as well as more reliable OOD evaluation. Our work aims to inform the meta-learning community about the importance and distinction of ID vs. OOD evaluation, as well as the subtleties of OOD evaluation with current benchmarks.
Introduction
Meta-learning considers learning algorithms that perform well over distribution of tasks 19 To do so, a meta-learning method first learns from a set of tasks sampled from a training task distribution (meta--rainingg) and then evaluates the quality of the learned algorithm using tasks from a test task distribution (meta-testing). The test task distribution can be the same as the training task distribution scenario we term in-distribution generalization evaluation or ID evaluation) or different task distribution (ont-of-distribution generalization evaluation or OOD evaluation).
37
In this work, argue that there is need to carefully consider current meta-learning practices in light of this ID vs. OOD categorization. In particular, meta-learning is commonly evaluated on sewt learning (FSL) benchmarks, which aim to evaluate meta-learning methods' ability to learn sample-efficient algorithms. Current benchmarks primarily focus on image classification and provide training tasks constructed from of train (base) classes that are completely disjoint and sometimes extremely different from the test (novel) classes used for test tasks. As we discuss in Section 3 this design choice imposes a natural shift in the train and test task distribution that makes current benchmarks reflective of OOD generalization. However, there are a number of reasons to also consider the distinct setting of ID evaluation. First, whether in terms of methodology or theory, many works motivate and analyze meta-learning under the assumption that train and test tasks are sampled iid from the same distribution (see Section 2}. Second, we identify growing number of applications, such as federated learning, where there is in fact need for sample-efficient algorithms
Authors contributed equally to this paper. "oode available at https://github.ee http://htwb.ooommm////tttttteeee -oo------ood
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

that can perform ID generalization. Crucially, we show across numerous benchmarks that methods that perform well OOD may perform significantly worse in ID settings. Our results highlight that is critical to clearly define which setting a researcher is targeting when developing new meta-learning methods, and we provide tools for modifying existing benchmarks to reflect both scenarios.
Beyond this, we also re-examine current OOD FSL benchmarks and analyze how the shift in the train and test task distributions may impact the reliability of OOD evaluations. We point out two concerns which we believe are not widely considered in the meta-learning community. First, unlike areas such as domain generalization where model selection challenges are more widely discussed [8] 23). we conduct to the best of our knowledge the first rigorous study demonstrating the difficulty of model selection due the shift in the validation and test task distributions in FSL benchmarks. Second, because the OOD scenario in meta-learning does not assume a specific test task distribution, there is room for different test distributions to used for evaluation. We show that comparing which meta-learning method performs better can be unreliable not only over different OOD FSL benchmarks, but also within a single benchmark depending on the number of novel classes.
Our main contributions are: i) We clearly outline both ID and OOD FSL evaluation scenarios and explain why most popular FSL benchmarks target OOD evaluation (Section ii) We provide realistic examples of the ID scenario and show that the performance of popular meta-learning methods can drastically differ in ID vs. OOD scenarios (Section 4. iii) For existing OOD FSL benchmarks, we highlight concerns with a) current model selection strategies for meta-learning methods, and b) the reliability of we suggest suitable modifications to the current FSL. benchmarks to allow for II) evaluation, and explain how to construct FSL benchmarks to provide more reliable OOD evaluation. Our hope in highlighting these evaluation concerns is for future researchers to consider them when evaluating newly proposed meta-learning methods or designing new FSL benchmarks.
method comparisons (Section (5). iv) To remedy these concerns.
2 Related Work
Current FSL benchmarks. A plethora of few-shot image classification benchmarks (e.g., mini- ImageNet (mini in short) 42), CIFAR-FS A) have been developed for FSL evaluation. These benchmarks typically provide three disjoint sets of classes (base, validation. novel) taken from standard classification datasets, e.g., ImageNet or CIFAR-100 [36 32 42]. Training, val, and test tasks are then constructed from these classes respectively, which, as we discuss in Section 3 can induce a shift in their corresponding task distributions. Distribution mismatch can be particularly large with non- non-random splits created at the super class level, e.g., FC- 30. or dataset level, e.g. Meta- Dataset [4).. Recently, Arnol and Sha (2) propose an automated approach to construct different class splits from the same dataset to allow for varying degrees of task distribution shifts; Triantafillou et al. [1 separate the distribution shifts on Meta-Dataset into weak vs. strong generalization depending on whether the novel classes are taken from the used training datasets or not. Both works find that the meta-learning methods that perform better in one distribution shift scenario might perform worse in another, providing further evidence to our OOD performance comparison inconsistency argument in Section 5.2 Beyond these canonical ways of task construction through set of classes, Ren et al. (35 propose new benchmarks in a new flexible few-shot learning (FFSL) setting, where the aim is to classify examples into a context instead of object class. During testing. they perform OOD evaluation on unseen contexts. Inspired by their approach, we also provide an FSL benchmark where tasks are specified by contexts (Section 4 though we differ by exploring ID evaluation.
Mismatch between meta-learning theory/methodooogy and evaluation. Despite theoretical works which analyze meta-learning OOD generalization 13), there are many theoretical meta-learning works [c.g.. I 3 8, 22 that first assume the train and test tasks are iid sampled from the same distribution despite validating their analyses
OOD FSL benchmarks. Additionally, several popular meta-learning methods 25 16 31 that are motivated in the ID scenario are largely evaluated on OOD benchmarks (see Appendix A.. Prior work of Lee et al. 24 explores ID vs. OOD; however they treat the FSL setup as ID when the disjoint base and novel classes are from the same dataset and OOD only when they not (e.g.. base from ImageNet, novel from CUB). We emphasize that even the disjointedness of base and novel from the same dataset can create a task distribution shift and hence unlike 24] we do consider current FSL benchmarks (like mini) to be ID (Section 3).
FSL. ID evaluation. We are unaware of any FSL classification benchmarks that are explicitly advertised for ID evaluation. As we discuss in Section 4 a growing number of works 7 20 22

---

use meta- meta-learning for personalized federated learning, but do not clearly discuss the in in-distribution nature of these benchmarks nor how they differ from standard FSL benchmarks. In recent work, Chen et al. [10] extend current FSL. benchmarks to evaluate their proposed method both ID) and OOD, but only to further improve OOD performance on the original FSL benchmark's novel classes. Our work uses similar setup for constructing ID) evaluations with current OOD FSL benchmarks, but focuses on showing that certain meta-learning methods/design choices can improve OOD performance at the cost of ID performance. Prior work [17 34 incremental few-shot/low-she learning also explores performance on both base and novel classes simultaneously. However, they differ in their methodology, as they use supervised learning (not learning) to classify over the base classes.
OOD evaluation in other fields. Train and test distribution shifts are also found in domain gener- alization [5] 29 where the goal is to find model that works well for different test environment. Due to this shift, Gulrajani and Lopez-Paz, [8] specifically discuss difficulties in performing model selection in domain generalization benchmarks. They argue that it is the responsibility of the method designer (not benchmark designer) to determine model selection strategies for their method, and propose several model selection methods, mainly targeting hyperparameter selection. Unlike domain generalization, FSL benchmarks often have a pre-determined disjoint set of validation classes to construct validation tasks, so the need for developing model selection methods may be less obvious. In Section 5 motivated by [88], we explore strategies for model selection for meta- weea--earning. However, in contrast to [18], we focus on algorithm snapshot selection for meta-learning, which is required by hyperparameter selection as a subroutine (exact definitions see Section 5).
3 FSL benchmarks: Background & Focus on OOD evaluation
Background and notation. In this work, we employ general definition of meta-learning FSL task: a task T is a distribution over the space of support and query dataset pairs (S, Q), where S, Q sets of examples from example space X' V. The support set S is used by algorithm to produce an adapted model which is then evaluated by the corresponding query set Q. Each time we interact with a task T, (S, Q) pair is sampled iid from T, the mechanism for which depends on the specific application; we provide multiple examples below. As discussed in Section meta-learning aims learn an algorithm over distribution of tasks P(T). During meta-training, we assume access to N pairs of {(S,, Q+) Hee[N] sampled from a training task distribution P (T) in the following way: first, N tasks are sampled iid from the training task Ti ~ Ptt((T); then, for each task Te, a support query pair (S,, Q.) ~ T7 is sampled iid. During meta-testing,
assume there is test task distribution Pre (T) where fresh (S, Q) samples are similarly sampled based on Pte (T). We define a learning scenario to be in-distribution (ID) if Pt, It and out-o--distributinn (OOD) if irr o IP teee Whenever Pt. Ft.. the induced train and test marginal distributions of (S, Q) are also identical.
Construction of (S, Q) pairs in FSL. Most popular FSL benchmarks share similar structure: they provide three disjoint sets of classes: base classes CB. validation classes Cy, and novel classes CN. where any class in these sets specifies distribution P. over the example space X. An n-way k-shot q-query task To in these benchmarks is specified by a length n non-repeated class tuple (cI, Cn) where e E [c*] == ((d1,... dn) E C" dl # dj Vi # j]. Te generates random (S, pairs in the following way: For every class C,, ht support examples S: ~ (Pa,)* and q query examples Qi ~ (P,,)" )" are sampled. The support and query set is formed by the union of such labelled examples from each class: S = U._1 ((x, i), Vx E S.). Q ( i), Vr E Qi). By specifying the base and novel classes, the FSL benchmark has provided a collection of tasks for be extremely large. For example, 64'
training (Te: c [CH]] and test [Te : e E [c These sets
in mini, which has 64 base classes, the total number of 5-way training tasks is
764-5) R 9.1 x 10
However, it is not explicitly specified what underlying task distribution P(T) generates these sets of tasks. We believe this may have led prior work to discuss FSL benchmarks in the context of ID evaluation [e.g.. 1 88, contrary to what we outline below.
Current FSL benchmarks target OOD evaluation. We now informally discuss our reasoning for arguing that current FSL benchmarks reflect OOD evaluation (we provide a more formal proof by contradiction in Appendix B). In particular, the training and test tasks in FSL benchmarks are indeed iid sampled from the same underlying task distribution, then this underlying distribution must be induced by a distribution over class tuples of an even larger class set C₁ (C, 2 (Cg U Cv Ux)).
'We note that "training task distribution" here refers to the true underlying distribution of training tasks, not the empirical distribution supported over the finite set of sampled training tasks.
3

---

We consider the following dichotomy:
when Cr.] (nnN): In this case, the total number of classes n.N covered by the sampled tasks (counting repetition) is greater than the number of underlying classes Then with high probability, both the training and test tasks would each cover significant portion of all the classes in CL. making it extremely unlikely to have an empty intersection as in the current FSL benchmarks. ii) when Ccz] = (2(NN): In this alternative case, the total number of classes (even counting repetitions) used by sampled tasks is still smaller than the number of underlying classes |Ct/-- Thus the sampled tasks cannot cover all the underlying classes. Under this regime, the number of classes covered by the training tasks alone would scale linearly with N, as repeating an already-seen class in a new task sample is relatively rare. Since FSL benchmarks typically use a large number of training tasks during meta- meta-training (N > 10"), it is improbable that all the training tasks would together only cover a very low number of classes (64 the case of mini).
Randomized class partitions do not imply randomized task partitions. Another issue that may cause some to view the current FSL benchmarks as performing ID evaluation is that in some of these benchmarks, the base, val, novel classes are random partitions of iid drawn classes from a class level distribution (specifically minilmageNet, CIFAR-FS; but not FC-100, mieredImageNet as the classes are not partitioned randomly). The logic here is that in standard machine learning practice, randomly partitioning iid sampled data points into train and test guarantees that the train and test samples are drawn itd from the same underlying distribution. However, it is important to notice that the first class citizen in common FSL benchmarks is not a class, but a task (represented by a class tuple). So. only a randomized partition of iid sampled class tuples would guarantee in-distribution sampling.
How can view Ptr, Pt. in common FSL benchmarks? Based the discussion above, we need to view train and test tasks in current FSL benchmarks as coming from different distributions, i.e., Pree of Pte. In order to ensure that both sets of tasks are still sampled iid from their respective distributions, it is convenient to view the train/test tasks as being iid sampled from a uniform distribution over all possible train/test class tuples induced by CB/Cn i..., Ptr Pcg Unif( ([Te E [CAl)) and test Fie = Fex == Unif( (Te c E (cx])) - a view which we will adopt in the rest of the paper.
4 Evaluating n-Distribution Performance
Although (as discussed in Section 3 current FSL benchmarks target OOD evaluation, example applications where ID generalization is instead required, and provide easily constructible benchmarks mirroring these scenarios. As will show, this distinction is important because meta-learning methods may perform markedly different in ID vs OOD scenarios.
explore
Example I (Federated Learning): Multiple works 12 20 22 27 have considered applying meta- learning methods in federated learning, in which the goal is to learn across a distributed network of devices (28] [6). Meta-learning can produce personalized models for unseen devices/users, improving over a single globally-learned model's performance. In this setting, a popular benehmark is the FEMNIST 66 handwriting recognition dataset. For FEMNIST,
assume there exists distribution of writers (idd in a federated network where each writer (with a unique id) is associated with a few-shot classification problem to recognize over the different character classes the writer has written for. We associate each writer id with a task Tia which randomly generates a support set with one random example per class and a query set with varying number of random examples per class.
ID evaluation on FEMNIST. We are given a total of 3500 writers sampled iid from P(id) and we randomly partition them into a 2509/538 /38 split for training, validation, and test tasks, following similar practices used in prior FL work (20 7. Note that this random split is performed at the task/id level. As such, we can treat the training and test tasks as being sampled iid from the same task distribution, unlike current FSL benchmarks.
Example 2 (Online Recommendation): Ren et al. (35] propose the use of Zappos 43 dataset as a meta-learning benchmark where each task is binary classification of shoe images into an attribute context. This mimics an online shopping recommendation problem, where each user has different shoe preferences based on specific shoe attributes (hence a single global predictive model would not perform well), and the recommendation system must quickly learn a user's likes/dislikes through a few interactions. In this simplified setup, we a universal set of shoe attributes A, and each user's preference is represented by a specific pair of unique attributes Ta representing a user with attribute preference a generates 2-way k-shot q-query (S, Q) pair by tid sampling k + q examples both from the set of images that carry both attributes in a (positive
(a1, (2), 01 o 0z A task

---

Table 1: Ranking in () of meta-algorithmms test performance on i) ID benchmarks FEMNIST, Zappos-ID (with either 1000 or 50 training tasks); and ii) OOD FSL benchmark minlmmaeNee..
D Method
FEMNIST
Zappus- ID
minilnageNet 5w.s
*wls
2w1Bk
(1000 train tacks)
2w.5 tai taske))
main
PN
(194.7 0.41%
88 40 + 0.13%
(1) 86.58 + 0.15%
77.67 : 0.17%
76.22 # 0.14%
Ridge
(4944.11 + 0.42%
88.01 +: 0.14%
85.56 + 0.16%
74.75 + 0.16%
77) + 0.15%
SVM
(9)91.22 0.45%
87) 0. 141%
82.12 # 0.16%
(4)70.00 + 0.17%
(A)77.72 # 0.15%
FO-MAML
N/A
81.90 + 0. 0.14%
80.14 # 0.15%
69.85 = 0.18%
(4.75.9 + 0.17%
examples) and from the set that does not (negative examples), Our task distribution is uniform distribution over tasks of all attribute pairs (P((7) = Unif([Ta : a E [A
ID evaluation on Zappos. Unlike in 335 where Zappos is used to measure OOD performance by having disjoint train and test attributes Aue n = c. this work we use Zappos for ID evaluations by tid sampling both meta- taata--raiinna and meta-test tasks from the same distribution I' (7)) Through this modification of Ren et al. [s setup, we sample 1000 / 50 attribute pairs from an attribute set |A| = 36 to construct 1000 / 50 training tasks (each with randomly sampled (S, Q)) and evaluate ID performance on another 25000 test tasks sampled in the same way. Our evaluation setup captures realistic setting where the goal is to learn an algorithm that can generalize to the entire online shopper population despite being trained only on a randomly chosen subset of shoppers.
Remark, Even with ID evaluation it is possible to encounter unseen classes/attributes in meta-test tasks, specifically when the number of meta-train tasks is smaller than the number of underlying classes/attribttes (Section B). However, a suitable number of fid sampled meta-train tasks is needed to ensure good performance, which would naturally encompass a larger set of meta-train classes than those considered by OOD FSL benchmarks. For example, there are still 16 attribute pairs from the test tasks that are unseen in the 1000 training tasks on Zappos-ID. but the dataset is still in-distribution since the sampling distributions of train and test attribute pairs (and thus of tasks) are identical.
ID benchmark results. We evaluate the ID performance of four popular weeannne methods: Prototypical Networks (PN) [39], MetaOptNet-SS VM (SVM) (25), MetaOptNeet
Regression and FOMAML [15 on our identified ID FSL benchmarks (Table [ Since FEMNIST's
(RR) 25 4
tasks have varying number of ways, FOMAML cannot be directly used due to the logit layer shape mismatch. We note that the performance order of the four methods are consistent on all three ID benchmarks yet surprisingly almost completely opposite to the performance order observed on the OOD benchmark mini (except for FOMAML). In terms of the actual performance differences, we notice that the ID performance advantage of PN over SVM becomes particularly large (> 3%) when we reduce of the number of training tasks for Zappos-II to 50; in contrast, on the OOD benchmark mini, SVM instead outperforms PN by 1.5% (a significant number as many newly proposed meta-learning methods often only report improvements over previous methods by 1.5% or less). These performance differences make it clear that the performance ranking flips between ID and OOD indeed exist, and as a result, these common OOD FSI. benchmarks (like mini) cannot be used to compare ID performance without modifications, giving further evidence to the danger of such practices (see Section 2). To further understand this phenomenon, we propose a way to also enable ID performance evaluation over these common OOD FSL benchmarks and see if there still exists difference in ID, OOD performance orders.
Modifying OOD FSL benchmarks for ID evaluation. From our previous discussion, we have shown that we can think of FSL training tasks as being sampled iid from the task distribution Pcg := (Te e E [CB])). To conduct an in-distribution evaluation, we need to sample fresh test tasks idd from Pcg. For freshly sampled To ~ IPcg. we also need to iid sample a fresh support query pair (S, Q) ~ Te- To ensure independent sampling from the already seen meta-training (S, Q) pairs, we need to introduce new examples from P. for each class c E Cg. Thus we construct slightly modified versions of four common FSL benchmarks i) miniImageNet-Mod (mini-M) [42). in which we find (s 700) unused examples (from ImageNet) for each base class and use them to evaluate the performance over I' P'Cz. Here the original 600 examples of each base class are still only used for meta-training ii) CIFAR-FS-Mod (cifar-M) (30 FC-100-Mod (FC-M) [4, and tieredImageNet-Mel (fiered-M) :3:: As don't have additional samples for base classes, randomly partition each base class's 's current examples into an approximate 80/20 split where the training tasks are constructed using the former and the latter is reserved for ID evaluation.
ID vs. OOD conflicts still exist. In addition to evaluating the four aforementioned meta-learning methods, we also consider two supervised pretraining methods: the supervised learning baseline (SB)
5

---

35.0
61
92.0
77.0
15.5
166
34.3
91.5
94.5
72.0
95.0
33.5
12.0
79
sage
01.0
U1
63.5
07.0
64.3
RI Rated:
54.0
$4.5
144
nidpe
7H MG
92.02
50.5
52.0
Kikge Hora
93.5
77
557 07.0
92.5
89.5
89.5
90.5
69.5
54.5
92.0
92.5
82
85.0
mini-M Sw5s (Best
75
89.2 0
bicrad-M Sw5s (Best
78
(c)
18.0-
1.67
(Best
22.0
cifar-M (Gest Val)
81
Figure 1: We show the BaseGen and NovelGen performance tradeoff (for best validation snapshots); over the choice of set of four meta- neta-learning and two supervised pre-trainite methods on mini-M M (a) and tiered-M (b); over the number of ways to train PN on mini-M M and different learning rates to train FOMAML on FC-M (c); over the use of FIX- ML (S, Q) generation strategy or not (ML) with SVM, RR and PN on cifar-M M in (d).
p and Meta-Baseline (MB) 10. Both of these methods have been shown to outperform many meta- learning methods on the current OOD FSL benchmarks. For succintness, we call the generalization performance over the training task distribution ee,, BaseGen and performance over Fey NovelGen (we will refer to performance on validation tasks from Pc, ValGen in a later section). We plot the BaseGen and NovelGen performances of the aforementioned methods on minilmageNet-Mod and fiered dotted line connecting BaseGen and NovelGen value of same learned algorithm snapshot of a meta- learning method. We see that the ID/OOD performance order flips still exist within current FSL benchmarks themselves (the dotted lines of different methods cross each other frequently), showing that the issue of improving OOD at the expense of ID is a common realistic phenomenon for multiple benchmarks. In addition, despite outperforming all meta-learning methods on NovelGen, the non-meta-leame learning methods SB and MB cannot beat the best meta-learning methods on BaseGen, which demonstrates their restrictive advantage only in the OOD setting. More broadly speaking, these OOD FSL datasets constructed with belief that there exists commonality between the training and test task distributions so that an algorithm capturing this commonality using the training distribution alone would also generalize to the test tasks. Thus the phenomenon of improving OOD while sacrificing ID means the algorithm has in some sense failed to learn the essence/commonaltty from the given training tasks. Additionally, we that the BaseGen ranking of the four meta-learning methods over these common FSL bechmarks are exactly the same as the ranking over our two newly proposed ID benchmarks in Table [ We suggest that researchers who want to also test on their proposed methods' ID generalization performance can perform BaseGen evaluation method in our proposed way as is a simple addition their existing NovelGen evaluation setup.
in Figure [b)(b) (other datasets see Appendix
E Figure 4 ab)()), with
Certain OOD training choices might harm ID generalization. In addition to checking the discrepancy of ID vs OOD generalization comparison of different meta-learning methods, now ask, for a given meta-learning method, whether the meta-training choices found to be most effective on NovelGen would still be optimal for BaseGen.
i) Meta-training hyperparameters: In Figure we see that a higher learning rate when training FOMAML finds algorithms that have higher BaseGen, whereas lower learning rates are better for NovelGen. Additionally, we see that the proposed technique of training with more ways for PN in [39 can lead to better NovelGen performance but worse BaseGen performance than the 5-way trained PN whose training and test task configurations match.
ii) Meta-training (S,Q) generation alternatives: It was found in 38 that always using the same support examples for every base class when constructing S (FIX-ML) can improve NovelGen performance for several meta-learning methods over multiple OOD FSL benchmarks. However, restricting the training (S, Q)'s diversity sampled from Pcg seems counter-intuitive and we wouldn't I expect to improve the in-distribution generalization BaseGen. In Figure [d), we indeed see that MI only improves NovelGen performance at the expense of BaseGen by training over a much less diverse set of tasks.
These two observations above caution that some training techniques to boost test accuracy FSL benchmarks might only work for the OOD scenario but not the ID scenario.
Challenges With Out-of-Distribution Evaluation
After identifying some example benchmarks for ID evaluation, we come back to the current OOD FSL benchmarks to further examine some subtle challenges in OOD evaluation. Here, instead of focusing on the distinction between ID vs. OOD, we now look at some reliability and inconsistency problems

---

min/mmagenet-Mod 64w
tireedimagenetModd PN 20w
CIFAR-FS-M SVM
0.8
0.7
Supavbee
Accurary
0.6
0.5
Accuracy
Accuracy
05
0.4
0.3
0,2
70-
BaseGien hovdlGen
0.1
Epochs
0.0
mih M ciard
PC M ZEspos
10
25
30 0b 55 Epochs
10 30 to 40 Epochs
(8)
(b)
(c)
Figure 2: We plot the (Base, Val, Novel)Gen progression of way trained PN mini-M in (a) and of SVM trained on cifar-M in (d). In (h) we compute the Kendall rank correlation coefficient (p) between the validation and test rankings
model snapshots for IC (trained on cifar) and algorithm snapshots for PN, SVM. RR on OOD datasets mini-M (last 40 epochs), cifar-M (last 40), FC-M (last 10), tiered-M (last 20) and ID dataset Zappos-ID (last 30); in (c) we show the BaseGen tracking NovelGen for 20- trained PN tiered-M.
within the OOD evaluation itself. In particular, we highlight two concerns: 1) Despite providing a validation set of classes for task construction during mnta-validattin, it is not clear whether this is reliable way to perform model selection. 2) As there is no requirement on how similar the training and test task distributions need be in the OOD setting, there can be inconsistencies in method comparisons when the evaluation setup is slightly modified.
5.1 Model Selection
To compare set of meta-learning methods on the OOD test task distribution, one needs to select a representative algorithm learned by each method to be evaluated on the test task distribution. To do so, one should first decide what set of hyperparameter configurations to choose from for given meta- learning method (we define the entire set all hyperparameters for a training run as a hyperparameter config). For each such considered config, after its training is completed, we need to choose one of the algorithm snapshots saved during training to represent it (which we call snapshot selection). Then the set of hyperparameter configs are compared based their respectively selected snapshots, and single config is then chosen among them (which we term hyperparameter selection). This config's s selected snapshot will represent the given meta- vecaalearning method to be finally evaluated the test tasks. We refer to the combined problem of hyperparameter and snapshot selection as model selection. (See Appendix F.T for a simplified example of snapshot and hyperparameter selection; see H22 for the distinction between the commonly used technique early-stopping and snapshot selection.)
Snapshot Hyperparameter selection. If the snapshot selection strategy cannot reliably identify a snapshot with good test performance, unfairly represented by mischosen snapshot, leading to erroneous hyperparameter selection. Thus we believe snapshot selection is a more fundamental problem and we focus our analyses on it (more on hyperparameter selection in Appendix F.5). In OOD FSL benchmarks, reserved set of validation classes (disjoint from the set of novel classes) is provided, which as we have argued, provides tasks which are not sampled tid from the test task distribution. As result, we ask: is performing validation on the given validation tasks reliable, and, if not, are there other options? In contrast, there is typically need for such concerns in standard ID supervised learning, where the validation set is sampled iid from the test distribution.
possible that some hyperparameter configs will be
5.1.1 Option 1: Snapshot selection tradition using ValGen.
By providing a set of validation classes, it has become the default practice for meta-learning methods to use ValGen performance for snapshot selection. However, because IPc, and Fcx are different tasks distributions, it is not clear whether higher ValGen performance is strongly correlated with a higher NovelGen performance. In Figure a). we plot the progression of ValGen and NovelGen of a 64-way trained PN on mindmageNNet-Moo 5w5s tasks. We notice that ValGen is consistently higher than NovelGen, indicating that meta-val performance is not an accurate estimator of NovelGen. More importantly, we see trendwise that while ValGen is generally non-decreasing, NovelGen starts to decrease after epoch 30. Thus the snapshot selected according to the best ValGen is not the snapshot with the best possible meta-test performance. In fact, this loss of NovelGen performance due to choosing the best ValGen model instead of the actual best NovelGen model can be particularly large, with values being 1.1% % for SVM, 1. 1.2% for RR, 0.7% for PN, and 0.9% for FOMAML on the FC-100 dataset. These performances losses for each method are especially concerning considering the differences among the best possible NovelGen performance of these different methods are often smaller than 1.5%.
aaa.
7

---

Ranking similarity analysis. In light of the above observation, we ask a more general quantitative question: How similar is the ranking of the training snapshots using meta-val performance (ValGen) to the ranking using the meta-test performance (NovelGen)? To answer this, we compute the Kendall rank correlation coefficien p [21] between the ValGen and NovelGen rankings of algorithm snapshots trained on four OOD FSL. benchmarks (Figure and our ID) benchmark Zappos-IID III whose validation and test tasks come from the same distribution IP, A. More concretely, for each meta- learning method and dataset combination, we save the algorithm snapshots (one from each epoch) throughout meta-training and rank these algorithm snapshots according to their ValGen and NovelGen value respectively. Then p is computed between these two rankings for this specific method, dataset) combination. For snapshot selection through ValGen to work reliably, we need p to be close to 1. For context, we also compute for standard supervised image classification problem (IC), where train, val and test examples are sampled from the same example-level distribution.
Unreliabity of ValGen snapshot selection. From Figure 1), we see that when using validation samples generated iid from the test distribution (Zappos-II) ID) and supervised learning IC), the value of is consistently higher than the OOD benchmarks, indicating the validation performance can more reliably track the trend of test performance in the ID setting than on the OOD FSL benchmarks. In particular, for the cifar-M and FC-M datasets, the ValGen ranking of algorithm snapshots seems to be only weakly correlated with the true meta-test ranking for all the meta- meta-learning methods. In fact, the meta-val and meta-test rankings of the most useful snapshots can sometimes even be negatively correlated (p Rs -0.12 < 0 over all snapshots after epoch 30 in the training scenario shown in Figure 2a)). These results show that on the OOD FSL benchmarks, snapshot selection using the pre-assigned validation tasks can sometimes be unreliable/unable to identify a snapshot candidate with top-tier meta-test performance among all the snapshots.
5.1.2 Option 2: Snapshot selection alternative using BaseGen.
Beyond using OOD validation tasks for snapshot selection, inspired by the domain generalization community 23 18] we can alternatively also consider using the IDD performance over the training task distribution for snapshot selection. Perhaps due to the lack of an ID evaluation setup in common FSL benchmarks, this possibility has not been widely considered. Enabled by modifications of current FSL benchmarks, we can now evaluate the ID generalization performance (BaseGen) throughout training in addition to ValGen.
We plot how BaseGen and NovelGen progress for meta-learning methods trained on two different datasets in (d)). Here we see that on Miered-M, the BaseGen and NovelGen of PN both increase fairly consistently; thus picking the algorithm snapshot with the highest BaseGen performance (roughly the end-of-training snapshot) would also give close-to-best NovelGen. However, cifar-M, after the learning rate drop at epoch 20, SVM's BaseGen keeps improving while NovelGen starts deteriorating. In this case, selecting snapshots according to the best BaseGen would pick much worse snapshot than picking according to the best ValGen. (For concrete numbers of how much snapshot selection through BaseGen vs. ValGen could impact the chosen snapshot's 's NovelGen performance in each of these two cases, see Appendix F.3 This ambiguity of whether ID or OOD Validation snapshot selection is better has also been documented in domain generalization, where Gulrajani and Lopez-Paz [18 find ID) model selection can perform better in some settings while Koh et al. (23 find OOD validation model selection is better in others. Despite this ambiguity, we believe the commonly neglected in-distribution (BaseGen) snapshot selection approach should be considered by users of OOD FSL benchmarks as a viable alternative to the default ValGen selection approach in proper settings.
5.2 Inconsistencies in Meta-learning Method Performance Comparisons
After discussing concerns regarding OOD model selection for a given mela-learning method, we now analyze the reliability and consistency of conclusions drawn from comparing different meta-learning methods OOD performance these benchmarks. In particular, we focus on two cases:
Inconsistency example I: Limited number of novel classes in a single benchmark. Since in OOD FSL we specifically care about the learned algorithms' ability to quickly learn many unseen concepts, we should not be satisfied with an algorithm performing well only on tasks constructed from a small number of pre-selected novel classes. However, for many widely-used FSL benchmarks
P 11, e means correlation, while p 1/p -1 means exactly same/opposite rankings.
8

---

1Cal
[0x1
C2
CF
1F.0 5%
Dunaset Methods
ZAPPOS OOD
mubal ODD
(IC)
0.5%
100 imgsiclass
3%
1%
2w
2w:5
5w.5
172
Randar 23 dleveeeeeoy
dians dosett is So.
(1)
0.5%
20
120
15%
25%
PN
(9)80.51
(R) 74.67
(3) 76.22
(8)
0.5%
20
552
20%
24%
20.135%
20.14%
20.14%
(ii)
1.0%
64
20
552
10%
12%
Ridge
(3) 80.51
(2) 73.50
(3) 77.20
0.5
0.0
0.0
(10)
(V)
0.5%
0.5%
64
160
20
552
552
20%
0%
23%
7%
SVM
(8) 79.70
10.15%%
+00.18%
(8) 72.84
20.14%
+0.1445
73777.2
10.15%
10.10%
0.5
(v)
0.5%
351
20
552
9%
15%
74.82
(4)
76.96
030
[Vi8)
0.5%
351
160
552
0%
1%
MAML
#-0.14%
+00100%
#0.17%
(h)
Figure 3: In (a), we show the CDF plot of As, (Cw) - As, (Cx) over 100 randomly chosen sets of C.v with 20 (out of total 120)= novel classes each. In (b), for different values of true performance difference e and values of underlying class size [Cz]. training class size ((Cp)). and evaluation class size (CCx )): show the percentage of conclusion flips (CF) and improvement exaggerations (IE) with 5 0.5% computed over 100 evaluations. In (c), we demonstrate the inconsistencies in performance rankings for PN, SVM, RR, and FOMAMI. on two OOD benchmarks: Zappos-OOD and mini-OOD.
(minilmageNet, CIFAR-FS, FC-100), only 20 novel classes are used for meta- meta-testing. Ideally, even we don't it measure it, we would hope that our learned algorithm would also generalize to tasks made by other sets of classes different from the fixed small set of novel classes.
Formal setup. We suppose the existence of a much larger collection of classes Ct., where the novel classes CN used for meta-testing is a small random subset with each element class sampled uniformly and non-repeatedly from CL and fixed thereafter for NovelGen evaluation. Ideally, our goal is to evaluate an algorithm snapshot 6 on the task distribution Pcc, (denote this performance by A,(C))). yet during evaluation we only have access to the novel classes in Cy and thus we can only compute the performance Ag(Cx). It is easy to see that when we randomize over different choices of Cv. the expected performance over the sampled novel classes would match the true performance: Bcv [Ag(x)]
(Cz)) However, when using a single randomly sampled novel set, the estimator As(CN) can have high variance (see Appendix F. Instead of relying on Ap(Cx) to directly estimate (Cr.), we ask a more relaxed question: for a pair of algorithms As, and Ag, (given by two meta-learning methods), if the true performance Ag, (CL) (C,(CL) = >>>>> O. how frequently will we observe an opposite conclusion i.e., P(A,, (CN) < Ac,(C))) over a randomly sampled CN (we call this event conclusion flip)? Additionally, it is also possible that the observed performance difference on Cn is greater than the true difference e by some amount > O. In this case, the NovelGen observation would make the algorithm snapshot é1 look better than de more than it actually is on Cz. Thus we also ask what is the probability of IP(A,, (Cv) - A₂, (Cx) > c C+ 5) and we call such events improvement exaggerations. To answer both these questions empirically, we first suggest some larger class sets C, for mini and tiered. For both we select unused classes from ImageNet disjoint from the base and validation classes. For liered, we use all the remaining 1000 - 351 (base) 997 (novel) = 552 classes as Ct. while for mini, we randomly choose 120 or 552 (to match |Cy| in fiered) unused classes. We fix these Ct choices in the following analysis.
Checking the frequency of conclusion flips and exaggerations. Figure 3 (a) shows the empirical CDF of the performance differences As, (Cx) Asz (Cx) computed over 100 randomly sampled 2ize-20 novel class sets C. for fixed pair of PN and RR algorithm snapshots whose true performance difference over the larger 120 classes Cg is 0.5%. In 15% of the cases the performance order is flipped from the true order, while in 25% of them improvements are exaggerated by more than 0.5% (total difference greater than 1%). Moreover, for some of the performance order flips, the observed performance difference can be quite negative < -0.5% thus significantly opposite to the true performance order. (Here for each run we evaluate both methods 20,000 tasks sampled from Pc, in order to significantly reduce the randomness in estimating the true As, (CN), Agz (Cx).)
Comparison to supervised learning. We check for the conclusion flip and improvement exaggeration frequency when only a random subset of the full test set (100 randomly drawn test images from each base class in mini-M) is used to compare two supervised learning image classification models with the same full test set performance difference of c = 0.5 (row (IC) in Table b)). Here we that compared to supervised learning, the chances of getting an incorrect performance comparison (row (i) in Table is much higher for the meta-learning OOD FSL benchmarks when evaluating only on 20 randomly chosen novel classes (as done in several FSL benchmarks).
Larger |Cr, makes even less reliable but larger e helps. If we were to care about an even larger set of underlying classes (C2rl = 552) despite still using only 20 random novel classes for evaluation
9

---

comparison, the conclusions are even less reliable (Table[[]b (i) vs (ii). On the other hand, we do see that the performance comparison becomes comparatively more consistent if the true performance difference is higher (1% in (iii) compared to 0.5% in (ii)), despite that there still exists statistically significant chance (10%) of getting an opposite conclusion.
OOD evaluations in current FSL benchmarks. In practice, because 1) we never specify exactly what and how big the underlying set of classes that we care about is, and 2) some of the recent meta-learning methods (SVM vs PN on cifar in Table 2 of [25] R2-D2 vs GNN on mini in Table I of 4.. FIX- MLL (38 sometimes only improve over the prior works by < 1%, we believe researchers should be aware of the possibility of getting a performance conclusion that is inconsistent over a single randomly chosen and fixed set of 20 novel classes used by some of these benchmarks.
Actionable suggestions. Since the size of the unknown underlying larger class set Ct. and the true performance difference might not be something one can directly control when designing the OOD benchmark, we now discuss two actionable choices that can reduce the chances of conclusion flips:
Use more novel classes in the evaluation: By comparing (iv) vs (v) and (vi) vs (vii) in Table 3 we see that the frequency of conclusion flips and improvement exaggerations are much lower when 160 novel classes are used as opposed to 20 when |Cz] is the same.
ii) Train on more base classes: The tiered dataset has more base classes (351 compared to 64 for mini) to train on. When comparing PN and RR snapshots trained on modified version of tiered with fewer (randomly sampled 64 out of 351 to match mini) base classes, we see that the CF frequency is twice as high compared when 351 base classes are used (Table (abbleBh))ii (iv(v) vs (vi)).
Based on these two trends, for more reliable comparisons of meta-learning methods' OOD perfor- mance we suggest using datasets like magred and MetaDataset (both with much larger set of base and novel classes) in addition to the smaller benchmarks like minilmageNet, CIFAR-FS, and FC-100, which some recent works [e.g.. 30 still solely rely upon.
Inconsistency example 2: Inconsistency across multiple OOD FSL benehmarks. Unlike the ID scenario where the training and test task distribution are the same, the similarity between training and test distributions in the OOD FSL benchmarks can vary significantly. Ideally, we want a meta- learming method to be consistently better on multiple OOD benchmarks with different type/degree of distribution shifts. Since Ren et al. (35) originally use the Zappos dataset for OOD evaluation, we also perform similar evaluation on new attribute pairs based on their setup. At test time, we use an attribute set A' disjoint from the one used in the Zappos-IID setup A, and sample attribute pairs from A' only. This induces a test task distribution A different from the training task distribution P.A- We evaluate different meta-learning methods on these Zappos-OOf OOD tasks to see if the performance order is consistent with other OOD FSL benchmarks (Table 3())). Here we see that despite SVM outperforming RR and PN on mini NovelGen, the performance order of these three methods are completely flipped on Zappos-OOD Similar observations can be made from TADAM underper- forming PN in Table 2 of [35] despite TADAM being shown to outperform PN on the other more commonly FSL benchmarks. This inconsistency over different rypes of OOD FSL benchmarks is in stark contrast to the consistency of performance rankings over the 6 different II) benchmarks (FEMNIST, Zappos-II,, and the BaseGinn results of the 4 current FSL benchmarks (Section 4). Based on these findings, we caution meta-learning researchers to be aware of such conclusion inconsistencies over different OOD FSL scenarios and reason carefully about the generality of their empirical findings when using only specific types of OOD datasets.
6 Conclusion
In this paper, we categorize meta few-shot learning evaluation into two settings: in-distribution (ID) and out-of-distribution (OOD). After explaining why common FSL benchmarks reflect OOD evaluation, we identify realistic needs for ID FSL evaluation and provide new benchmarks as well as suggestions on how to modify existing OOD FSL benchmarks to allow for ID evaluation. Through experiments performed on these ID benchmarks, we demonstrate surprising phenomenon that many meta-learning methods/training techniques improve OOD performance while sacrificing ID performance. Beyond this, through quantitative analyses, we show that even in the OOD scenario, current FSL benchmarks may present subtle challenges with both model selection for a given meta- learning method and reliable performance comparisons of different methods. For these concerns, we provide initial suggestions and alternatives with the hope of alleviating these issues. Overall, we aim to raise awareness about the dichotomy of FSI. evaluation and to motivate the meta-learning community to collectively reason about ways to improve both ID and OOD methodology and evaluation.
10

---

This work was supported in part by the National Science Foundation Grant IIS IIS1838017, a Google Faculty Award, Facebook Faculty Award, and the CONIX Research Center, Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the NSF or any other funding agency.
Acknowledgements.
References
[1] M. Al-Shedivat, L., Li, E. Xing, and A. Talwalkar. On data efficiency of meta-learning. In International Conference on Artificial Intelligence and Statistics, pages 1369-1377. PMLR, 2021.
[2] S. M. Arnold and Sha. Embedding adaptation is still needed for few-shot learning. arXiv preprint arXiv:2 10:.02204.07555 2021.
[3] Y. Bai, Chen, P. Zhou, T. Zhao, Lee, S. Kakade, H. Wang, and C. Xiong. How important is the train-validation split in meta-learning? arXiv preprint arXiv:2010.05843 2020.
[4] L. Bertinetto, J. F. Henriques, P. H. Torr, and A. Vedaldi. Meta-learning with differentiable closed-form solvers. In International Conference on Learning Representations, 2019.
[5] G. Blanchard, G. Lee, and C. Scott. Generalizing from several related classification tasks to a new unlabeled sample. Advances in neural information processing systems, 24:2178-2186, 2011.
[6] S. Caldas, S. K. Duddu, P. Wu, T. L., Konečný, H. B. McMahan, V. Smith, and A. Talwalkar. Leaf: A benchmark for federated settings. arXiv preprint arXiv:/812.01099 2018.
[7] F. Chen, M. Luo, Z. Dong, Z. Li, and X. He. Federated meta-learning with fast convergence and efficient communication. arXiv preprint 1802.0777 2018.
[8] J. Chen, X.-M. Wu, Y. Yi Q. LI, L.-M. Zhan, and F.-l. Chung. closer look at the training strategy for modern meta-leaming. Advances in Neural Information Processing Systems, 33, 2020.
[9] W.-Y. Chen, Y.-C. Liu, Z. Kira, Y.-.. F. Wang, and J.-B. Huang. A closer look at few-shot classification. International Conference on Learning Representations, 2019.
[10] Y. Chen, X. Wang, Z. Liu, H. Xu, and T. Darrell. new meta-baseline for few-shot learning. arXiv preprint arXiv:2003.0439) 2020.
[11] S. S. Du, Hu, M. Kakade, D. Lee, and Q Lei. Few-shot learning via learning the representation, provably. arXiv preprint arXiv:2002.09434 094344 2020.
[12] A. Fallah, A. Mokhtari, and A. Ozdaglar. Personalized federated learning with theoretical guar- antees: A model-agnostic meta-learning approach. Advances in Neural Information Processing Systems, 33, 2020.
[13] A. Fallah, A. Mokhtari, and Ozdaglar. Generalization of model-agnostic meta- algorithms: Recurring and unseen tasks. arXiv preprint arXiv:2102.03882. 2021.
[14] C. Finn and S. Levine. Meta-learning and universality: Deep representations and gradient descent can approximate any learning algorithm. In International Conference on Learning Representations, 2018. URL URL httpppt///ppeereeeieee http://wwww....ettoooooo/t/oelaa
[15] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference Machine Learning, pages 1126-1135, 2017.
[16] C. Finn, K. Xu, and S. Levine. Probabilistic model-agnostic meta-learning. In Advances in Neural Information Processing Systems, pages 9516 2018.
[17] S. Gidaris and N. Komodakis. Dynamic few-shot visual learning without forgetting. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pages 4367-4375, 2018
11

---

[18] I. Guirajani and D. Lapez-Paz. In search of lost domain generalization. arXiv preprint arXiv:2007.01434, 2020.
[19] Hospedales, A. Antoniou, P. Micaelli, and A. Storkey. Meta-learning neural networks: A survey. arXiv preprint arXiv:2004.05449) 2020.
[20] Y. Jiang, J. Konečný, K. Rush, and S. Kannan. Improving federated learning personalization
via model agnostic meta learning. arXiv preprint arXiv:1909. 12488, 2019.
[21] M. G. Kendall. Rank correlation methods. Griffin, London, 3d ed. edition, 1962.
[22] M. Khodak, M.-F. F. Balcan, and A. S. Talwalkar. Adaptive gradient- meta-learning methods. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019. URL htsp::ttp///rooceddinng (4aa0dd66622e0455222222121 (tw0:6622164499--999999 pdf
[23] P. W. Koh, S. Sagawa, H. Marklund, S. M. Xie, M. Zhang, A. Balsubramani, W. Hu, M. Ya- sunaga, R. L. Phillips, I. Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. arXiv preprint arXiv:2012.07421, 2020.
[24] H. B. Lee, H. Lee, D. Na, S. Kim, M. Park, E. Yang, and S. J. Hwang. Learning to balance: Bayesian meta- meta-learning for imbalanced and out-of-distribution tasks. International Conference on Learning Representations, 2020.
[25] K. Lee, S. Maji, A. Ravichandran, and S. Soatto. Meta-learning with differentiable convex opti- mization. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 10657-10665, 2019.
[26] T. Li, A. K. Sahu, A. Talwalkar, and V. Smith. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 37(3)500-60, 2020.
[27] S. Lin, G. Yang. and J. Zhang. A collaborative learning framework via federated meta- learning. In 2020 IEEE 40th International Conference on Distributed Computing Systems (ICDCS), pages 289-299, Los Alamitos, CA, USA, dec 2020. IEEE Computer Society. doi: 10.1 11119//CDCS5777742200000000 1109/ICDCS47774 2001477777722000.000
URL htLLstts:://oo ieeecompttersocietty org/10
[28] H. B. McMahan et al. Advances and open problems in federated learning. Foundations and Trends@ in Machine Learning, 14(1), 2021.
[29] K. Muandet, D. Balduzzi, and B. Schölkopf. Domain generalization via invariant feature representation. In International Conference on Machine Learning. pages 10-18. PMLR, 2013.
[30] B. N. Oreshkin, Rodriguez, and A. Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. In Advances in Neural Information Processing Systems, 2018.
[31] A. Rajeswaran, C. Finn, S. M. Kakade, and S. Levine. Meta-learning with implicit gradients. In Advances in Neural Information Processing Systems, pages 113-124, 2019.
[32] S. Ravi and H. Larochelle. Optimization model for few-shot learning. In International Conference on Learning Representations, 2017.
[33] M. Ren, E. Triantafillou, Ravi, Snell, K. Swersky, J. B. Tenenbaum, H. Larochelle, and R. S. Zemel. Meta-learning for semi-supervised few-shot classification. arXiv preprint arXiv:1803. 00676, 2018
[34] M. Ren, R. Liao, Fetaya, and R. Zemel. Incremental few-shot learning with attention attractor networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox. and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL httph:tt///// http::////ooccceee.ee/ee www.eeeewww.nn.nnnseeee....em www..ipp..cppppee////00/.. ffi1e/88330022000996122222626
Paper pdf
[35] M. Ren, E. Triantafillou, K.-C Wang, Lucas, Snell, X. Pitkow, S. Tolias, and R. Zemel. Flexible few-shot learning with contextual similarity. arXiv preprint arxiv:2012.05895, 2020.
12

---

[36] A. Santoro, S. Bartunov, M. Botvinick, D. Wierstra, and T. Lillicrap. Meta-learning with memory- neural networks. In International conference on machine learning, pages 1842-1850. PMIR, 2016.
[37] J. Schmidhuber. Evolationary principles in self-referential learning, learning how to learn: the meta-meta... hook. PhD thesis, Technische Universität München, 1987.
[38] A. Setlur, O. Li, and V. Smith. Is support set diversity necessary for meta-learning? arXiv preprint 1rXiv:2011.14045 2020.
[39] Snell, K. Swersky, and R. Zemel. Prototypical networks for few-shot learning. In Advances in neural information processing systems, pages 4077-4087, 2017.
[40] E. Triantafillou, T. Zhu, V. Dumoulin, P. Lamblin, U. Evci, K. Xu, R. Goroshin, C. Gelada, K. Swersky, P.-A. Manzagol, et al. Meta-dataset: A dataset of datasets for learning learn from few examples. In International Conference on Learning Representations, 2020.
[41] E. Triantafillou, H. Larochelle, R. Zemel, and V. Dumoulin. Learning universal template for sew-shot dataset generalization. In International Conference on Machine Learning. 2021.
[42] O. Vinyals, C. Blundell, Lillicrap. D. Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems, pages 3630-3638, 2016.
[43] A. Yu and K. Grauman. Fine-grained visual comparisons with local learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 192-199, 2014.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)?
[Yes] We include the code and instructions to setup the dataset and run experiments in the github link in the footnote on the first page of the paper. We also provide link to download the data.
(b) Did you specify all the training details (c.g., data splits, hyperparameters, how they were chosen)? [Yes] We document these in the Appendix.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)?
[Yes] We provide error bars for the estimated performance over each task distribution.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Details Appendix.
4. If you using existing assets (e.g., code, data, models) curating/releusing assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We use public datasets and cite the creators.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material or as URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "qFcT-g-vENd", "reviewer": "Reviewer_Qp31", "summary": "This paper proposes to distinguish two types of evaluation settings for meta-learning algorithms: in-distribution (ID) and out-of-distribution (OOD). They claim that using few-shot learning benchmarks as a test bed for meta-learning algorithms leads to developing models for and studying only OOD, whereas other real-world settings require ID generalization, and the developed models may not work well there out-of-the-box. They investigate the trade-offs between ID and OOD performance of different models by creating a set of ID tasks and demonstrating different rankings between models on ID versus OOD. They also discuss other challenging facets of the evaluation protocol for meta-learning algorithms, including how model selection is performed and the reliability of conclusions drawn when an insufficient number of classes are used to create test tasks.\n", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "High-level review\n-----------------------\nThe paper is well-written and easy to understand for the most part. The discussion around ID and OOD tasks is interesting and alerting the community that models developed for one might not work optimally for the other is a useful insight and a nice contribution. The paper also contributes thorough empirical analyses on various facets of meta-learning evaluation that are interesting. However, I have a number of concerns about the paper, summarized here (with more details in the section below). First, despite performance differences and differences in ranks between models in ID and OOD pointed out by the authors, these differences are small. I think a more thorough evaluation is needed to establish the performance differences of various models in these two settings. Along the same lines, the proposed ID setting is very similar to some related work (“low-shot” or “incremental” learning) which is cited but without comparing and consolidating insights with the findings of that community (e.g. about the nature of methods that work well in one setting versus the other) which would have been very useful. Finally, some topics investigated seem inconclusive and I’m unsure what the take-away should be (e.g. the model selection experiments) while others, albeit interesting, don’t seem particularly surprising, like the unreliability of performance comparisons between methods on benchmarks with small meta-test sets. Indeed recent benchmarks have moved towards larger-scale and more diverse test sets [2,3,4,8]. For these reasons at this stage I am recommending weak rejection despite the merits of the paper (more details and suggestions below).\n\nDetailed comments\n--------------------------\n- [General] The narrative unnecessarily revolves around *meta-learning* evaluation / *meta-learning* methods specifically. The episodic evaluation protocols explored in this paper also can be used more generally for transfer learning approaches (and in fact some of the experiments include these models too). Perhaps using the term \"few-shot learning evaluation\" or \"episodic evaluation\" more appropriately captures the more general scope.\n\n- [proof-by-contradiction] I don’t think the assumption about the lower bound on the number of training tasks (lines 135-136) is realistic. There could be enough tasks for training and yet still there could be a really large set of novel classes, so that the overall size of C_L is much bigger than the number of training classes, e.g. in Meta-Dataset when training on ImageNet only and evaluating on classes from a large number of other datasets.\n\n- [ID vs OOD results] In line 212, the authors mention that the performance order of different models is consistent on ID benchmarks yet “almost completely opposite” on OOD ones. It should be noted though that in both settings the performance of these methods is really similar to each other (with the exception of FO-MAML which underperforms the rest consistently in both settings). This makes me unsure if the trend is significant, and I wonder, for instance, if different model selection techniques would eliminate this discrepancy. What was the criterion used for model selection in these experiments? Using a validation set of held-out *classes* is probably more appropriate for model selection for OOD evaluation, while using one of held-out images of *seen classes* is probably most appropriate for ID.\n\n- [Ranking similarity analysis] I'm not sure that comparing the ranking is a good metric as it doesn't take into account the actual performance difference of the snapshots. For example, if one snapshot performs similarly to another, then their ordering in the ranked list does not really matter. Perhaps a more appropriate metric is a weighted average of the snapshots’ performance, with weights determined as a function of the ranks?\n\nModifying OOD FSL benchmarks for ID\n-----------------------------------------------------\n- Note that the modifications proposed for this are the same as those used in “low-shot” benchmarks [1]. Why not just re-use those benchmarks, which would also allow a broader comparison with few-shot learning models defined for that context?\n\n- It should be noted that these ID benchmarks no longer reflect few-shot learning problems, since the classes used in test tasks are the same as those used for training. Therefore it might not be unexpected or surprising if a model developed for few-shot learning does not work well for a different problem. Though I agree it’s interesting to identify whether this is the case.\n\n- A more compelling ID setting in my opinion (which actually can be thought of as few-shot learning still in some sense, as opposed to the proposed setting) is to first construct all possible N-way tasks (N-tuples of classes) and split those into train / validation / test tasks, following the “randomized tasks partition” scheme mentioned earlier. This modified variant can still be thought of as an ID \"few-shot learning\" setting: each given test task might contain classes that were seen at training, but those classes weren’t seen *together* (i.e. in the same task) during training.\n\n“Challenges with Evaluation”\n--------------------------------------\n- The model selection analysis seems inconclusive. What is the take-away in terms of when to use one approach versus the other? As a general rule of thumb, does using a validation set of held-out classes work better as a model selection technique for OOD while a validation set of held-out examples of seen classes for ID?\n\n- [Limited number of novel classes] I agree with the argument made about inconclusive results when a small number of test classes are considered. However, in addition to the *size*, I would argue that the *diversity* of the test set really matters to broadly evaluate different methods OOD. Even if we can reliably measure the performance difference on a specific (large enough) set of test classes, this doesn't necessarily speak to a model's broad generalization ability if the chosen set of test classes are similar to each other. Running an additional experiment to investigate this aspect would be really interesting too. This would present an argument in favour of using benchmarks with diverse sets of test classes like Meta-Dataset or VTAB [2,3,4,8]. \n\n- [Consistency across OOD FSL benchmarks] The authors point out the difference in relative performance of models between mini-OOD and Zappos-OOD from [5]. I would argue though that the setup in [5] reflects a different problem, not just a different benchmark. Specifically, they define a scenario where instead of having rigid class groupings that depend on the object classes (e.g. cats vs dogs), they define classes *flexibly* based on some hidden context (given by conjunctions of attributes). Therefore, images that belong to the same \"class\" in one episode might actually belong to different classes in a different episode. This requires more flexibility and is a more challenging problem compared to standard few-shot classification. So to me, this experiment of checking for (in)consistency across FSL benchmarks would be more convincing if performed on different benchmarks for the same problem setting, e.g. standard few-shot classification. Still, it should be noted that SVM slightly outperforms RR and PN in one case and slightly underperforms on the other, so this is not a drastic difference either way. \n\nRelated work\n------------------\n- [6] makes a similar separation of the test tasks of Meta-Dataset into ‘weak generalization’ and ‘strong generalization’ sets, where the former introduces held-out classes at test time from *seen* datasets (some other classes of which were used for training), whereas the latter introduces classes of held-out *datasets* altogether (none of their classes used for training). These can also be thought of as a form of ID versus OOD evaluation, and they also show trade-offs between models performing better on one setting but worse on another. It would be useful to mention this connection.\n\n- [7] creates training and test tasks automatically via a clustering framework that allows to control the degree of difficulty (similarity between the resulting sets of training and test classes out of the overall available set of classes). They also show that models’ performance is “flipped” for their “easy” and “hard” settings. Perhaps their framework can be thought of as constructing a spectrum between ID and OOD evaluation.\n\nMinor comments\n-----------------------\n- In Figure 2c, why is there no curve for ValGen like in the remaining sub-figures of Figure 2?\n- RE: Table 1 and lines 206-209, ProtoMAML [2] can be used instead of MAML for handling varying numbers of classes\n- Line 126: “mini” → “miniImageNet” ?\n- Line 128: “have lead” → “have led”\n- Line 392: “set set” → “test set”\n\nReferences\n----------------\n- [1] Dynamic few-shot visual learning without forgetting. Gidaris and Komodakis. CVPR 2018.\n- [2] Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples. Triantafillou, Zhu, Dumoulin, Lamblin, Evci, Xu, Goroshin, Gelada, Swersky, Manzagol and Larochelle. ICLR 2020.\n- [3] A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark. Zhai*, Puigcerver*, Kolesnikov*, Ruyssen, Riquelme, Lucic, Djolonga, Pinto, Neumann, Dosovitskiy, Beyer, Bachem, Tschannen, Michalski, Bousquet, Gelly and Houlsby. 2020.\n- [4] Comparing Transfer and Meta Learning Approaches on a Unified Few-Shot Classification Benchmark. Dumoulin*, Houlsby*, Evci, Zhai, Goroshin, Gelly and Larochelle. 2021.\n- [5] Flexible few-shot learning with contextual similarity. Ren*, Triantafillou*, Wang*, Lucas*, Snell, Pitkow, Tolias, and Zemel. 2020.\n- [6] Learning a Universal Template for Few-shot Dataset Generalization. Triantafillou, Larochelle, Zemel and Dumoulin. ICML 2021.\n- [7] Embedding Adaptation is Still Needed for Few-Shot Learning. Arnold and Sha. 2021.\n- [8] A Broader Study of Cross-Domain Few-Shot Learning. Guo∗, Codella∗, Karlinsky, Codella, Smith, Saenko, Rosing, and Feris. 2020.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have not addressed these aspects. Even if not directly applicable to this paper specifically, I would encourage them to think about the potential negative impacts of meta-learning and few-shot learning works more broadly.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "jAGfHl3-dqT", "reviewer": "Reviewer_rpPm", "summary": "- They explain that in standard meta-learning datasets the evaluation is OOD - the test tasks are not sampled from the same distribution as train tasks. They explain that ID meta-learning is important too.\n- They consider 2 ID meta-learning datasets and 1 OOD meta-learning dataset (Table 1). They show the ranks of algorithms on the ID datasets is flipped from the OOD dataset. On the other hand, the 2 ID datasets are consistent.\n- They show that the ranks of algorithms are flipped between zappos-OOD and mini-ImageNet-OOD, suggesting evaluation on OOD datasets can have high variance\n- They show that the training tasks, validation tasks, and test tasks, can follow different curves during training time, so using each of these for model selection would give a different model\n- They explain that meta-learning datasets often have very few validation tasks, which also leads to high variance in the results\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nPost rebuttal: updated score to 6 (borderline but leaning a bit towards accept), comments below. I'm starting to appreciate the overall message, and it seems useful - I think with better writing (removing less significant pieces, focusing more on the core message) this could be a strong paper.\n\n---------------------------------------------------------------------------------------\n\nStrengths\n- It's nice that they examine many different types of inconsistencies: ID vs OOD, difference between OOD datasets, different methods for model selection, inconsistencies due to small number of tasks in the validation set\n- I agree with their general conclusion: OOD is a very broad term and can capture all sorts of distribution shifts. It's difficult to get a method that performs well across OOD datasets, so we need to focus on more structured shifts, for example ID meta-learning with a large number of classes\n\nWeaknesses\n- As a primarily empirical paper, where the main message is about the inconsistency of OOD meta-learning datasets, it would be good to have more comprehensive evaluations, that is more datasets and methods\n- A key argument is that ID meta-learning datasets rank methods more consistently than OOD meta-learning datasets. I'd like to see more datasets to see if this is really the case.\n- They say that model ordering can completely flip across different OOD datasets. \nOne note: the accuracies of these methods are very close. It's natural that when using different datasets you'd get some variation, e.g. by 1%. For example, in table 1, the difference between PN and Ridge is 0.01%. For FO-MAML, where the effect size is larger, the ordering is consistent across datasets.\n- Correlation between valgen selection and testgen selection: they showed that the rank correlation coefficient for snapshots on meta-learning datasets is much lower than for supervised learning, and is generally lower than for zappos-id, their ID meta-learning dataset.\n- I think some concrete numbers could help me. For each dataset, if you select the best model using val gen for each method, vs select the best model using base gen and test gen, how different are the accuracies on test gen? \n- How were hyperparameters tuned for each method? The appendix shows hyperparameter values for each method, but ideally we'd run a grid search and choose the best hyperparameters for each model. Particularly when the accuracy numbers are so close, this could change the results a lot.\n- Their actionable insight 1 on page 9, of using more novel classes, is certainly true if there are more novel classes available.\n- Actionable insight 2 is interesting - it's not clear to me why training on more base classes decreases the test variance. This could be examined more. Is it because when training on more classes, there is a larger separation between methods, so we need fewer test tasks to rank them? Is it specific to this one dataset (tiered ImageNet), or more general?\n\nI think the paper has useful things to say, and could be compelling with a more thorough investigation of its findings. The paper would be more compelling if they had a couple more ID datasets, and methods, ideally one or two more OOD datasets, and if they grid search over their hyperparameters or run ablations on them.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Sounds good!", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4+2 hours"}, "extra_text": {}}, {"review_id": "RGp4KZfi0Y5", "reviewer": "Reviewer_qFSe", "summary": "This paper analyzes current meta-learning evaluations and finds that there is a significant discrepancy between how they are often theoretically and experimentally studied. They find that most meta-learning benchmarks are constructed with OOD test sets, whereas theory focuses primarily on ID test sets. The authors conduct numerous evaluation to understand the impact of this design oversight.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main strength of this paper is pointing out the difference between OOD and ID test sets for meta-learning, and the following experiments the authors ran on the FSL benchmarks to understand the empirical impact of this difference.\n\nOf particular importance are the ranking experiments the authors run, which finds that methods that work on ID benchmarks can in fact be the worst for OOD benchmarks. This finding is very important for practitioners in this area to be aware of what setting their application is in.\n\nI also liked the experiment where the authors converted existing OOD benchmarks into ID benchmarks and re-ran existing methods on the converted datasets. The high degree of ranking flips is concerning and is another point to show where the theory and experiments diverge for FSL.\n\nThe writing exposition was very clear, except that there is a considerable amount of bold and italics throughout the text which hinders reading flow (nit).\n\nI vote to accept its paper based on its clear and practically relevant insight which is much needed for the community, as well as the strong supporting experiments.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors did not address limitations and societal impact, which I agree with. They are pointing out flaws in the current evaluation methodology.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "BTOo-XhxWND", "reviewer": "Reviewer_mSBf", "summary": "This paper claims that few shot learning / meta learning benchmarks are testing a form of OOD generalization and are not testing generalization to IID sampled tasks from a task distribution. The main observation is that most benchmarks train and test on tasks with disjoint sets of classes, which is improbable if the label spaces for each task are sampled IID. The authors convert benchmarks into a \"ID\" setting by re-sampling the dataset and show that there is a tradeoff in performance between the settings. In particular, they find that methods that perform well on the benchmark (which is testing OOD generalization) may perform worse in the ID setting, and they note that there are some challenges when doing model selection in the OOD setup.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Quality/Originality\n\nThe discussion of ID/OOD evaluation in FSL datasets is insightful, and the authors took care to hold out examples in each class for ID evaluation (now that classes can appear in both train and test). The same model backbone is used throughout for comparison between methods. However, main concerns are that the comparisons don't seem to be run on all the datasets considered, and the picture would be more complete with these, and the conclusions given throughout the paper are somewhat inconsistent. Clarity on these would improve my score.\n\n- The observations could be bolstered by running more datasets using the method in Figure 1a,b, where each dataset is converted into ID/OOD evaluations and the numbers are compared. For example, in Figure 1d, there is still inconsistency but only for 2 of 5 methods. What happens if you convert Zappos and FEMNIST to OOD evaluations as well? The tables comparing the rankings across datasets are valuable but less of a controlled setting. \n- Similarly, in line 427, the paper claims a contrast between the results in Figure3c and Table 1. However, we only have one example of BaseGen comparisons between datasets and within datasets in Table 1, as well as one comparison between BaseGen and OOD in Figure 3c, when it seems that we should be able to make these comparisons between all the datasets, given that they can all be converted into ID/OOD versions.\n- The message between Section 4 and 5 are somewhat contradictory at times. For example, Section 4 describes an ID/OOD conflict, but Section 5 says that using BaseGen(ID) as a model selection measure for NovelGen(OOD) should be considered. \n- The paper says that current meta learning methods are worse in the ID version of the dataset than the OOD (normal) version. Can the authors explain why methods developed using theoretical/conceptual insights from the ID setting are worse in the ID setting than OOD? Even though there could be an effect from tuning the models, at a fundamental level shouldn't ID be easier and match the statistical nature of the methods better? Some discussion of this in the paper may be useful.\n- As stated in line 134, another possible explanation for disjoint classes between train and test tasks is that there is a much larger ambient set of classes. For consistency, suppose that the benchmarks just don't have enough training tasks for this ambient space of tasks. In this case, the ID theory/setup holds but the benchmarks are not totally fair, which may lead to inconsistent results. This explanation doesn't seem to be rejected by this paper? The remedies may be different in this case.\n- ID evaluation using BaseGen (accuracy on held-out examples from base classes that were sampled from for training) for meta-learning datasets with a small number of classes (such as mini) may not test generalization to any new classes, so this could exaggerate the difference between ID and OOD for these datasets. \n- The observation in L256 that higher learning rate is better for BaseGen while lower learning rates are better for OOD generalization (NovelGen) are counterintuitive since higher learning rate is like regularization. Is there any explanation for this? \n\n\n\nClarity \n\nThe explanation of the why benchmarks seem more like OOD evaluation is clear. Sections 4/5 were less clear in terms of organization - while Section 4 seems to be about ID evaluation, the main points are about ID vs OOD, which muddles the distinction between Sections 4 and 5. They both make statements about inconsistency and ID vs OOD, but sometimes (as stated above) the messages seem somewhat at odds, and muddles the picture.\n\n- Some common quantities seem to be renamed - for example, early stopping is renamed to snapshot selection. I think using the common terminology could make things clearer.\n- The definition of BaseGen/ValGen/NovelGen can be made clearer - at first glance, BaseGen sounds like training accuracy, but actually it's held out examples from the training classes.\n\n\nSignificance\n\nTypically, few-shot learning / meta-learning methods are developed with the assumption that tasks are sampled IID from some task distribution. This paper shows that this common assumption is not satisfied in the empirical setup of few-shot benchmarks, which may have implications on empirical evaluation of the methods. A potential outcome is to evaluate meta-learning/few-shot methods in both the original (OOD) setting and a modified ID setting. This would encourage methods that would work in both of these evaluation settings.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper does not address the limitations / social impacts of the work.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission discusses the properties of in-distribution (ID, i.e. same task distribution) and out-of-distribution (OOD, i.e. subject to a task distribution shift) meta-learning evaluation. It contends that current few-shot classification benchmarks reflect OOD performance, and that approaches which perform well in the OOD setting may not necessarily perform well in the ID setting. Finally, it highlights model selection pitfalls and issues with the consistency of meta-learning performance comparisons.

Reviewers found that the submission raises interesting questions, as evidenced by their engagement in the discussions. They agree that the discussion surrounding ID and OOD evaluation for few-shot classification is insightful and of value to the research community, especially since a lot of existing theoretical work makes an ID assumption. Other claimed contributions are less appealing to some reviewers:
- The model selection results appear inconclusive.
- The research community is already moving towards larger and more diverse benchmarks, which lessens the impact of the observation that small test sets are unreliable.
- The 6 common ID datasets and 4 common OOD datasets and the number of approaches investigated remains a small sample size to support the claims made in the paper according to some reviewers. 

Reviewers weigh these strengths and weaknesses differently, and resultingly their opinion is split on acceptance. I read the paper to form an independent opinion and think that it raises interesting questions, even if imperfectly. In any case, I don't believe any reviewer is _indifferent_ to it, which to me is a good sign that its publication would have an impact on the few-shot learning community. I therefore recommend acceptance.
Decision: Accept (Poster)