Two Sides of Meta-Learning Evaluation: In vs. Out of Distribution
Authors: Amrith Setlur
Oscar Li
Virginia Smith
Venue: NeurIPS 2021 Poster (2021)
Keywords: meta-learning
meta-learning evaluation
few-shot learning
out of distribution

Abstract: We categorize meta-learning evaluation into two settings: $\textit{in-distribution}$ [ID], in which the train and test tasks are sampled $\textit{iid}$ from the same underlying task distribution, and $\textit{out-of-distribution}$ [OOD], in which they are not. While most meta-learning theory and some FSL applications follow the ID setting, we identify that most existing few-shot classification benchmarks instead reflect OOD evaluation, as they use disjoint sets of train (base) and test (novel) classes for task generation. This discrepancy is problematic because -- as we show on numerous benchmarks -- meta-learning methods that perform better on existing OOD datasets may perform significantly worse in the ID setting. In addition, in the OOD setting, even though current FSL benchmarks seem befitting, our study highlights concerns in 1) reliably performing model selection for a given meta-learning method, and 2) consistently comparing the performance of different methods. To address these concerns, we provide suggestions on how to construct FSL benchmarks to allow for ID evaluation as well as more reliable OOD evaluation. Our work aims to inform the meta-learning community about the importance and distinction of ID vs. OOD evaluation, as well as the subtleties of OOD evaluation with current benchmarks.

Two Sides of Meta-Learning Evaluation: In vs. Out of Distribution
Amrith Setlur'
Oscar Li oscarliecmu, edu
Virginia Smith²
asetlurčcs cmu edu "Language Technologies Institute
smi thv edu Machhne Learning Department School of Computer Science, Carnegie Mellon University
Abstract
We categorize meta-learning evaluation into two settings: in-distribution [ID], in which the train and test tasks are sampled iid from the same underlying task distribution, and out-of- offiiiiiibbtii [OOD], in which they are not. While most meta- learning theory and some FSL applications follow the ID setting, we identify that most existing few-shot classification benchmarks instead reflect OOD evaluation, as they use disjoint sets train (base) and test (novel) classes for task generation. This discrepancy is problematic because-as we show on numerous benchmarks- meta-learning methods that perform better on existing OOD datasets may perform significantly worse in the ID setting. In addition, in the OOD setting, even though current FSL benchmarks seem befitting, our study highlights concerns in I) reliably performing model selection for given meta-learning method, and 2) consistently comparing the performance of different methods. To address these concerns, provide suggestions on how to construct FSL benchmarks to allow for ID evaluation as well as more reliable OOD evaluation. Our work aims to inform the meta-learning community about the importance and distinction of ID vs. OOD evaluation, as well as the subtleties of OOD evaluation with current benchmarks.
Introduction
Meta-learning considers learning algorithms that perform well over distribution of tasks 19 To do so, a meta-learning method first learns from a set of tasks sampled from a training task distribution (meta--rainingg) and then evaluates the quality of the learned algorithm using tasks from a test task distribution (meta-testing). The test task distribution can be the same as the training task distribution scenario we term in-distribution generalization evaluation or ID evaluation) or different task distribution (ont-of-distribution generalization evaluation or OOD evaluation).
37
In this work, argue that there is need to carefully consider current meta-learning practices in light of this ID vs. OOD categorization. In particular, meta-learning is commonly evaluated on sewt learning (FSL) benchmarks, which aim to evaluate meta-learning methods' ability to learn sample-efficient algorithms. Current benchmarks primarily focus on image classification and provide training tasks constructed from of train (base) classes that are completely disjoint and sometimes extremely different from the test (novel) classes used for test tasks. As we discuss in Section 3 this design choice imposes a natural shift in the train and test task distribution that makes current benchmarks reflective of OOD generalization. However, there are a number of reasons to also consider the distinct setting of ID evaluation. First, whether in terms of methodology or theory, many works motivate and analyze meta-learning under the assumption that train and test tasks are sampled iid from the same distribution (see Section 2}. Second, we identify growing number of applications, such as federated learning, where there is in fact need for sample-efficient algorithms
Authors contributed equally to this paper. "oode available at https://github.ee http://htwb.ooomma////tttttteeee -oo------ood
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

that can perform ID generalization. Crucially, we show across numerous benchmarks that methods that perform well OOD may perform significantly worse in ID settings. Our results highlight that is critical to clearly define which setting a researcher is targeting when developing new meta-learning methods, and we provide tools for modifying existing benchmarks to reflect both scenarios.
Beyond this, we also re-examine current OOD FSL benchmarks and analyze how the shift in the train and test task distributions may impact the reliability of OOD evaluations. We point out two concerns which we believe are not widely considered in the meta-learning community. First, unlike areas such as domain generalization where model selection challenges are more widely discussed [8] 23). we conduct to the best of our knowledge the first rigorous study demonstrating the difficulty of model selection due the shift in the validation and test task distributions in FSL benchmarks. Second, because the OOD scenario in meta-learning does not assume a specific test task distribution, there is room for different test distributions to used for evaluation. We show that comparing which meta-learning method performs better can be unreliable not only over different OOD FSL benchmarks, but also within a single benchmark depending on the number of novel classes.
Our main contributions are: i) We clearly outline both ID and OOD FSL evaluation scenarios and explain why most popular FSL benchmarks target OOD evaluation (Section ii) We provide realistic examples of the ID scenario and show that the performance of popular meta-learning methods can drastically differ in ID vs. OOD scenarios (Section 4. iii) For existing OOD FSL benchmarks, we highlight concerns with a) current model selection strategies for meta-learning methods, and b) the reliability of we suggest suitable modifications to the current FSL. benchmarks to allow for II) evaluation, and explain how to construct FSL benchmarks to provide more reliable OOD evaluation. Our hope in highlighting these evaluation concerns is for future researchers to consider them when evaluating newly proposed meta-learning methods or designing new FSL benchmarks.
method comparisons (Section (5). iv) To remedy these concerns.
2 Related Work
Current FSL benchmarks. A plethora of few-shot image classification benchmarks (e.g., mini- ImageNet (mini in short) 42), CIFAR-FS A) have been developed for FSL evaluation. These benchmarks typically provide three disjoint sets of classes (base, validation. novel) taken from standard classification datasets, e.g., ImageNet or CIFAR-100 [36 32 42]. Training, val, and test tasks are then constructed from these classes respectively, which, as we discuss in Section 3 can induce a shift in their corresponding task distributions. Distribution mismatch can be particularly large with non- non-random splits created at the super class level, e.g., FC- 30. or dataset level, e.g. Meta- Dataset [4).. Recently, Arnol and Sha (2) propose an automated approach to construct different class splits from the same dataset to allow for varying degrees of task distribution shifts; Triantafillou et al. [1 separate the distribution shifts on Meta-Dataset into weak vs. strong generalization depending on whether the novel classes are taken from the used training datasets or not. Both works find that the meta-learning methods that perform better in one distribution shift scenario might perform worse in another, providing further evidence to our OOD performance comparison inconsistency argument in Section 5.2 Beyond these canonical ways of task construction through set of classes, Ren et al. (35 propose new benchmarks in a new flexible few-shot learning (FFSL) setting, where the aim is to classify examples into a context instead of object class. During testing. they perform OOD evaluation on unseen contexts. Inspired by their approach, we also provide an FSL benchmark where tasks are specified by contexts (Section 4 though we differ by exploring ID evaluation.
Mismatch between meta-learning theory/methodooogy and evaluation. Despite theoretical works which analyze meta-learning OOD generalization 13), there are many theoretical meta-learning works [c.g.. I 3 8, 22 that first assume the train and test tasks are iid sampled from the same distribution despite validating their analyses
OOD FSL benchmarks. Additionally, several popular meta-learning methods 25 16 31 that are motivated in the ID scenario are largely evaluated on OOD benchmarks (see Appendix A.. Prior work of Lee et al. 24 explores ID vs. OOD; however they treat the FSL setup as ID when the disjoint base and novel classes are from the same dataset and OOD only when they not (e.g.. base from ImageNet, novel from CUB). We emphasize that even the disjointedness of base and novel from the same dataset can create a task distribution shift and hence unlike 24] we do consider current FSL benchmarks (like mini) to be ID (Section 3).
FSL. ID evaluation. We are unaware of any FSL classification benchmarks that are explicitly advertised for ID evaluation. As we discuss in Section 4 a growing number of works 7 20 22

---

use meta- meta-learning for personalized federated learning, but do not clearly discuss the in in-distribution nature of these benchmarks nor how they differ from standard FSL benchmarks. In recent work, Chen et al. [10] extend current FSL. benchmarks to evaluate their proposed method both ID) and OOD, but only to further improve OOD performance on the original FSL benchmark's novel classes. Our work uses similar setup for constructing ID) evaluations with current OOD FSL benchmarks, but focuses on showing that certain meta-learning methods/design choices can improve OOD performance at the cost of ID performance. Prior work [17 34 incremental few-shot/low-she learning also explores performance on both base and novel classes simultaneously. However, they differ in their methodology, as they use supervised learning (not learning) to classify over the base classes.
OOD evaluation in other fields. Train and test distribution shifts are also found in domain gener- alization [5] 29 where the goal is to find model that works well for different test environment. Due to this shift, Gulrajani and Lopez-Paz, [8] specifically discuss difficulties in performing model selection in domain generalization benchmarks. They argue that it is the responsibility of the method designer (not benchmark designer) to determine model selection strategies for their method, and propose several model selection methods, mainly targeting hyperparameter selection. Unlike domain generalization, FSL benchmarks often have a pre-determined disjoint set of validation classes to construct validation tasks, so the need for developing model selection methods may be less obvious. In Section 5 motivated by [88], we explore strategies for model selection for meta- weea--earning. However, in contrast to [18], we focus on algorithm snapshot selection for meta-learning, which is required by hyperparameter selection as a subroutine (exact definitions see Section 5).
3 FSL benchmarks: Background & Focus on OOD evaluation
Background and notation. In this work, we employ general definition of meta-learning FSL task: a task T is a distribution over the space of support and query dataset pairs (S, Q), where S, Q sets of examples from example space X' V. The support set S is used by algorithm to produce an adapted model which is then evaluated by the corresponding query set Q. Each time we interact with a task T, (S, Q) pair is sampled iid from T, the mechanism for which depends on the specific application; we provide multiple examples below. As discussed in Section meta-learning aims learn an algorithm over distribution of tasks P(T). During meta-training, we assume access to N pairs of {(S,, Q+) Hee[N] sampled from a training task distribution P (T)( in the following way: first, N tasks are sampled iid from the training task Ti ~ Ptt((T); then, for each task Te, a support query pair (S,, Q.) ~ T7 is sampled iid. During meta-testing,
assume there is test task distribution Pre (T) where fresh (S, Q) samples are similarly sampled based on Pte (T). We define a learning scenario to be in-distribution (ID) if Pt, It and out-o--distributinn (OOD) if irr o IP teee Whenever Pt. Ft.. the induced train and test marginal distributions of (S, Q) are also identical.
Construction of (S, Q) pairs in FSL. Most popular FSL benchmarks share similar structure: they provide three disjoint sets of classes: base classes CB. validation classes Cy, and novel classes CN. where any class in these sets specifies distribution P. over the example space X. An n-way k-shot q-query task To in these benchmarks is specified by a length n non-repeated class tuple (cI, Cn) where e E [c*] == ((d1,... dn) E C" dl # dj Vi # j]. Te generates random (S, pairs in the following way: For every class C,, ht support examples S: ~ (Pa,)* and q query examples Qi ~ (P,,)" )" are sampled. The support and query set is formed by the union of such labelled examples from each class: S = U._1 ((x, i), Vx E S.). Q ( i), Vr E Qi). By specifying the base and novel classes, the FSL benchmark has provided a collection of tasks for be extremely large. For example, 64'
training (Te c [CH]] and test [Te : e E [c These sets
in mini, which has 64 base classes, the total number of 5-way training tasks is
764-5) R 9.1 x 10
However, it is not explicitly specified what underlying task distribution P(T) generates these sets of tasks. We believe this may have led prior work to discuss FSL benchmarks in the context of ID evaluation [e.g.. 1 88, contrary to what we outline below.
Current FSL benchmarks target OOD evaluation. We now informally discuss our reasoning for arguing that current FSL benchmarks reflect OOD evaluation (we provide a more formal proof by contradiction in Appendix B). In particular, the training and test tasks in FSL benchmarks are indeed iid sampled from the same underlying task distribution, then this underlying distribution must be induced by a distribution over class tuples of an even larger class set C₁ (C, 2 (Cg U Cv Ux)).
'We note that "training task distribution" here refers to the true underlying distribution of training tasks, not the empirical distribution supported over the finite set of sampled training tasks.
3

---

We consider the following dichotomy:
when Cr.] (nnN): In this case, the total number of classes n.N covered by the sampled tasks (counting repetition) is greater than the number of underlying classes Then with high probability, both the training and test tasks would each cover significant portion of all the classes in CL. making it extremely unlikely to have an empty intersection as in the current FSL benchmarks. ii) when Ccz] = (2(NN): In this alternative case, the total number of classes (even counting repetitions) used by sampled tasks is still smaller than the number of underlying classes |Ct/-- Thus the sampled tasks cannot cover all the underlying classes. Under this regime, the number of classes covered by the training tasks alone would scale linearly with N, as repeating an already-seen class in a new task sample is relatively rare. Since FSL benchmarks typically use a large number of training tasks during meta- meta-training (N > 10"), it is improbable that all the training tasks would together only cover a very low number of classes (64 the case of mini).
Randomized class partitions do not imply randomized task partitions. Another issue that may cause some to view the current FSL benchmarks as performing ID evaluation is that in some of these benchmarks, the base, val, novel classes are random partitions of iid drawn classes from a class level distribution (specifically minilmageNet, CIFAR-FS; but not FC-100, mieredImageNet as the classes are not partitioned randomly). The logic here is that in standard machine learning practice, randomly partitioning iid sampled data points into train and test guarantees that the train and test samples are drawn itd from the same underlying distribution. However, it is important to notice that the first class citizen in common FSL benchmarks is not a class, but a task (represented by a class tuple). So. only a randomized partition of iid sampled class tuples would guarantee in-distribution sampling.
How can view Ptr, Pt. in common FSL benchmarks? Based the discussion above, we need to view train and test tasks in current FSL benchmarks as coming from different distributions, i.e., Pree of Pte. In order to ensure that both sets of tasks are still sampled iid from their respective distributions, it is convenient to view the train/test tasks as being iid sampled from a uniform distribution over all possible train/test class tuples induced by CB/Cn i..., Ptr Pcg Unif( ([Te E [CAl)) and test Fie = Fex == Unif( (Te c E (cx])) - a view which we will adopt in the rest of the paper.
4 Evaluating n-Distribution Performance
Although (as discussed in Section 3 current FSL benchmarks target OOD evaluation, example applications where ID generalization is instead required, and provide easily constructible benchmarks mirroring these scenarios. As will show, this distinction is important because meta-learning methods may perform markedly different in ID vs OOD scenarios.
explore
Example I (Federated Learning): Multiple works 12 20 22 27 have considered applying meta- learning methods in federated learning, in which the goal is to learn across a distributed network of devices (28] [6). Meta-learning can produce personalized models for unseen devices/users, improving over a single globally-learned model's performance. In this setting, a popular benehmark is the FEMNIST 66 handwriting recognition dataset. For FEMNIST,
assume there exists distribution of writers (idd in a federated network where each writer (with a unique id) is associated with a few-shot classification problem to recognize over the different character classes the writer has written for. We associate each writer id with a task Tia which randomly generates a support set with one random example per class and a query set with varying number of random examples per class.
ID evaluation on FEMNIST. We are given a total of 3500 writers sampled iid from P(id) and we randomly partition them into a 2509/538 /38 split for training, validation, and test tasks, following similar practices used in prior FL work (20 7. Note that this random split is performed at the task/id level. As such, we can treat the training and test tasks as being sampled iid from the same task distribution, unlike current FSL benchmarks.
Example 2 (Online Recommendation): Ren et al. (35] propose the use of Zappos 43 dataset as a meta-learning benchmark where each task is binary classification of shoe images into an attribute context. This mimics an online shopping recommendation problem, where each user has different shoe preferences based on specific shoe attributes (hence a single global predictive model would not perform well), and the recommendation system must quickly learn a user's likes/dislikes through a few interactions. In this simplified setup, we a universal set of shoe attributes A, and each user's preference is represented by a specific pair of unique attributes Ta representing a user with attribute preference a generates 2-way k-shot q-query (S, Q) pair by tid sampling k + q examples both from the set of images that carry both attributes in a (positive
(a1, (2), 01 o 0z A task

---

Table 1: Ranking in () of meta-algorithmms test performance on i) ID benchmarks FEMNIST, Zappos-ID (with either 1000 or 50 training tasks); and ii) OOD FSL benchmark minlmmaeNee..
D Method
FEMNIST
Zappus- ID
minilnageNet 5w.s
*wls
2w1Bk
(1000 train tacks)
2w.5 tai taske))
main
PN
(194.7 0.41%
88 40 + 0.13%
(1) 86.58 + 0.15%
77.67 : 0.17%
76.22 # 0.14%
Ridge
(4944.1 + 0.42%
88.01 +: 0.14%
85.56 + 0.16%
74.75 + 0.16%
77) + 0.15%
SVM
(9)91.22 0.45%
87)
0. 141%
82.12 # 0.16%
(4)70.00 + 0.17%
(A)77.72 # 0.15%
FO-MAML
N/A
81.90 + 0. 0.14%
80.14 # 0.15%
69.85 = 0.18%
(4.95.9 + 0.17%
examples) and from the set that does not (negative examples), Our task distribution is uniform distribution over tasks of all attribute pairs (P((7) = Unif([Ta : a E [A
ID evaluation on Zappos. Unlike in 335 where Zappos is used to measure OOD performance by having disjoint train and test attributes Aue n = c. this work we use Zappos for ID evaluations by tid sampling both meta- taata--raiinna and meta-test tasks from the same distribution I'A (7)) Through this modification of Ren et al. [s setup, we sample 1000 / 50 attribute pairs from an attribute set |A| = 36 to construct 1000 / 50 training tasks (each with randomly sampled (S, Q)) and evaluate ID performance on another 25000 test tasks sampled in the same way. Our evaluation setup captures realistic setting where the goal is to learn an algorithm that can generalize to the entire online shopper population despite being trained only on a randomly chosen subset of shoppers.
Remark, Even with ID evaluation it is possible to encounter unseen classes/attributes in meta-test tasks, specifically when the number of meta-train tasks is smaller than the number of underlying classes/attribttes (Section B). However, a suitable number of fid sampled meta-train tasks is needed to ensure good performance, which would naturally encompass a larger set of meta-train classes than those considered by OOD FSL benchmarks. For example, there are still 16 attribute pairs from the test tasks that are unseen in the 1000 training tasks on Zappos-ID. but the dataset is still in-distribution since the sampling distributions of train and test attribute pairs (and thus of tasks) are identical.
ID benchmark results. We evaluate the ID performance of four popular weean-ne methods: Prototypical Networks (PN) [39], MetaOptNet-SS VM (SVM) (25), MetaOptNeet
Regression and FOMAML [15 on our identified ID FSL benchmarks (Table [ Since FEMNIST's
(RR) 25 4
tasks have varying number of ways, FOMAML cannot be directly used due to the logit layer shape mismatch. We note that the performance order of the four methods are consistent on all three ID benchmarks yet surprisingly almost completely opposite to the performance order observed on the OOD benchmark mini (except for FOMAML). In terms of the actual performance differences, we notice that the ID performance advantage of PN over SVM becomes particularly large (> 3%) when we reduce of the number of training tasks for Zappos-II to 50; in contrast, on the OOD benchmark mini, SVM instead outperforms PN by 1.5% (a significant number as many newly proposed meta-learning methods often only report improvements over previous methods by 1.5% or less). These performance differences make it clear that the performance ranking flips between ID and OOD indeed exist, and as a result, these common OOD FSI. benchmarks (like mini) cannot be used to compare ID performance without modifications, giving further evidence to the danger of such practices (see Section 2). To further understand this phenomenon, we propose a way to also enable ID performance evaluation over these common OOD FSL benchmarks and see if there still exists difference in ID, OOD performance orders.
Modifying OOD FSL benchmarks for ID evaluation. From our previous discussion, we have shown that we can think of FSL training tasks as being sampled iid from the task distribution Pcg := (Te e E [CB])). To conduct an in-distribution evaluation, we need to sample fresh test tasks idd from Pcg. For freshly sampled To ~ IPcg. we also need to iid sample a fresh support query pair (S, Q) ~ Te- To ensure independent sampling from the already seen meta-training (S, Q) pairs, we need to introduce new examples from P. for each class c E Cg. Thus we construct slightly modified versions of four common FSL benchmarks i) miniImageNet-Mod (mini-M) [42). in which we find (s 700) unused examples (from ImageNet) for each base class and use them to evaluate the performance over I P'Cz. Here the original 600 examples of each base class are still only used for meta-training ii) CIFAR-FS-Mod (cifar-M) (30 FC-100-Mod (FC-M) [4, and tieredImageNet-Mel (fiered-M) :3:: As don't have additional samples for base classes, randomly partition each base class's 's current examples into an approximate 80/20 split where the training tasks are constructed using the former and the latter is reserved for ID evaluation.
ID vs. OOD conflicts still exist. In addition to evaluating the four aforementioned meta-learning methods, we also consider two supervised pretraining methods: the supervised learning baseline (SB)
5

---

35.0
61
92.0
77.0
15.5
166
34.3
91.5
94.5
72.0
95.0
33.5
12.0
79
sage
01.0
U1
63.5
07.0
64.3
RI Rated:
54.0
$4.5
144
nidpe
7H MG
92.02
50.5
52.0
Kikge Hora
93.5
77
557 07.0
92.5
89.5
89.5
90.5
69.5
54.5
92.0
92.5
82
85.0
mini-M Sw5s (Best
75
89.2 0
bicrad-M Sw5s (Best
78
(c) Choice (Best
18.0-
22.0
cifar-M (Gest Val)
81
Figure 1: We show the BaseGen and NovelGen performance tradeoff (for best validation snapshots); over the choice of set of four meta- neta-learning and two supervised pre-trainite methods on mini-M M (a) and tiered-M (b); over the number of ways to train PN on mini-M M and different learning rates to train FOMAML on FC-M (c); over the use of FIX- ML (S, Q) generation strategy or not (ML) with SVM, RR and PN on cifar-M M in (d).
p and Meta-Baseline (MB) 10. Both of these methods have been shown to outperform many meta- learning methods on the current OOD FSL benchmarks. For succintness, we call the generalization performance over the training task distribution ee,, BaseGen and performance over Fey NovelGen (we will refer to performance on validation tasks from Pc, ValGen in a later section). We plot the BaseGen and NovelGen performances of the aforementioned methods on minilmageNet-Mod and fiered dotted line connecting BaseGen and NovelGen value of same learned algorithm snapshot of a meta- learning method. We see that the ID/OOD performance order flips still exist within current FSL benchmarks themselves (the dotted lines of different methods cross each other frequently), showing that the issue of improving OOD at the expense of ID is a common realistic phenomenon for multiple benchmarks. In addition, despite outperforming all meta-learning methods on NovelGen, the non-meta-leame learning methods SB and MB cannot beat the best meta-learning methods on BaseGen, which demonstrates their restrictive advantage only in the OOD setting. More broadly speaking, these OOD FSL datasets constructed with belief that there exists commonality between the training and test task distributions so that an algorithm capturing this commonality using the training distribution alone would also generalize to the test tasks. Thus the phenomenon of improving OOD while sacrificing ID means the algorithm has in some sense failed to learn the essence/commonaltty from the given training tasks. Additionally, we that the BaseGen ranking of the four meta-learning methods over these common FSL bechmarks are exactly the same as the ranking over our two newly proposed ID benchmarks in Table [ We suggest that researchers who want to also test on their proposed methods' ID generalization performance can perform BaseGen evaluation method in our proposed way as is a simple addition their existing NovelGen evaluation setup.
in Figure [b)(b) (other datasets see Appendix
E Figure 4 ab)()), with
Certain OOD training choices might harm ID generalization. In addition to checking the discrepancy of ID vs OOD generalization comparison of different meta-learning methods, now ask, for a given meta-learning method, whether the meta-training choices found to be most effective on NovelGen would still be optimal for BaseGen.
i) Meta-training hyperparameters: In Figure we see that a higher learning rate when training FOMAML finds algorithms that have higher BaseGen, whereas lower learning rates are better for NovelGen. Additionally, we see that the proposed technique of training with more ways for PN in [39 can lead to better NovelGen performance but worse BaseGen performance than the 5-way trained PN whose training and test task configurations match.
ii) Meta-training (S,Q) generation alternatives: It was found in 38 that always using the same support examples for every base class when constructing S (FIX-ML) can improve NovelGen performance for several meta-learning methods over multiple OOD FSL benchmarks. However, restricting the training (S, Q)'s diversity sampled from Pcg seems counter-intuitive and we wouldn't I expect to improve the in-distribution generalization BaseGen. In Figure [d), we indeed see that FI only improves NovelGen performance at the expense of BaseGen by training over a much less diverse set of tasks.
These two observations above caution that some training techniques to boost test accuracy FSL benchmarks might only work for the OOD scenario but not the ID scenario.
Challenges With Out-of-Distribution Evaluation
After identifying some example benchmarks for ID evaluation, we come back to the current OOD FSL benchmarks to further examine some subtle challenges in OOD evaluation. Here, instead of focusing on the distinction between ID vs. OOD, we now look at some reliability and inconsistency problems

---

min/mmagenet-Mod 64w
tireedimagenetModd PN 20w
CIFAR-FS-M SVM
0.8
0.7
Accurary
0.6
0.5
Accuracy
Accuracy
05
0.4
0.3
0,2
70-
BaseGien hovdlGen
0.1
Epochs
0.0
mih M ciard
PC M ZEspos
10
25
30 0b 55 Epochs
10 30 to 40 Epochs
(8)
(b)
(c)
Figure 2: We plot the (Base, Val, Novel)Gen progression of way trained PN mini-M in (a) and of SVM trained on cifar-M in (d). In (h) we compute the Kendall rank correlation coefficient (p) between the validation and test rankings
model snapshots for IC (trained on cifar) and algorithm snapshots for PN, SVM. RR on OOD datasets mini-M (last 40 epochs), cifar-M (last 40), FC-M (last 10), tiered-M (last 20) and ID dataset Zappos-ID (last 30); in (c) we show the BaseGen tracking NovelGen for 20- trained PN tiered-M.
within the OOD evaluation itself. In particular, we highlight two concerns: 1) Despite providing a validation set of classes for task construction during mnta-validattin, it is not clear whether this is reliable way to perform model selection. 2) As there is no requirement on how similar the training and test task distributions need be in the OOD setting, there can be inconsistencies in method comparisons when the evaluation setup is slightly modified.
5.1 Model Selection
To compare set of meta-learning methods on the OOD test task distribution, one needs to select a representative algorithm learned by each method to be evaluated on the test task distribution. To do so, one should first decide what set of hyperparameter configurations to choose from for given meta- learning method (we define the entire set all hyperparameters for a training run as a hyperparameter config). For each such considered config, after its training is completed, we need to choose one of the algorithm snapshots saved during training to represent it (which we call snapshot selection). Then the set of hyperparameter configs are compared based their respectively selected snapshots, and single config is then chosen among them (which we term hyperparameter selection). This config's s selected snapshot will represent the given meta- vecaalearning method to be finally evaluated the test tasks. We refer to the combined problem of hyperparameter and snapshot selection as model selection. (See Appendix F.T for a simplified example of snapshot and hyperparameter selection; see H22 for the distinction between the commonly used technique early-stopping and snapshot selection.)
Snapshot Hyperparameter selection. If the snapshot selection strategy cannot reliably identify a snapshot with good test performance, unfairly represented by mischosen snapshot, leading to erroneous hyperparameter selection. Thus we believe snapshot selection is a more fundamental problem and we focus our analyses on it (more on hyperparameter selection in Appendix F.5). In OOD FSL benchmarks, reserved set of validation classes (disjoint from the set of novel classes) is provided, which as we have argued, provides tasks which are not sampled tid from the test task distribution. As result, we ask: is performing validation on the given validation tasks reliable, and, if not, are there other options? In contrast, there is typically need for such concerns in standard ID supervised learning, where the validation set is sampled iid from the test distribution.
possible that some hyperparameter configs will be
5.1.1 Option 1: Snapshot selection tradition using ValGen.
By providing a set of validation classes, it has become the default practice for meta-learning methods to use ValGen performance for snapshot selection. However, because IPc, and Fcx are different tasks distributions, it is not clear whether higher ValGen performance is strongly correlated with a higher NovelGen performance. In Figure a). we plot the progression of ValGen and NovelGen of a 64-way trained PN on mindmageNNet-Moo 5w5s tasks. We notice that ValGen is consistently higher than NovelGen, indicating that meta-val performance is not an accurate estimator of NovelGen. More importantly, we see trendwise that while ValGen is generally non-decreasing, NovelGen starts to decrease after epoch 30. Thus the snapshot selected according to the best ValGen is not the snapshot with the best possible meta-test performance. In fact, this loss of NovelGen performance due to choosing the best ValGen model instead of the actual best NovelGen model can be particularly large, with values being 1.1% % for SVM, 1. 1.2% for RR, 0.7% for PN, and 0.9% for FOMAML on the FC-100 dataset. These performances losses for each method are especially concerning considering the differences among the best possible NovelGen performance of these different methods are often smaller than 1.5%.
aaa.
7

---

Ranking similarity analysis. In light of the above observation, we ask a more general quantitative question: How similar is the ranking of the training snapshots using meta-val performance (ValGen) to the ranking using the meta-test performance (NovelGen)? To answer this, we compute the Kendall rank correlation coefficien p [21] between the ValGen and NovelGen rankings of algorithm snapshots trained on four OOD FSL. benchmarks (Figure and our ID) benchmark Zappos-IID III whose validation and test tasks come from the same distribution IP, A. More concretely, for each meta- learning method and dataset combination, we save the algorithm snapshots (one from each epoch) throughout meta-training and rank these algorithm snapshots according to their ValGen and NovelGen value respectively. Then p is computed between these two rankings for this specific method, dataset) combination. For snapshot selection through ValGen to work reliably, we need p to be close to 1. For context, we also compute for standard supervised image classification problem (IC), where train, val and test examples are sampled from the same example-level distribution.
Unreliabity of ValGen snapshot selection. From Figure 1), we see that when using validation samples generated iid from the test distribution (Zappos-II) ID) and supervised learning IC), the value of is consistently higher than the OOD benchmarks, indicating the validation performance can more reliably track the trend of test performance in the ID setting than on the OOD FSL benchmarks. In particular, for the cifar-M and FC-M datasets, the ValGen ranking of algorithm snapshots seems to be only weakly correlated with the true meta-test ranking for all the meta- meta-learning methods. In fact, the meta-val and meta-test rankings of the most useful snapshots can sometimes even be negatively correlated (p Rs -0.12 < 0 over all snapshots after epoch 30 in the training scenario shown in Figure 2a)). These results show that on the OOD FSL benchmarks, snapshot selection using the pre-assigned validation tasks can sometimes be unreliable/unable to identify a snapshot candidate with top-tier meta-test performance among all the snapshots.
5.1.2 Option 2: Snapshot selection alternative using BaseGen.
Beyond using OOD validation tasks for snapshot selection, inspired by the domain generalization community 23 18] we can alternatively also consider using the IDD performance over the training task distribution for snapshot selection. Perhaps due to the lack of an ID evaluation setup in common FSL benchmarks, this possibility has not been widely considered. Enabled by modifications of current FSL benchmarks, we can now evaluate the ID generalization performance (BaseGen) throughout training in addition to ValGen.
We plot how BaseGen and NovelGen progress for meta-learning methods trained on two different datasets in (d)). Here we see that on Miered-M, the BaseGen and NovelGen of PN both increase fairly consistently; thus picking the algorithm snapshot with the highest BaseGen performance (roughly the end-of-training snapshot) would also give close-to-best NovelGen. However, cifar-M, after the learning rate drop at epoch 20, SVM's BaseGen keeps improving while NovelGen starts deteriorating. In this case, selecting snapshots according to the best BaseGen would pick much worse snapshot than picking according to the best ValGen. (For concrete numbers of how much snapshot selection through BaseGen vs. ValGen could impact the chosen snapshot's 's NovelGen performance in each of these two cases, see Appendix F.3 This ambiguity of whether ID or OOD Validation snapshot selection is better has also been documented in domain generalization, where Gulrajani and Lopez-Paz [18 find ID) model selection can perform better in some settings while Koh et al. (23 find OOD validation model selection is better in others. Despite this ambiguity, we believe the commonly neglected in-distribution (BaseGen) snapshot selection approach should be considered by users of OOD FSL benchmarks as a viable alternative to the default ValGen selection approach in proper settings.
5.2 Inconsistencies in Meta-learning Method Performance Comparisons
After discussing concerns regarding OOD model selection for a given mela-learning method, we now analyze the reliability and consistency of conclusions drawn from comparing different meta-learning methods OOD performance these benchmarks. In particular, we focus on two cases:
Inconsistency example I: Limited number of novel classes in a single benchmark. Since in OOD FSL we specifically care about the learned algorithms' ability to quickly learn many unseen concepts, we should not be satisfied with an algorithm performing well only on tasks constructed from a small number of pre-selected novel classes. However, for many widely-used FSL benchmarks
P [-1, e means correlation, while p 1/p -1 means exactly same/opposite rankings.
8

---

1Cal
[0x1
C2
CF
1F.0 5%
Dunaeet Methods
ZAPPOS OOD
mubal ODD
(IC)
0.5%
100 imgsiclass
3%
1%
2w
2w:5
5w.5
172
Randar 23 dleveeeeeoy
dians dosett is So.
(1)
0.5%
20
120
15%
25%
PN
(9)80.51
(R) 74.67
(3) 76.22
(8)
0.5%
20
552
20%
24%
20.135%
20.14%
20.14%
(ii)
1.0%
64
20
552
10%
12%
Ridge
(3) 80.51
(2) 73.50
(3) 77.20
0.5
0.0
0.0
(10)
(V)
0.5%
0.5%
64
160
20
552
552
20%
0%
23%
7%
SVM
(8) 79.70
10.15%%
+00.18%
(8) 72.84
20.14%
+0.1445
73777.2
10.15%
10.10%
0.5
(v)
0.5%
351
20
552
9%
15%
74.82
(4)
76.96
030
[Vi8)
0.5%
351
160
552
0%
1%
MAML
#-0.14%
+00100%
#0.17%
(h)
Figure 3: In (a), we show the CDF plot of As, (Cw) - As, (Cx) over 100 randomly chosen sets of C.v with 20 (out of total 120)= novel classes each. In (b), for different values of true performance difference e and values of underlying class size [Cz]. training class size ((Cp)). and evaluation class size (CCx )): show the percentage of conclusion flips (CF) and improvement exaggerations (IE) with 5 0.5% computed over 100 evaluations. In (c), we demonstrate the inconsistencies in performance rankings for PN, SVM, RR, and FOMAMI. on two OOD benchmarks: Zappos-OOD and mini-OOD.
(minilmageNet, CIFAR-FS, FC-100), only 20 novel classes are used for meta- meta-testing. Ideally, even we don't it measure it, we would hope that our learned algorithm would also generalize to tasks made by other sets of classes different from the fixed small set of novel classes.
Formal setup. We suppose the existence of a much larger collection of classes Ct., where the novel classes CN used for meta-testing is a small random subset with each element class sampled uniformly and non-repeatedly from CL and fixed thereafter for NovelGen evaluation. Ideally, our goal is to evaluate an algorithm snapshot 6 on the task distribution Pcc, (denote this performance by A,(C))). yet during evaluation we only have access to the novel classes in Cy and thus we can only compute the performance Ag(Cx). It is easy to see that when we randomize over different choices of Cv. the expected performance over the sampled novel classes would match the true performance: Bcv [Ag(x)]
(Cz)) However, when using a single randomly sampled novel set, the estimator As(CN) can have high variance (see Appendix [. Instead of relying on Ap(Cx) to directly estimate (Cr.), we ask a more relaxed question: for a pair of algorithms As, and Ag, (given by two meta-learning methods), if the true performance Ag, (CL) (C,(CL) = >>>>> O. how frequently will we observe an opposite conclusion i.e., P(A,, (CN) < Ap, over a randomly sampled CN (we call this event conclusion flip)? Additionally, it is also possible that the observed performance difference on Cn is greater than the true difference e by some amount > O. In this case, the NovelGen observation would make the algorithm snapshot é1 look better than de more than it actually is on Cz. Thus we also ask what is the probability of IP(A,, (Cv) - A₂, (Cx) > c C+ 5) and we call such events improvement exaggerations. To answer both these questions empirically, we first suggest some larger class sets C, for mini and tiered. For both we select unused classes from ImageNet disjoint from the base and validation classes. For liered, we use all the remaining 1000 - 351 (base) 997 (novel) = 552 classes as Ct. while for mini, we randomly choose 120 or 552 (to match |Cy| in fiered) unused classes. We fix these Ct choices in the following analysis.
Checking the frequency of conclusion flips and exaggerations. Figure 3 (a) shows the empirical CDF of the performance differences As, (Cx) Asz (Cx) computed over 100 randomly sampled 2ize-20 novel class sets C. for fixed pair of PN and RR algorithm snapshots whose true performance difference over the larger 120 classes Cg is 0.5%. In 15% of the cases the performance order is flipped from the true order, while in 25% of them improvements are exaggerated by more than 0.5% (total difference greater than 1%). Moreover, for some of the performance order flips, the observed performance difference can be quite negative < -0.5% thus significantly opposite to the true performance order. (Here for each run we evaluate both methods 20,000 tasks sampled from Pc, in order to significantly reduce the randomness in estimating the true As, (CN), Agz (Cx).)
Comparison to supervised learning. We check for the conclusion flip and improvement exaggeration frequency when only a random subset of the full test set (100 randomly drawn test images from each base class in mini-M) is used to compare two supervised learning image classification models with the same full test set performance difference of c = 0.5 (row (IC) in Table b)). Here we that compared to supervised learning, the chances of getting an incorrect performance comparison (row (i) in Table is much higher for the meta-learning OOD FSL benchmarks when evaluating only on 20 randomly chosen novel classes (as done in several FSL benchmarks).
Larger |Cr, makes even less reliable but larger e helps. If we were to care about an even larger set of underlying classes (C2rl = 552) despite still using only 20 random novel classes for evaluation
9

---

comparison, the conclusions are even less reliable (Table[3[b) (i) vs (ii). On the other hand, we do see that the performance comparison becomes comparatively more consistent if the true performance difference is higher (1% in (iii) compared to 0.5% in (ii)), despite that there still exists statistically significant chance (10%) of getting an opposite conclusion.
OOD evaluations in current FSL benchmarks. In practice, because 1) we never specify exactly what and how big the underlying set of classes that we care about is, and 2) some of the recent meta-learning methods (SVM vs PN on cifar in Table 2 of [25] R2-D2 vs GNN on mini in Table I of 4.. FIX- MLL (38 sometimes only improve over the prior works by < 1%, we believe researchers should be aware of the possibility of getting a performance conclusion that is inconsistent over a single randomly chosen and fixed set of 20 novel classes used by some of these benchmarks.
Actionable suggestions. Since the size of the unknown underlying larger class set Ct. and the true performance difference might not be something one can directly control when designing the OOD benchmark, we now discuss two actionable choices that can reduce the chances of conclusion flips:
Use more novel classes in the evaluation: By comparing (iv) vs (v) and (vi) vs (vii) in Table 3 we see that the frequency of conclusion flips and improvement exaggerations are much lower when 160 novel classes are used as opposed to 20 when |Cz] is the same.
ii) Train on more base classes: The tiered dataset has more base classes (351 compared to 64 for mini) to train on. When comparing PN and RR snapshots trained on modified version of tiered with fewer (randomly sampled 64 out of 351 to match mini) base classes, we see that the CF frequency is twice as high compared when 351 base classes are used (Table (abbleBh)(ii (iv(v) vs (vi)).
Based on these two trends, for more reliable comparisons of meta-learning methods' OOD perfor- mance we suggest using datasets like magred and MetaDataset (both with much larger set of base and novel classes) in addition to the smaller benchmarks like minilmageNet, CIFAR-FS, and FC-100, which some recent works [e.g.. 30 still solely rely upon.
Inconsistency example 2: Inconsistency across multiple OOD FSL benehmarks. Unlike the ID scenario where the training and test task distribution are the same, the similarity between training and test distributions in the OOD FSL benchmarks can vary significantly. Ideally, we want a meta- learming method to be consistently better on multiple OOD benchmarks with different type/degree of distribution shifts. Since Ren et al. (35) originally use the Zappos dataset for OOD evaluation, we also perform similar evaluation on new attribute pairs based on their setup. At test time, we use an attribute set A' disjoint from the one used in the Zappos-IID setup A, and sample attribute pairs from A' only. This induces a test task distribution A different from the training task distribution P.A- We evaluate different meta-learning methods on these Zappos-OOf OOD tasks to see if the performance order is consistent with other OOD FSL benchmarks (Table 3())). Here we see that despite SVM outperforming RR and PN on mini NovelGen, the performance order of these three methods are completely flipped on Zappos-OOD Similar observations can be made from TADAM underper- forming PN in Table 2 of [35] despite TADAM being shown to outperform PN on the other more commonly FSL benchmarks. This inconsistency over different rypes of OOD FSL benchmarks is in stark contrast to the consistency of performance rankings over the 6 different II) benchmarks (FEMNIST, Zappos-II,, and the BaseGinn results of the 4 current FSL benchmarks (Section 4). Based on these findings, we caution meta-learning researchers to be aware of such conclusion inconsistencies over different OOD FSL scenarios and reason carefully about the generality of their empirical findings when using only specific types of OOD datasets.
6 Conclusion
In this paper, we categorize meta few-shot learning evaluation into two settings: in-distribution (ID) and out-of-distribution (OOD). After explaining why common FSL benchmarks reflect OOD evaluation, we identify realistic needs for ID FSL evaluation and provide new benchmarks as well as suggestions on how to modify existing OOD FSL benchmarks to allow for ID evaluation. Through experiments performed on these ID benchmarks, we demonstrate surprising phenomenon that many meta-learning methods/training techniques improve OOD performance while sacrificing ID performance. Beyond this, through quantitative analyses, we show that even in the OOD scenario, current FSL benchmarks may present subtle challenges with both model selection for a given meta- learning method and reliable performance comparisons of different methods. For these concerns, we provide initial suggestions and alternatives with the hope of alleviating these issues. Overall, we aim to raise awareness about the dichotomy of FSI. evaluation and to motivate the meta-learning community to collectively reason about ways to improve both ID and OOD methodology and evaluation.
10

---

This work was supported in part by the National Science Foundation Grant IIS IIS1838017, a Google Faculty Award, Facebook Faculty Award, and the CONIX Research Center, Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the NSF or any other funding agency.
Acknowledgements.
References
[1] M. Al-Shedivat, L., Li, E. Xing, and A. Talwalkar. On data efficiency of meta-learning. In International Conference on Artificial Intelligence and Statistics, pages 1369-1377. PMLR, 2021.
[2] S. M. Arnold and Sha. Embedding adaptation is still needed for few-shot learning. arXiv preprint arXiv:2 10:.00204.07555 2021.
[3] Y. Bai, Chen, P. Zhou, T. Zhao, Lee, S. Kakade, H. Wang, and C. Xiong. How important is the train-validation split in meta-learning? arXiv preprint arXiv:2010.05843 2020.
[4] L. Bertinetto, J. F. Henriques, P. H. Torr, and A. Vedaldi. Meta-learning with differentiable closed-form solvers. In International Conference on Learning Representations, 2019.
[5] G. Blanchard, G. Lee, and C. Scott. Generalizing from several related classification tasks to a new unlabeled sample. Advances in neural information processing systems, 24:2178-2186, 2011.
[6] S. Caldas, S. K. Duddu, P. Wu, T. J. Konečný, H. B. McMahan, V. Smith, and A. Talwalkar. Leaf: A benchmark for federated settings. arXiv preprint arXiv:/812.01099 2018.
[7] F. Chen, M. Luo, Z. Dong, Z. Li, and X. He. Federated meta-learning with fast convergence and efficient communication. arXiv preprint 1802.0777 2018.
[8] J. Chen, X.-M. Wu, Y. Yi Q. LI, L.-M. Zhan, and F.-l. Chung. closer look at the training strategy for modern meta-leaming. Advances in Neural Information Processing Systems, 33, 2020.
[9] W.-Y. Chen, Y.-C. Liu, Z. Kira, Y.-.. F. Wang, and J.-B. Huang. A closer look at few-shot classification. International Conference on Learning Representations, 2019.
[10] Y. Chen, X. Wang, Z. Liu, H. Xu, and T. Darrell. new meta-baseline for few-shot learning. arXiv preprint arXiv:2003.0439) 2020.
[11] S. S. Du, Hu, M. Kakade, D. Lee, and Q Lei. Few-shot learning via learning the representation, provably. arXiv preprint arXiv:2002.09434 094344 2020.
[12] A. Fallah, A. Mokhtari, and A. Ozdaglar. Personalized federated learning with theoretical guar- antees: A model-agnostic meta-learning approach. Advances in Neural Information Processing Systems, 33, 2020.
[13] A. Fallah, A. Mokhtari, and Ozdaglar. Generalization of model-agnostic meta- algorithms: Recurring and unseen tasks. arXiv preprint arXiv:2102.03882. 2021.
[14] C. Finn and S. Levine. Meta-learning and universality: Deep representations and gradient descent can approximate any learning algorithm. In International Conference on Learning Representations, 2018. URL URL httppp:///ppeereeeieee http://wwww....eeeoooooo/t/oelaa
[15] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference Machine Learning, pages 1126-1135, 2017.
[16] C. Finn, K. Xu, and S. Levine. Probabilistic model-agnostic meta-learning. In Advances in Neural Information Processing Systems, pages 9516 2018.
[17] S. Gidaris and N. Komodakis. Dynamic few-shot visual learning without forgetting. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pages 4367-4375, 2018
11

---

[18] I. Guirajani and D. Lapez-Paz. In search of lost domain generalization. arXiv preprint arXiv:2007.01434, 2020.
[19] Hospedales, A. Antoniou, P. Micaelli, and A. Storkey. Meta-learning neural networks: A survey. arXiv preprint arXiv:2004.05449) 2020.
[20] Y. Jiang, J. Konečný, K. Rush, and S. Kannan. Improving federated learning personalization
via model agnostic meta learning. arXiv preprint arXiv:1909. 12488, 2019.
[21] M. G. Kendall. Rank correlation methods. Griffin, London, 3d ed. edition, 1962.
[22] M. Khodak, M.-F. F. Balcan, and A. S. Talwalkar. Adaptive gradient- meta-learning methods. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019. URL htsp::ttp///rooceddinng (4aa0d09665210444222222221 (tw0:6622164499--999999 pdf
[23] P. W. Koh, S. Sagawa, H. Marklund, S. M. Xie, M. Zhang, A. Balsubramani, W. Hu, M. Ya- sunaga, R. L. Phillips, I. Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. arXiv preprint arXiv:2012.07421, 2020.
[24] H. B. Lee, H. Lee, D. Na, S. Kim, M. Park, E. Yang. and S. J. Hwang. Learning to balance: Bayesian meta- meta-learning for imbalanced and out-of-distribution tasks. International Conference on Learning Representations, 2020.
[25] K. Lee, S. Maji, A. Ravichandran, and S. Soatto. Meta-learning with differentiable convex opti- mization. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 10657-10665, 2019.
[26] T. Li, A. K. Sahu, A. Talwalkar, and V. Smith. Federated learning: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 37(3)500-60, 2020.
[27] S. Lin, G. Yang. and J. Zhang. A collaborative learning framework via federated meta- learning. In 2020 IEEE 40th International Conference on Distributed Computing Systems (ICDCS), pages 289-299, Los Alamitos, CA, USA, dec 2020. IEEE Computer Society. doi: 10. 11119//CDCS5777742200000000 1109/ICDCS47774 20w1447777722000.000
URL htLLstts:://oo ieeecompttersocietty org/10
[28] H. B. McMahan et al. Advances and open problems in federated learning. Foundations and Trends@ in Machine Learning, 14(1), 2021.
[29] K. Muandet, D. Balduzzi, and B. Schölkopf. Domain generalization via invariant feature representation. In International Conference on Machine Learning. pages 10-18. PMLR, 2013.
[30] B. N. Oreshkin, Rodriguez, and A. Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. In Advances in Neural Information Processing Systems, 2018.
[31] A. Rajeswaran, C. Finn, S. M. Kakade, and S. Levine. Meta-learning with implicit gradients. In Advances in Neural Information Processing Systems, pages 113-124, 2019.
[32] S. Ravi and H. Larochelle. Optimization model for few-shot learning. In International Conference on Learning Representations, 2017.
[33] M. Ren, E. Triantafillou, Ravi, Snell, K. Swersky, J. B. Tenenbaum, H. Larochelle, and R. S. Zemel. Meta-learning for semi-supervised few-shot classification. arXiv preprint arXiv:1803. 00676, 2018
[34] M. Ren, R. Liao, Fetaya, and R. Zemel. Incremental few-shot learning with attention attractor networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox. and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL httph:tt///// http://////ccceeig....m.. www.eeeewww.nn.nnnseeee....em www..ipp..cppppee////00/.. ffi1e/88330022000996122222626
Paper pdf
[35] M. Ren, E. Triantafillou, K.-C Wang, Lucas, Snell, X. Pitkow, S. Tolias, and R. Zemel. Flexible few-shot learning with contextual similarity. arXiv preprint arxiv:2012.05895, 2020.
12

---

[36] A. Santoro, S. Bartunov, M. Botvinick, D. Wierstra, and T. Lillicrap. Meta-learning with memory- neural networks. In International conference on machine learning, pages 1842-1850. PMIR, 2016.
[37] J. Schmidhuber. Evolationary principles in self-referential learning, learning how to learn: the meta-meta... hook. PhD thesis, Technische Universität München, 1987.
[38] A. Setlur, O. Li, and V. Smith. Is support set diversity necessary for meta-learning? arXiv preprint 1rXiv:2011.14045 2020.
[39] Snell, K. Swersky, and R. Zemel. Prototypical networks for few-shot learning. In Advances in neural information processing systems, pages 4077-4087, 2017.
[40] E. Triantafillou, T. Zhu, V. Dumoulin, P. Lamblin, U. Evci, K. Xu, R. Goroshin, C. Gelada, K. Swersky, P.-A. Manzagol, et al. Meta-dataset: A dataset of datasets for learning learn from few examples. In International Conference on Learning Representations, 2020.
[41] E. Triantafillou, H. Larochelle, R. Zemel, and V. Dumoulin. Learning universal template for sew-shot dataset generalization. In International Conference on Machine Learning. 2021.
[42] O. Vinyals, C. Blundell, Lillicrap. D. Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems, pages 3630-3638, 2016.
[43] A. Yu and K. Grauman. Fine-grained visual comparisons with local learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 192-199, 2014.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)?
[Yes] We include the code and instructions to setup the dataset and run experiments in the github link in the footnote on the first page of the paper. We also provide link to download the data.
(b) Did you specify all the training details (c.g., data splits, hyperparameters, how they were chosen)? [Yes] We document these in the Appendix.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)?
[Yes] We provide error bars for the estimated performance over each task distribution.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Details Appendix.
4. If you using existing assets (e.g., code, data, models) curating/releusing assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] We use public datasets and cite the creators.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material or as URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "xQWwY2loQOK", "reviewer": "Reviewer_hGy7", "summary": "In this paper, the authors focus on certain limitations of popular few-shot learning benchmarks (e.g. MiniImageNet or TieredImageNet) that are used to evaluate meta-learning algorithms. The authors argue that these benchmarks, although generally known to be testing in-domain generalization, actually test out-of-domain generalization and therefore the meta-learning algorithms that are benchmarked using these datasets over optimize for out-of-domain generalization at the cost of in-domain generalization. The authors also shed some lights on some other pitfalls of meta-learning training and evaluation procedures like how to select the best model during training and how increasing the amount of novel classes for meta-test can make the test-time performance evaluation more robust.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In this paper, the authors challenge an existing assumption in common few-shot learning benchmarks (e.g. MiniImageNet or TieredImageNet) that are used to evaluate meta-learning algorithms that these benchmarks actually test out-of-domain generalization although the benchmarks claim to be in-domain (as the classes are split from the same dataset). Empirically, the authors show that by evaluating meta-learning algorithms using these benchmarks, those are incentivized to do better on out-of-domain generalization at the cost of in-domain generalization. Apart from this, the authors also provide analysis on some other aspect of meta-training like how to pick the best model during training and how to design the test set better to improve the test time evaluation procedure. Overall I found the paper to be easy to read and found the notations to be clear. I also like some of the areas that the authors have focussed on like 1)  how to do model selection during meta-training 2) including additional samples from the base classes to do in-domain evaluation and 3) include more novel classes in the meta-test segment to make the evaluation more robust. \n\nHowever, for me, there are certain major drawbacks of the paper that I'd like to point out:\n*  The main claim that existing benchmarks focus on out-of-domain generalization and the meta-learning methods optimize for that are not solidly justified via the experiment(s). From Table 1, we can see that the methods perform pretty much identically on both in-domain and out-of-domain (as per the notion of this paper) rather than the claim that the trend is reversed between these two scenarios.   PN, Ridge and SVM all perform within $1$ percent or less on all the three benchmarks (especially when considering the confidence intervals, the gap is insignificant). This, in my opinion, is not sufficient to justify this whole claim of in-domain vs out-of-domain and that the methods optimize one vs the other. To me, it looks like they all perform quite similarly on both in-domain and out-of-domain.\n* Secondly, I am not convinced with the rationale of why the authors consider the classes belonging to meta-test as out-of-domain. I would have liked to see a more rigorous analysis of domain-gap between the tasks sampled from the base and novel classes to arrive at this conclusion. For example, the authors can try using an algorithm like [Task2Vec](https://arxiv.org/abs/1902.03545) to embed the tasks from these two splits and then compare their mutual distances to verify their claim.\n* On the topic of including additional images from the base classes, this idea is not new and has already been used in [this](https://arxiv.org/abs/1804.09458) paper by Gidaris et al.\n*  I agree with the author's claim that using meta-validation set might not be the right way to figure out the best model during training. However, the authors do not provide a clear alternative and suggests also considering using the best model that gets the lowest loss on the meta-train set. As a practitioner, it is not clear to me which method should I choose.\n* Although the paper is easy-to-read, the organization of the topics is not great.  In my opinion, the paper tries to address too many points without a very clear suggestion or solution for most of those. \n\nOverall, I think the paper has some merit when it comes to addressing a few concerns with meta-learning training and evaluation, but in its current form, it does not have enough novelty nor the claims are sufficiently backed by the experimental results. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There are no potential negative societal impact of their work as far as I understand.", "review_text": "", "extra_scores": {"ethical_concerns": "There are no ethical concerns of their work as far as I understand.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3-4 hours"}, "extra_text": {}}, {"review_id": "sij_15LbHAl", "reviewer": "Reviewer_Scao", "summary": "This study extensively introduces the distinction between ID and OOD scenarios in a meta-learning framework. This points out that model selection and performance comparison in OOD scenarios have some problems. To resolve this issue, the authors introduce protocols for FSL benchmarks in ID and OOD evaluation.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work points out the importance of the distinction between ID and OOD scenarios in a meta-learning framework. The current evaluation protocol, especially in FSL, focuses on an OOD scenario rather than ID. However, I still believe that validating with an OOD scenario makes more sense than the one with an ID setting, because the objective of meta-learning (compared to inductive learning) is to learn a model (or an agent) that is quickly capable of acquiring a new skill in a different environment. \n\n### Strengths\n* This work points out the current FSL benchmark aims to validate meta-learning models in an out-of-distribution (OOD) setting, rather than an in-distribution (ID) scenario. In addition, the authors presented several examples in ID scenarios to highlight the difference between two scenarios. \n* I’d like to see the challenges in OOD scenarios. Specifically, I agree that model selection is problematic in this setting. \n\n### Weaknesses\nClarity should be improved. In the submission history above, the one of critical reviews on an earlier version of this work is the lack of clarity. However, the current version is not that clear in many places. \n\n\n### Detailed comments\nIn my understanding, FSL benchmarks have been deliberately designed in OOD scenarios to validate meta-learning algorithms on how to quickly adapt new skills in different environments. Therefore, I believe that the current evaluation protocol isn’t invalid. \n\nI couldn’t follow the logic behind that randomized class partitions don’t imply randomized task partitions. Specifically, I’m not sure why a task is represented by a class tuple, rather than a class. \n\nSome attempts have been proposed to mitigate the negative effects of domain mismatch between the meta-training and meta-testing. I think that the papers below are related to the analysis presented here.\n* TAEML: Task-Adaptive Ensemble of Meta-Learners, NeurIPS workshop on MetaLearn, 2018. \n* Domain-Agnostic Few-Shot Classification by Learning Disparate Modulators, NeurIPS workshop on MetaLearn, 2019. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This work is purely theoretical, so this doesn't bring any negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "nqXbIWgZwW", "reviewer": "Reviewer_ficQ", "summary": "The paper studies the evaluation of meta-learning algorithms on in-distribution and out-of distribution learning tasks. Through an argument based on a proof by contradiction, the paper shows how most of the current evaluation benchmarks for few-shot learning are based on out-of-distribution evaluations. The paper outlines the difference between in-distribution and out-of-distribition evaluations, provide examples for the necessity of in-distribution evaluations, highlight concerns about model selection strategies and algorithm comparison when using OOD evaluation benchmarks, and propose suggestions to remedy these concerns by introducing new IID benchmarks and modifications to the existing OOD evaluations.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "##  Originality\n\n- The categorization of meta-learning evaluation into in-distribution and out-of-distribution is an important distinction for meta-learning researchers and practitioners to consider. \n\n- The suggested insights about the subtleties in out-of-distribution evaluation are informative for the meta-learning community. \n\n- The related work section is adequate and covers recent related work in this area.\n\n## Quality\n\n-  The paper is technically sound and the claims in the paper are supported by empirical evaluations, however, almost all the experiments were performed on computer vision tasks, and other applications for meta-learning in language and speech were not considered.  \n- The paper could also benefit from a more in depth theoretical analysis. \n\n\n## Clarity\n\n- The paper is clear, well organized, and easy to follow, except for the distinction between “Snapshot selection” vs “Hyper-parameter selection”. It’d be useful if the authors could provide a concrete example to distinguish between the two concepts.\n\n## Significance\n\n- The work is significant and presents new insights on the evaluation of meta-learning methods, model selection, and comparison between different approaches.\n\n- The experimental analysis is thorough, however, except for the proof by contradiction argument for out-of-distribution evaluation of the current benchmarks, the paper lacks the theoretical insights.\n\n- Except for a small section for “Actionable Suggestions”, the paper lacks more insights to mitigate the inconsistencies of the current few-shot learning benchmarks, and issues regarding model selection and comparison of different approaches.\n \n ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- Results are mostly empirical, and in many cases lack theoretical analysis.\n- The papers highlights significant inconsistencies and issues regarding the current meta-learning evaluation benchmark, this is very useful for the meta-learning community to know, however, I still think the suggestions provided in the paper doesn’t address all of these issues, for instance, the answers for these questions are not clear to me:\n\n1) Out of the four meta-learning approaches used in the paper (proto-nets, meta-opt-net, foMAML, and R2D2), which approach consistently works best for in-distribution and out-of-distribution learning tasks?\n2) It’s possible that a practitioner would be interested in developing meta-learning algorithms that work well for both in-distribution and out-of-distribution learning tasks. For instance, in “Example 1” for federated learning, maybe we’d also be interested in deploying a model for a new user writing in a totally different language script, what would be the best practice recommendations for model selection and comparison in this setting?  ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "Phz6JfXricA", "reviewer": "Reviewer_QWos", "summary": "This work differentiates two evaluation procedures for meta-learning approaches: *in-distribution (ID)* and *out-of-distribution (OOD)*. In the former, the *tasks* used in the meta-training and meta-testing stage come from the *same distribution* whereas in the latter, the tasks come from *different distributions*.\nThe authors give a formal proof to show that current Few-Shot Learning (FSL) benchmarks only give OOD performance and not ID performance. They propose updated versions of these benchmarks to reflect ID performance of popular FSL models. They show that the performance can vastly differ between OOD and ID, and that the ranking in terms of performance of the models is almost inverted. Furthermore, they also verify if selecting the best model based on validation performance is really correlated with the best performance in an ID setting, and show that it is not always the case. Finally, they investigate the inconsistencies in performance between multiple version of the same models in relation to the number of classes in the novel set used for meta-testing. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "__**Originality:**__  This paper challenges current meta-learning benchmarks and provide formal justifications to consider them as OOD even when evaluating on the same dataset, rather than ID. As mentioned by the authors, some prior works in related topics already considered a comparison of performance on *base classes* and *novel classes*, such as for transfer learning, incremental or continual learning methods. Meta-learning methods, however, were always compared using a novel set of classes in the literature. This is an interesting and novel way to evaluate meta-learning methods.  \n\n__**Quality:**__  Their main argument about the OOD meta-learning benchmarks is supported by a formal proof-by-contradiction. The authors also proved it empirically by modifying the meta-learning benchmarks and showing that performance rankings on the current OOD benchmarks does not translate to their proposed ID-version of the benchmarks. However, they do not give explanations as to why some methods can have drastically different performance rankings between ID and OOD (like PN), while others do not (like FOMAML).\nFinally, the authors also highlight the problem of model selection with current OOD benchmarks. However, I wonder if the inconsistencies found in the performance comparisons are linked to the high variance of performance between tasks, since only mean accuracies are compared. This variance is known to be higher than the $\\epsilon = 0.5\\%$ chosen by the authors on certain benchmarks.\n\n__**Clarity:**__  A lot of important details only appear in the appendix. In the main paper, the section 3 and particularly the part explaining the formal result, which is the starting point of the paper, are quickly explained. I had to read the appendix to better understand the proof and the result. All the experimental settings are also relegated to the appendix, even though I think they are important to assess if the experiments and comparisons are sounded or not. Other than that, the paper is well written, and its objectives are clear. \n\n__**Significance:**__ I think the paper is valuable to meta-learning researchers and practitioners, even more so for applications which would benefit from true ID benchmarks as detailed in the paper. There are still important problems, such as the model selection, but the paper offers a novel view to current meta-learning benchmarks.\n\n### Minor Remarks\n\n- Some minor typos I noticed:\n  - l.247-248: \"is in some sense fail~~s~~*ing*\"\n  - l.392: \"set ~~set~~\"", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors did not discuss potential negative impacts of their work. Even though the work presented here is not fully theoretical, it is high upstream of the applications. To study the possible negative impacts of this kind of work is to study the negative impacts of the whole field, since it is challenging the way we are evaluating meta-learning models. As any other field in machine learning, meta-learning could also be used for harmful applications, such as quick racial profiling of a user.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "I spent 8-10 hours to review this paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper examines few-shot meta-learning from the perspective of in-distribution vs out-of-distribution task splits, and demonstrates that optimising for one criterion may lead to worse performance on the other. This distinction has not been very carefully considered by the community before. While there are some concerns, reviewers unanimously agree that the theoretical and empirical analysis will be of value to the meta-learning and few-shot learning community. I agree and recommend accept.
Decision: Accept (Poster)