Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples
Authors: Eleni Triantafillou
Tyler Zhu
Vincent Dumoulin
Pascal Lamblin
Utku Evci
Kelvin Xu
Ross Goroshin
Carles Gelada
Kevin Swersky
Pierre-Antoine Manzagol
Hugo Larochelle
Venue:  (2020)
Keywords: few-shot learning
meta-learning
few-shot classification

Abstract: Few-shot classification refers to learning a classifier for new classes given only a few examples. While a plethora of models have emerged to tackle it, we find the procedure and datasets that are used to assess their progress lacking. To address this limitation, we propose Meta-Dataset: a new benchmark for training and evaluating models that is large-scale, consists of diverse datasets, and presents more realistic tasks. We experiment with popular baselines and meta-learners on Meta-Dataset, along with a competitive method that we propose. We analyze performance as a function of various characteristics of test tasks and examine the models’ ability to leverage diverse training sources for improving their generalization. We also propose a new set of baselines for quantifying the benefit of meta-learning in Meta-Dataset. Our extensive experimentation has uncovered important research challenges and we hope to inspire work in these directions.

Published as a conference paper at ICLR 2020
META-DATASEI: A DATASET OF DATASETS FOR LEARNING TO LEARN FROM FEW EXAMPLES
Eleni Triantafillou", Tyler Zhu', Vincent Dumoulin', Pascal Lamblin', Utku Evci Kelvin Xu Ross Goroshin', , Carles Gelada', Kevin Swersky Pierre-Antoine Manzagol* & Hugo Larochelle* "University Toronto and Vector Institute, Gooole AI, 'University of California, Berkeley Correspondence to: elen [cs. toronto. eddu
ABSTRACT
Few-shot classification refers to learning classifier for new classes given only few examples. While a plethora of models have emerged to tackle it, we find the procedure and datasets that are used to assess their progress lacking, To address this limitation, we propose evaluating models that is large-scale. consists of diverse datasets, and presents more realistic tasks. We experiment with popular baselines and meta-learners META-DATASET, along with competitive method that we propose. We analyze performance as a function of various characteristics of test tasks and examine the models' ability to leverage diverse training sources for improving their generalization. We also propose new set of baselines for quantifying the benefit of meta-learning in -ATASE.. Our extensive experimentation has uncovered important research challenges and we hope to inspire work in these directions.
DETTA--A a new benchmark for training and
INTRODUCTION
Few-shot learning refers to learning new concepts from few examples, an ability that humans naturally possess, but machines still lack. Improving on this aspect would lead to more efficient algorithms that can flexibly expand their knowledge without requiring large labeled datasets. We focus on few-shot classífication: classifying unseen examples into one of N new 'test' classes, given only few reference examples of each. Recent progress this direction has been made by considering a meta- peta--roblem: though we are not interested in learning about any training class in particular, we can exploit the training classes for the purpose of learning to learn new classes fromfew examples, thus acquiring a learning procedure that can be directly applied to new few-shot learning problems too.
This intuition has inspired numerous models of increasing complexity (see Related Work for some examples). However, we believe that the commonly used setup for measuring success in this direction lacking. Specifically, two datasets have emerged as de facto benchmarks for few-shot learning: Omniglot (Lake et al. 2015), and mini-ImageNNe (Vinyals al. (016), and believe that both of them are approaching their limit in terms of allowing one to discriminate between the merits of different approaches. Omniglot a dataset of 1623 handwritten characters from 50 different alphabets and contains 20 examples per class (character). Most recent methods obtain very high accuracy on Omniglot, rendering the comparisons between them mostly uninformative. mini-ImageNNet is formed out of 100 ImageNet (Russakovsky et al. 2015) classes (64/16/20 for train/validation/fet and contains 600 examples per class. Albeit harder than Omniglot, has the same property that most recent methods trained on it present similar accuracy when controlling for model capacity. We advocate that a more challenging and realistic benchmark required for further progress this area.
More specifically, current benchmarks: 1) Consider homogeneous learning tasks. In contrast, real-life learning experiences are heterogeneous: they vary in terms the number of classes and examples per class, and are unbalanced. 2) Measure only within-dataset generalization. However, are eventually after models that can generalize to entirely new distributions (e.g.. datasets). 3) Ignore the relationships between classes when forming episodes. Specifically, the coarse grained classification of dogs and chairs may present different difficulties than the fine- ame-ggained classification of dog breeds, and current benchmarks do not establish distinction between the two.

---

Published as a conference paper at ICLR 2020
DDATASD aims to improve upon previous benchmarks in the above directions: is significantly larger-scale and is comprised of multiple datasets of diverse data distributions; its task creation is informed by class structure for ImageNet and Omniglot; it introduces realistic class imbalance; and it varies the number of classes in each task and the size of the training set, thus testing the robustness of models across the spectrum from very-low-shot learning onwards.
The main contributions of this work are: 1) A more realistic, large-scale and diverse environment for training and lesting few-shot learners. 2) Experimental evaluation of popular models, and a new set of baselines combining inference algorithms of meta-learners with non-episodic training. 3) Analyses of whether different models benefit from more data, heterogeneous training sources. pre-trained weights, and meta veta-training.. 4) A novel meta-learner that performs strongly on META- -ATTAS........
FEW-SHOT CLASSIFICATION: TASK FORMULATION AND APPROACHES
Task Formulation The end-goal of few-shot classification is to produce a model which, given new learning episode with N classes and few labeled examples (Ai per class, C E ......NV N), able to generalize to unseen examples for that episode, In other words, the model learns from training (support) set S = [(11 y1), (X2, y2)..... (xx. 9K) (with K = =. k2) and is evaluated on a held-out test (query) set Q = [(x vi). ) (x2, vz)..... (X7, v/- Each example (x. y) is formed of an input vector E R. and a class label y E {I...., N). Episodes with balanced training sets (i.e., ke = k, Vc) are usually described as 'N-way, k-shot' episodes. Evaluation episodes are constructed by sampling their N classes from a larger set Crest of classes and sampling the desired number of examples per class.
is
A disjoint set Cirain of classes is available to train the model; note that this notion of training is distinct from the training that occurs within a few-shot learning episode. Few-shot learning does not preseribe specific procedure for exploiting Cirain, but a common approach matches the conditions in which the model is trained and evaluated (Vinyals et al. 2016). In other words, training often (but not always) proceeds in an episodic fashion. Some authors use training and testing to refer to what happens within any given episode, and meta-training and meta-testing to refer to using Ctrain to turn the model into learner capable of fast adaptation and Crest for evaluating its success to learn using few shots, respectively. This nomenclature highlights the meta-learning perspective alluded to earlier, but to avoid confusion we will adopt another common nomenclature and refer to the training and test sets of an episode as the support and query sets and to the process of learning from Crrain simply as training. We use the term ineta-learner' to describe a model that is trained episodically, i... learns to learn across multiple tasks that are sampled from the training set Cirain.
Non-episodic Approaches to Few-shot Classification A natural non-episodic approach simply trains a classifier over all of the training classes Caroin at once, which can be parameterized by a neural network with linear layer on top with one output unit per class. After training, this neural network is used as an embedding function y that maps images into a meaningful representation space. The hope of using this model for few-shot learning is that this representation space is useful even for examples of classes that were not included in training. It would then remain to define an algorithm for performing few-shot classification on top of these representations of the images of task. We consider two choices for this algorithm, yielding the 'K-NN' and 'Finetune" variants of this bascline.
Given a test episode, the 'k-NN' baseline classifies each query example as the class that its 'closest' support example belongs to. Closeness is measured by either Euclidean or cosine distance in the learned embedding space; choice that we treat as a hyperparameter. On the other hand, the 'Finetune' baseline uses the support set of the given test episode to train a new 'output layer' on top of the embeddings g, and optionally finetune those embedding too (another hyperparameter), for the purpose of classifying between the N new classes of the associated task.
A variant of the 'Finetune' baseline has recently become popular: Baseline++ (Chen et al., 2019). originally inspired by Gidaris & Komodakis (2018); :): Qi et al. (2018). It uses 'cosine classifier' as the final layer (62 (f-nomialg embeddings and weights before taking the dot product), both during the non-episodic training phase, and for evaluation on test episodes. We incorporate this idea in our codebase by adding a hyperparameter that optionally enables using a cosine classifier for the 'N-NNN (training only) and 'Finetune" (both phases) baselines.
2

---

Published as a conference paper at ICLR 2020
Meta-Learners for Few-shot Classification In the episodic setting, models trained end-to-end for the purpose of learning to build classifiers from few examples. We choose to experiment with Matching Networks (Vinyals et al., 2016), Relation Networks (Sung et al. 2018). Prototypical Networks (Snell et a 2017) and Model Agnostic Meta-Learning (MAMI., Finn et al. 2017) since
they cover diverse set of approaches few-shot learning. We also introduce novel meta-learner which is inspired by the last two models.
In each training episode, episodic models compute for each query example x* E 2, the distribution for its label p(v"""",") conditioned on the support set S and allow to train this differentiably- parameterized conditional distribution end-to-end via gradient descent. The different models are distinguished by the manner in which this conditioning on the support set is realized. In all cases, the performance on the query set drives the update of the meta-learner's s weights, which include (and sometimes consist only of) the embedding weights. We briefly describe each method below.
Prototy pictolyppcaliioo Networks Prototypical Networks construct prototype for each class and then classify each query example as the class whose prototype is 'nearest" to it under Euclidean distance. More concretely, the probability that query example x" belongs to class & is defined
exp( exp(-119(x)) c2 cx
p(y" = k|x",, 5) Σ El'EL.....N) exp( 9xx(-19(x") - C₁' (2))
where C₁ is the 'prototype' for class k: the average of the embeddings of class k's support examples.
Matching Networks Matching Networks (in their simplest form) label each query example as (cosine) weightee linear combination of the support labels:
ply" k|x", 8)
Σ a(x", x:) 1
where 14 is the indicator function and a(x", ) is the cosine similarity between g(x") and g(x.), over all support examples X;, where 1 < (8).
Relation Networks Relation Networks are comprised of an embedding function g as usual, and 'relation module" parameterized by some additional neural network layers. They first embed each support and query using and create a prototype Pe for each class by averaging its support embeddings. Each prototype Pe is concatenated with each embedded query and fed through the relation module which outputs a number in [0, 1] representing the predicted probability that that query belongs to class The query loss is then defined the mean square error of that prediction compared to the (binary) ground truth. Both and the relation module are trained to minimize this loss.
MAML MAML uses linear layer parametrized by W and b on top of the embedding function g(:00 and classifies a query example
p(y" xx" S) = softmax(b's + W'g(x";0))
where the output layer parameters W' and b' and the embedding function parameters 0' are obtained by performing a small number of within-episode training steps on the support set S, starting from initial parameter values (b, W. The model is trained by backpropagating the query set loss through the within- gradient descent procedure and into (b, W. This normally requires computing second-order gradients, which can be expensive to obtain (both in terms of time and memory). For this reason, an approximation is often used whereby gradients of the within-episode descent steps are ignored. This variant is referred to first-order MAML (fo-MAML) and was used in our experiments. We did attempt to use the full-order version, but found it to be impractically expensive (e.g., it caused frequent out-of tittoo--mmory problems).
Moreover, since in our setting the number of ways varies between episodes, b, W are set to zero and not trained (i.e., b', W are the result of within-episode gradient descent initialized at 0). leaving only to be trained. In other words, MAML focuses on learning the within-episode initialization 0 of the embedding network so that it can be rapidly adapted for new task.
3

---

Published as a conference paper at ICLR 2020
Introducing Proto-MAML We introduce novel meta-leamer that combines the complementary strengths of Prototypical Networks and MAML: the former's 's simple inductive bias that is evidently effective for very-few-shot learning, and the latter's flexible adaptation mechanism.
As explained by Snell et al. (0017), Prototypical Networks can be re-interpreted as linear classifier applied to learned representation g(x). The use of squared Euclidean distance means that output logits are expressed as
-Ilg(x") c²ll² = - g(xxxx g(xx)'xxxxx) + 20% g(x')((x)))))))()) c. c₁ = 2c g(xx)x(xx))))))()) - ||cx|| + constant
where constant is class-inde scalar which can be ignored, as it leaves output probabilities unchanged. The k-th unit of the equivalent linear layer therefore has weights Wx.- = 2cg and biases by -I which are both differentiable with respect to as they are function of g(::0).
We refer to fo-jJProtoMMMM. as the (fo-) model where the task-specific linear layer of each episode is initialized from the Prototypical Network -euuvveequ weights and bias defined above and subsequently optimized as usual on the given support set. When computing the update for 0, we allow gradients to flow through the Prototypical Networr-equivaleen linear layer initialization. We show that this simple modification significantly helps the optimization of this model and outperforms vanilla fo- MAML by a large margin on
3 META-DATASET: A NEW FEW-SHOT CLASSIFICATION BENCHMARK
META-DATASE aims to offer an environment for measuring progress realistic few-shot classifica- tion tasks. Our approach is twofold: 1) changing the data and 2) changing the formulation of the task (i.e., how episodes are generated). The following sections describe these modifications in detail. The code is open source and publicly available") !!]
3.1 META-DATASET" DATA
META-DATASET's data is much larger in size than any previous benchmark, and is comprised of multiple existing datasets. This invites research into how diverse sources of data can be exploited by meta-lcarner, and allows us to evaluate a more challenging generalization problem, to new datasets altogether. Specifically, META-DATASE ILSVRC-2012 (ImageNet, [Russakovsky et al.) 2015). Omniglot (Lake et al. (2015). Aircraft (Maji) ctt al. 2013), CUB-200-2011 (Birds, Wah et al. 2011), Describable Textures (Cimpoi et al. 2014). Quick Draw (Jongejan et al.) 2016), Fungi (Schroeder & Cui, 2018), VGG Flower (Nilsback & Zisserman 2008). Traffic Signs (Houben et al. 2013) and MSCOCO (Lin et al. 2014). These datasets were chosen because they free and easy obtain, span variety of visual concepts (natural and -mmae) and vary in how ane-grained the class definition is. More information about each of these datasets is provided in the Appendíx.
leverages data from the following 10 datasets:
To ensure that episodes correspond to realistic classification problems, each episode generated in META-
classes from single dataset. Moreover, two of these datasets, Traffic Signs and MSCOCO, are fully reserved for evaluation, meaning that no classes from them participate in the training set. The remaining ones contribute some classes to each of the training, validation and test splits of classes, roughly with 70% 15% / 15% proportions. Two of these datasets, ImageNet and Omniglot, possess class hierarchy that we exploit in META-DATASET. For each dataset, the composition of splits is available onlind
ImageNet ImageNet is comprised of 82,115 'synsets', i.e., concepts of the WordNet ontology, and uses
it provides 'is-a" relationships for its synsets, thus defining DAG over them. the 1K synsets that were chosen for the ILSVRC 2012 classification challenge and defines a new class split for it and a novel procedure for sampling classes from it for episode creation, both informed by its class hierarchy.
Specifically, construct a sub-graph of the overall DAG whose leaves are the 1K classes of ILSVRC- 2012. We then "cut' this sub-graph into three pieces, for the training, validation, and test splits,
4

---

Published as a conference paper at ICLR 2020
such that there is overlap between the leaves of any of these pieces. For this, we selected the synsets 'carnivore" and 'device' as the roots of the validation and test sub-graphs, respectively. The leaves that reachable from 'carnivore' and 'device" form the sets of the validation and test classes, respectively. All of the remaining leaves constitute the training classes. This method of splitting ensures that the training classes are semantically different from the test classes. We end up with 712 training, 158 validation and 130 test classes, roughly adhering to the standard 70/ /
15 15 (%) splits.
Omniglot This dataset is one of the established benchmarks for few-shot classification as men- tioned earlier. However, contrary to the common setup that flattens and ignores its two-level hierarchy of alphabets and characters, we allow it to influence the episode class selection in META-DATASET, yielding fine-grained tasks. We also use the original splits proposed in Lake et al. (2015): (all char- acters of) the 'background' and "evaluation' alphabets are used for training and testing, respectively. However, we reserve the 5 smallest alphabets from the 'background" set for validation.
3.2 EPISODE SAMPLING
In this section we outline META- DATASET's algorithm for sampling episodes, featuring hierarchically- aware procedures for sampling classes of ImageNet and Omniglot, and an algorithm that yields realistically imbalanced episodes of variable shots and ways. The steps for sampling an episode for given split are: Step 0) uniformly sample dataset D, Step 1) sample a set of classes C from the classes of D assigned to the requested split, and Step 2) sample support and query examples from
Step 1: Sampling the episode's class set This procedure differs depending on which dataset is chosen. For datasets without known class organization, we sample the 'way' uniformly from the where
is either 50 as many as there are available. Then we sample 'way' many classes uniformly at random from the requested class split of the given dataset. ImageNet and Omniglot use Klass-structure-awaa aware procedures outlined below.
range [5, MAX-CLASSER
ImageNet class sampling We adopt a bicrarchy-aware sampling procedure: First, we sample an internal (non-leaf) node uniformly from the DAG of the given split. The chosen set of classes is then the set of leaves spanned by that node (or a random subset of it, if more than 50). We prevent nodes that are too close to the root to be selected as the internal node, as explained in more detail in the Appendix. This procedure enables the creation of tasks of varying degrees of fine-grainedness: the larger the height of the internal node, the more coarse-grained the resulting episode.
Omniglot class sampling We sample classes from Omniglot by first sampling an alphabet uni- formly at random from the chosen split of alphabets (train, validation or test). Then, the "way' of the episode is sampled uniformly at random using the same restrictions as for the rest of the datasets, but taking care not to sample a larger number than the number of characters that belong to the chosen alphabet. Finally, the prescribed number of characters of that alphabet are randomly sampled. This ensures that each episode presents a within-alphabet fine-grained classification.
Step Sampling the episode's examples Having already selected a set of classes, the choice of the examples from them that will populate an episode can be broken down into three steps. We provide bigh-level description here and elaborate the Appendix with the accompanying formulas.
Compute the query set size The query set is class-balanced, reflecting the fact that we care equally to perform well on all classes of an episode. The number of query images per class is set to a number such that all chosen classes have enough images to contribute that number and still remain with roughly half on their images to possibly add to the support set (in a later step). This number is capped to 10 images per class.
Step 2a:
Step 2b:
Compute the support set size We allow each chosen class to contribute to the support set at most 100 of its remaining examples (i.e., excluding the ones added to the query set). We multiply this remaining number by a scalar sampled uniformly from the interval (0, 1] to enable the potential available, we also interested
generation of 'few-shot' episodes even when multiple images in studying that end of the spectrum. We do enforce, however, that each chosen class has a budget for at least one image in the support set, and cap the total support set size 500 examples.
5

---

Published as a conference paper at ICLR 2020
Step 2c: Compute the shot of each class We now discuss how to distribute the total support set size chosen above across the participating classes. The un-normalized proportion of the support set that will be occupied by given chosen class is a noisy version of the total number of images of that class in the dataset. This design choice is made in the hopes of obtaining realistic class ratios, under the hypothesis that the dataset class statistics are reasonable approximation of the real- col-w statistics of appearances of the corresponding classes. We ensure that each class has at least one image in the support set and distribute the rest according to the above rule.
After these steps, we complete the episode creation process by choosing the prescribed number of examples of each chosen class uniformly at random to populate the support and query sets.
RELATED WORK
In this work evaluate four meta-leamers diversity of well-established models. Evaluating other few-shot classifiers on META-DATASET is beyond the scope of this paper, but we discuss some additional related models below.
that believe capture a good
Similarly to MAML, some train a meta-leame for quick adaptation to new tasks (Ravi & Larochelle 2017; Munkhdalai & Yu 2017: Rusu et al.) [019]] Yoon et al. 2018). Others relate to Prototypical Networks by learning a representation on which differentiable training can be performed on some form of classifier (Bertinetto al.) 2019 Gidaris & Komodakis 2018 Oreshkin al. 2018 Others relate to Matching Networks in that they perform comparisons between pairs of support and query examples, using either a graph neural network (Satorras & Estrach| 2018) or an attention mechanism (Mishra recurrent networks (Santoro et al.
et al., 2018). Finally, some make use of ceenorr-augmente 2016), some learn to perform data augmentation (Hariharan & Girshick 2017 Wang et al. 2018) in low-shot learning setting, and some learn to predict the parameters of lhrge-shot classifier from the parameters learned in a few-shot setting (Wang& & Hebert 2016 Wang et al. 2017). Of relevance to Proto-MAML is MAML+ (Antoniou et al. 2019), which consists of collection of adjustments to MAML. such as multiple meta-trained inner loop learning rates and derivative-order annealing. Proto-MAM instead modifies the output weight initialization scheme and could be combined with those adjustments.
Finally, DATTA-DTAASTTTTT relates to other recent image classification benchmarks. The CVPR 2017 Visual Domain Decathlon Challenge trains a model on 10 different datasets, many of which included in our benchmark, and measures its ability to generalize held-out examples for those same datasets but does not measure generalization to new classes (or datasets). Hariharan Girshick (017)) propose benchmark where model is given abundant data from certain base ImageNet classes and is tested on few-shot learning novel ImageNet classes in a way that doesn't 't compromise its knowledge of the base classes. |Wang al. (2018) build upon that benchmark and propose a new evaluation protocol for it. [Chen et al. (2019) investigate fine- fine-grained few-shot classification using the CUB dataset (Wah et 2011 also featured in our benchmark) and cross- domain transfer between mini- ImageNe and CUB. Larger-scale few-shot classification benchmarks were also proposed using CIFAR- 100 (Krizhevsky et al. 2009; Bertinetto et al. 2019 Oreshkin et al., 2018), fiered-ImageNet (Ren et al. 2018), and ImageNet-21k (Dhillon et al. 2019). Compared to these,
contains the largest set of diverse datasets in the context of few-shot learning and is additionally accompanied by algorithm for creating learning scenarios from that data that we advocate are more realistic than the previous ones.
5 EXPERIMENTS
Training procedure META-DATASEE does not prescribe a procedure for learning from the training data. In these experiments, keeping with the spirit of matching training and testing conditions, we trained the meta-learners via training episodes sampled using the same algorithm as we used for evaluation episodes, described above. The choice of the dataset from which to sample the next episode was random uniform. The non-episodic baselines are trained to solve the large classification problem that results from 'concatenating' the training classes of all datasets.
Validation Another design choice was to perform validation on (the validation split of) ImageNet
only, ignoring the validation sets of the other datasets. The rationale behind this choice is that the

---

Published as a conference paper at ICLR 2020
Table Few-shot classification results on only (top) and trained on all datasets (bottom).
using models trained on ILSVRC-2012
Test
k-NN
MatchingNet
ProcoNet
fo-MAML
RelationNet
io-Proto-MAML
II SVRC
41.03
45.78
45,00
50.50
45.51
34.69
49.53
Onniglot
37,07
52.27
59.98
Aircraft
46.81
68.69
48.97
53.10
56.24
40.73
55,95
Birds
50.13
57.31
62.21
68.79
63.61
49.51
68.66
Textures
66.36
69.08
64.15
66.56
68.04
52.97
06.49
Quick Draw
32.06
42,60
42.87
48.96
43.96
43.30
51,52
Fungi
36.16
38.20
33.97
39.71
32.10
30.55
39.96
VGG Flower
83.10
85.51
80.13
85.27
81.74
68.76
87.15
Traftic Signs
47.80
47.12
50.93
MSCOCO
30.38
34,86
34.99
41.00
35,30
29.15
43.74
Avg.
4.65
2.65
Test Source
K-NN
Finetune
MatchinggNt
ProtoNet
To 1--MLL
RelationNet
Io- Oo-Proo-MAALL
SVCC
38,55
43.08
36.08
44.50
30.89
Onniglot
74,60
71.11
78.25
79.56
83,92
86.57
82.69
Aircraft
64.98
72,03
69.17
71.14
76.41
69.71
73.23
Birds
66.35
59.82
56.40
67.01
62,43
54.14
69.88
Textures
63.38
69.14
61.80
68.18
56.50
68.25
Quick Draw
44.88
47.05
60.81
64.88
59.73
61.75
66.84
Fungi
37.12
38.16
33,70
40.26
33.54
32.56
41.99
VGG Flower
83.47
$1.90
79,94
76.08
88.72
Traflic Signs
40.11
66.74
55.57
46.48
42.91
37.48
52.42
MSCOCO
29.55
33.17
28.79
39.87
29.37
27.41
41.74
Avg. rank
5.05
3,6
4.95
2,85
4,25
5.8
1.5
performance on ImageNet has been known to be good proxy for the performance on different datasets. We used this validation performance select our hyperparameters, including backbone architectures, image resolutions and model-specific ones. We describe these further in the Appendix.
Pre-training We gave each meta-learner the opportunity to initialize its embedding function from the embedding weights to which the k-NN Baseline model trained on ImageNet converged to. We treated the choice of starting from scratch or starting from this initialization as a hyperparameter. For a fair comparison with the baselines, we allowed the non-episodic models to start from this initialization too. This is especially important for the baselines in the case of training on all datasets since it offers the opportunity to start from imagenet-pretrained weights.
Main results Table displays the accuracy of each model on the test set of each dataset, after they were trained ImageiNetonlly all datasets. Traffic Signs and MSCOCO in either case, as they are reserved for evaluation. We propose to use the average (over the datasets) rank of each method our metric for comparison, where smaller is better. A method receives rank 1 it has the highest accuracy, rank 2 if has the second highest, and so on. If two models share the best accuracy, they both get rank 1.5, and so on. We find that Yo-Proto-MAML the top-performer according to this metric, Prototypical Networks also perform strongly, and the Finetune Baseline notably presents a worthy opponen We include more detailed versions of these tables displaying confidence intervals and per-dataset ranks in the Appendix.
not used for training
Effect of training on all datasets instead of ImageNet only It's interesting to examine whether training on (the training splits of) all datasets leads to improved generalizaton compared to training on (the training split of) ImageNet only. Specifically, while we might expect that training on more data helps improve generalization, it an empirical question whether that still holds for heterogeneous data. We can examine this by comparing the performance of each model between the top and bottom sets of results of Table corresponding to the two training sources (ImageNet only and all datasets, respectively). For convenience, Figure visualizes this difference in barplot. Notably, for Omniglot, Quick Draw and Aircraft we observe a substantial increase across the board from training on all
We improved MAML's performance significantly in the time between the acceptance decision and publi- cation, thanks to suggestion to use more aggressive inner-loop learning rate. In the Appendix. we present the older MAMI. results side by side with the new ones, and discuss the importance of this hyperparameter for MAMI..
7

---

Published as a conference paper at ICLR 2020
50
ILSVRC
Omniglot
Aircraft
CU Birds
Textures
30
K-NN baseline
20
Finetune baselina
MatchingNet
-10
ProtoNet
-20
fo-MAML
50
Quick Draw
Fungi
VGG Flower
Trafficc Sign..
MSCDCO
RelationNett
fo-Proto-MAre
40
10
-10
-20
Figure The performance difference datasets, when training all datasets instead of ILSVRC only. A positive value indicates an improvement from all-dataset training.
sources. This is reasonable for datasets whose images are significantly different from ImageNet's ss we indeed expect to gain large benefit from training some images from (the training classes of) these datasets. Interestingly though, on the remainder of the test sources, we don't 't observe a gain from all-dataset training. This result invites research into methods for exploiting heterogeneous data for generalization to unseen classes of diverse sources. Our experiments show that learning 'naively' across the training datasets (e.g., by picking the next dataset to uniformly random) does not automatically lead to that desired benefit in most cases.
I-JN baseine
0.8-
Matching
Prototet
Accuracy
0.6
Relstionllet
[o-MMLL
Class Precision
0.6
Po-Proto-MAM MAML
0.4
K-W basel
Finetune buseline
Mal chinhhet
0.2
ProtoNet
MANL
0.0
10
30
50
20
40
60
100
Way
Shot
(a) Ways analysis
(b) Shots analysis
Figure 2: The effect of different ways and shots on test performance (w/ 95% confidence intervals) when training on ImageNet.
Ways and shots analysis We further study the accuracy as a function of 'ways' (Figure 2a and the class precision as function of 'shots' (Figure As expected, we found that the difficulty increases as the way increases, and performance degrades. More examples per class, on the other hand, indeed make it easier to correctly classify that class. Interestingly. though, not all models benefit at the same rate from more data: Prototypical Networks and fo- --oM-ott-MMMLL outshine other models in very-low-shot settings but saturate faster, whereas the Finetune baseline, Matching Networks, and fo-MAML improve at a higher rate when the shot increases. We draw the same conclusions when performing this analysis all datasets, and include those plots in the Appendix. As discussed in the Appendix, we recommend including this analysis when reporting results on Meta-Dataset, aside from the main table. The rationale is that we're only interested in performing well average, but also in performing well under different specifications of test tasks.
Effeet of pre-training In Figures 3al and 3b we quantify how beneficial is to initialize the embedding network of meta-leamers using the weights of the k-NN baseline pre-trained on ImageNet, as opposed to starting their episodic training from scratch. We find this procedure to often be beneficial, both for ImageNet-only training and for training on all datasets. It seems that this ImageNet-influnnce initialization drives the meta-learner towards a solution which yields increased performance natural image test datasets, especially ILSVRC, Birds, Fungi, Flowers and MSCOCO.
8

---

Published as a conference paper at ICLR 2020
IL9VRC
Omn
Aircoott
OU Binds
Tectunes
ILSVRC
Omeiglic
Aircratt
CU Birds
Textures
Muschighic
30
MAML
aotttric Rot
Proto MAM
Quickk
Fungl
VGG Flower
Traffie Sign.
MGCOCD
100
Dew
VIG Flnwer
Traftr
MEDOGG
Inka Ization
tre-buiral
from scratch
(a) Effect of pre- pre-training (ImageNet)
(b) Effect of pre- pre-training (All datasets)
LSVVR
Omnrigtee
Aircratt
CU Birds
Textures
ILSVRC
Oonigglor
Binds
Textures
70
Model Maichinghet
10
to Prote MWML
ProtoNct
18
Qutck Dram
Fungi
VSG Flawer
Traffccigg..
MSCOCC
QUICk DYaW
Fungi
VSG Flower
Traffic Sign.
NSCOCO
40
10
20
lrariigg
only
(c) Effect of meta-learning (ImageNet)
(d) Effect of meta-learning (All datasets)
Figure 3: The effects of pre-training and meta- meta-training (w/ 95% confidence intervals). (ImageNet) or (All datasets) the training source.
Perhaps unsusprisingly, though, underperforms on significantly different datasets such as Omniglot and Quick Draw. These findings show that, aside from the choice of the training data source(s) (e.g.. ImageNet only or all datasets, as discussed above), the choice of the initialization scheme can also influence to important degree the final solution and consequently the aptness of applying the resulting meta-learner to different data sources at test time. Finally, an interesting observation is that MAML seems benefit the most from the pre-trained initialization, which may speak to the difficulty of optimization associated with that model.
Effect of meta- veta-training We propose to disentangle the inference algorithm of each meta-leamner from the fact that it is meta-learned, to assess the benefit meta-learning on META-DATASER To this end, we propose a new set of baselines: 'Prototypical Networks Inference', 'Matching Networks Inference', and Prroo-rMMM Inference', that are trained non-episodically but evaluated episodically (for validation and testing) using the inference algorithm of the respective meta-leamner This is possible for these meta-learners as they don't 't have any additional parameters aside from the embedding function that explicitly need to be learned episodically (as opposed to the relation module of Relation Networks, for example). We compare each Inference-only method to its corresponding meta-learner in Figures[3eand]] 3d We find that these baselines are strong: when training on ImageNet only, we can usually observe small benefit from meta-learning the embedding weights but this benefit often disappears when training on all datasets, which case meta- learning sometimes actually hurts. We find this result very interesting and we believe it emphasizes the need for research on how to leta-learn across multiple diverse sources, an important challenge that META-DATASET puts forth.
Fine-grainedness analysis We use ILVRC-2012 to investigate the hypothesis that finer-grained tasks are harder than coarse-grained ones. Our findings suggest that while the test sub-graph is not rich enough exhibit any trend, the performance on the train sub-graph does seem to agree with this hypothesis. We include the experimental setup and results for this analysis in the Appendix.
CONCLUSION
We have introduced a new large-scale, diverse, and realistic environment for few-shot classification. We believe that our exploration of various models META-DATASET has uncovered interesting

---

Published as a conference paper at ICLR 2020
directions for future work pertaining meta-learning across heterogeneous data: it remains unclear what the best strategy for creating training episodes, the most appropriate validation creation and the most appropriate initialization. Current models don't 't always improve when trained on multiple sources and meta-learning is not always beneficial across datasets. Current models are also not robust the amount of data in test episodes, each excelling in different part of the spectrum. We believe that addressing these shortcomings consitutes an important research goal moving forward.
AUTHOR CONTRIBUTIONS
Eleni, Hugo, and Kevin came up with the benchmark idea and requirements. Eleni developed the core of the project, and worked on the experiment design and management with Tyler and Kevin, well as experiment analysis. Carles, Ross, Kelvin, Pascal, Vincent, and Tyler helped extend the benchmark by adding datasets. Eleni, Vincent, and Utku contributed the Prototypical Networks, Matching Networks, and Relation Networks implementations, respectively. Tyler implemented baselines. MAML (with Kevin) and Proto-MAML models, and updated the backbones support them. Writing was mostly led by Eleni, with contributions by Hugo, Vincent, and Kevin and help from Tyler and Pascal for visualizations. Pascal and Pierre-Antoine worked on code organization, efficiency, and open-soureing. Pascal and Vincent optimized the efficiency of the data input pipeline. code development process and reviewed most the changes, Hugo and Kevin supervised the overall direction of the research.
supervised the
ACKNOWLEDGMENTS
We would like to thank Chelsea Finn for fruitful discussions and advice on tuning fo-MAML and ensuring the correctness of implementation, as well as Zack Nado and Dan Moldovan for the initial dataset code that was adapted, and Cristina Vasconcelos for spotting an issue in the ranking of models. Finally, we'd like to thank John Bronskill for suggesting that we experiment with a larger inner-loop learning rate for MAML which indeed significantly improved our fo-MAML results on -DTTA-DT
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Antreas Antoniou, Harrison Edwards, and Amos Storkey. How to train your MAMI.. In Proceedings of the International Conference on Learning Representations, 2019.
Luca Bertinetto, Joao F. Henriques, Philip Torr, and Andrea Vedaldi. Meta-learning with differentiable closed-form solvers, In Proceedings of the International Conference on Learning Representations, 2019.
Wei-Yt Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang. and Jia- Huang. closer look at few-shot classification. In Proceedings of the International Conference Learning Representations, 2019.
M. Cimpoi, Maji, I. Kokkinos, Mohamed, and A. Vedaldi. Describing textures in the wild. In IEEE Conference Computer Vision and Pattern Recognition, 2014.
Guneet S. Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. A baseline for few-shot image classification. arXiv, abs/1909.02729, 2019.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leariing for fast adaptation of deep networks. In Proceedings of the International Conference of Machine Learning, 2017.
Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In IEEE Conferencee on Computer Vision and Putlern Recognition, 2018.
Bharath Hariharan and Ross Girshick. Low-shot visual recognition by shrinking and hallucinating features. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3018-3027, 2017.
Sebastian Houben, Johannes Stallkamp, Jan Salmen, Marc Schlipsing, and Christian Igel. Detection of traffic signs in real- world images: The German Traffic Sign Detection Benchmark. International Joint Conference on Neural Networks, 2013.
Jonas Jongejan, Henry Rowley, Takashi Kawashima, Jongmin Kim, and Nick Fox-Gieg. The Quick, Draw! - A.I. experiment. guickdraw htt.kkdrwwwwwtthhog...... wwthgoogge......oooooooooooooo com, 2016.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 33066266): 2015.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In European Conference on Computer Vision, pp. 740-755, 2014.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi, Fine-grained visual classification of aircraft. arXiv, abs/ ht.//1306.5511.. 2013.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta- learner. In Proceedings of the International Conference on Learning Representations, 2018.
Tsendsuren Munkhdalai and Hong Yu. Meta networks. In Proceedings the International Conference on Machine Learning, pp. 2554 2563, 2017.
M-E, Nilsback and A. Zisserman. Automated flower classification over large number of classes. In Proceedings of the Indian Conference on Computer Vision, Graphics and Image Processing, 2008.
Boris N. Oreshkin, Pau Rodriguez, and Alexandre Lacoste. TADAM: Task dependent adaptive metric for improved few-shot learning. In Advances in Neural Information Processing Systems, pp. 719-729, 2018.
Hang Qi. Matthew Brown, and David G Lowe. Low-shot learning with imprinted weights. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5822-5830, 2018.
11

---

Published as a conference paper at ICLR 2020
Sachin Ravi and Hugo Larochelle. Optimization model for few-shot learning. In Proceedings of the International Conference on Learning Representations, 2017.
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenenbaum, Hugo Larochelle, and Richard Zemel. Meta-learning for semi-supervised few-shot classification. In Proceedings of the International Conference on Learning Representations, 2018.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):21 1-222,, 2015.
Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-leaming with latent embedding optimization. In Proceedings of the International Conference on Learning Representations, 2019.
Tim Salimans and Durk P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In Advances in Neural Information Processing Systems, pp. 901-909, 2016.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy Lillicrap. Meta- learning with memory--ugmenter neural networks. Proceedings of the International Conference Machine Learning, pp. 1842 1850, 2016.
Victor Garcia Satorras and Joan Bruna Estrach. Few-shot learning with graph neural networks. In Proceedings of the International Conference on Learning Representations, 2018.
Brigit Schroeder and Yin Cui. FGVCx fungi classification challenge 2018. github. com/
vi s sidipedia/faaee http///acx__uggiiiiiaeaeeaa (anng__coppp 2018.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems. pp. 4077-4087, 2017.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales.
Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1199-1208, 2018.
Oriol Vinyals, Charles Blundell, Tim Lillicrap, and Daan Wierstra. Matching networks for one shot learning. In Advances in Neural Information Processing Systems, pp. 3630-3638, 2016.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD Birds-200-22 200-2011 Dataset. Technical Report CNS-TR-2011-40 California Institute of Technology, 2011.
Yu-Xiong Wang and Martial Hebert. Learning to learn: Model regression networks for easy small sample learning. In European Conference on Computer Vision, pp. 616-634. Springer, 2016.
Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. Learning to model the tail. In Advances in Neural Information Processing Systems, pp. 7029-7033 2017.
Yu-Xiong Wang, Ross Girshick, Martial Hebert, and Bharath Hariharan. Low-shot learning from imaginary data. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recogni- tion, pp. 7278 7286, 2018.
Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-learning. In Advances in Neural Information Processing Systems, 2018.
12

---

Published as a conference paper at ICLR 2020
APPENDIX
.1 RECOMMENDATION FOR REPORTING RESULTS ON META-DATASET DATASET
We recommend that future work META-DATASER reports two of results:
The main tables storing the average (over 600 test episodes) accuracy each method on each dataset, after it has been trained on ImageNet only and on All datasets, where the evaluation metric is the average rank. This corresponds to Table in our case (or the more complete version in Table 2 in the Appendix).
2. The plots that measure robustness in variations of shots and ways. In our case these are Figures 2h) and 2a in the main text for ImageNet-only training, and Figures 5b and 5a in the Appendix for the case of training on all datasets.
We propose use both of these aspects to evaluate performance desirable to perform well on average, but also to perform well under different specifications of test tasks, as it is not realistic in general to assume that will know in advance what setup (number of ways and shots) will be encountered at test time. Our final source code will include scripts for generating these plots and for automatically computing ranks given a table to help standardize the procedure for reporting results.
-DTTA-DTAAST: it is not only
.2 DETAILS OF SETA-DATASEET'S SAMPLING ALGORITHM
We now provide a complete description of certain steps that were explained on higher level in the main paper.
STEP 1: SAMPLING THE EPISODE'S CLASS SET
ImageNet class sampling The procedure we use for sampling classes for an ImageNet episode is the following. First, we sample node uniformly at random from the set of 'eligible' nodes of the DAG structure corresponding to the specified split (train, validation or test). An internal node than 392 leaves. The number 392
"eligible" for this selection if it spans at least 5 leaves, but was chosen because it is the smallest number so that, collectively, all eligible internal nodes span all leaves in the DAG. Once eligible node is selected, some of the leaves that spans will constitute the classes of the episode. Specifically, if the number of those leaves is no greater than 50, we use all of them. Otherwise, we randomly choose 50 of them.
This procedure enables the creation of tasks of varying degrees of fine-grainedness. For instance, the sampled internal node has small height, the leaf classes that spans will represent semantically- related concepts, thus posing a fine- tine-grinee classification task. As the height of the sampled node increases, we 'zoom out' to consider broader scope from which we sample classes and the resulting episodes are more coarse- -aanse.graind..
STEP 2: SAMPLING THE EPISODE' EXAMPLES
a) Computing the query set size The query set class-balanced, reflecting the fact that we care equally to perform well on all classes of an episode. The number of query images per class is computed as:
g min {
{10, 10,
min [0.5 rec
where C is the set of selected classes and Im(c) denotes the set of images belonging to class c. The min over classes ensures that each class has at least images to add to the query set, thus allowing it to be class-balanced. The 0.5 multiplier ensures that enough images of each class will be available to add the support set, and the minimum with 10 prevents the query set from being too large.
b) Computing the support set size
We compute the total support set size as:
SS| min
500,
Σ
[ min min (100, [100,]Im(c) gt1
oEc
13

---

Published as a conference paper at ICLR 2020
where 3 is scalar sampled uniformly from interval (0, 1]. Intuitively, each class average contributes either all remaining examples (after placing q of them in the query set) if there are less than 100 or 100 otherwise, to avoid having too large support sets. The multiplication with 3 enables the potential generation of smaller support sets even when multiple images are available, since we are also interested in examining the very-low-shot end of the spectrum. The 'ceiling' operation ensures that each selected class will have at least one image in the support set. Finally, we cap the total support set size 500.
c) Computing the shot of each class We are now ready compute the 'shot' of each class. Specifically, the proportion of the support set that will be devoted class is computed as:
exp(a.)[Im(e)
Re
exp(d')/m(c))
c'Ec
where a, is sampled uniformly from the interval (og()))).oog(2). Intuitively, the un un-normalized proportion of the support set that will be occupied by class e is noisy version of the total number of images of that class in the dataset [m(c). This design choice is made in the hopes of obtaining realistic class ratios, under the hypothesis that the dataset class statistics are reasonable approximation of the real-world statisties of appearances of the corresponding classes. The shot of class is then set to:
he min Re (SS |C]) + 1, [m((c)] }}
which ensures that least one example is selected for each class, with additional examples selected proportionally to Re, if enough are available.
.3 DATASETS
META BETTA-DANE DATASET formed of data originating from 10 different image datasets. A complete list of the datasets we use is the following.
2
15
5
&
H
1SA
(a) ImageNet
(b) Omniglot
(c) Aircraft
(d) Birds
(e) DTD
(f) Quick Draw
(g)
Fungi
(h) VGG Flower
ii) Traffic Signs
(i) MSCOCO
Figure 4: Training examples taken from the various datasets forming META-DATASER DATASET.
ILSVRC-2012 (ImageNet, Russakovsky et al., 2015) A dataset of natural images from 1000 categories (Figure 4ah. We removed some images that were duplicates of images in another dataset in
META-DATASET (43 images that were also part of Birds) other standard datasets of interest (92 from Caltech-101 and 286 from Caltech-256) The complete list duplicates is part of the source code release.
Omniglot (Lake et al. 2015) A dataset images of 1623 handwritten characters from 50 different alphabets, with 20 examples per class (Figure 4b). While recently Vinyals et al. (016)) proposed
14

---

Published as a conference paper at ICLR 2020
new split for this dataset, we instead make use of the original intended split Lake al. (2015) which is more challenging since the split is on the level of alphabets (30 training alphabets and 20 evaluation alphabets), not characters from those alphabets, therefore posing more challenging generalization problem. Out of the 30 training alphahets, we hold out the smallest ones (i.e., with the least numher of character classes) form our validation set, and use the remaining 25 for training.
Aircraft (Maji al. 2013 A dataset of images of aircrafts spanning 102 model variants, with
100 images per class (Figure 4c) The images are cropped according to the providing bounding boxes, in order not include other aircrafts, the copyright text the bottom of images.
2011) A dataset for fine-graine classification of 200 different CUB-200-2011 (Birds, Wah et al.
bird species (Figure 4d). . We did not use the provided bounding boxes to crop the images, instead the full images used, which provides harder challenge.
Describable Textures (DTD, Cimpoi et al. 2014) A texture database, consisting of 5640 images, organized according to a list of 47 terms (categories) inspired from human perception (Figure 4e).
dataset 50 million black-and and-white drawings across 345 categories, contributed by players of the game Quick, Draw! (Figure 4f).
Quick Draw (Jongejan et al. 2016
Fungi (Schroeder & Cui 2018 A large dataset of approximately 100K images of nearly 1,500
wild mushrooms species (Figure
VGG Flower (Nilsback & Zisserman, 2008 A dataset of natural images of 102 flower categories.
The flowers chosen to be ones commonly occurring in the United Kingdom. Each class consists of between 40 and 258 images (Figure
Traffic Signs (Houben al., 2013) A dataset of 50,000 images of German road signs in 43 classes (Figure
MSCOCO Lin et al. 2014) A dataset of images collected from Flickr with 1.5 million object instances belonging to 80 classes labelled and localized using bounding boxes. We choose the train2017 split and create images crops from original images using each object instance's groundtruth bounding box (Figure
.4 HYPERPARAMETER:
We used three architectures: a commonly-used four-layer convolutional network, an 18-layer residual network and wide residual network. While some of the baseline models performed best with the latter, we noticed that the meta-learners preferred the resnet-18 18 backbone and rarely the four-layer- convnet. For Relation Networks only, also allow the option the aforementioned three, inspired by the four-layer-convee used in the Relation Networks paper (Sung et al. 20118). The main difference is that they used the usual max-poo max-pooling operation only the first two layers, omitting it in the last two, yielding activations of larger spatial dimensions. In our case, we found that these increased spatial dimensions did not in memory, so as compromise we used max-pooling on the first 3 out of the 4 layer of the convnet.
another architecture, aside from
For fo-MAML and fo-Prrto-MAMIL MAML, we tuned the inner-loop learning rate, the number of inner loop steps, and the number of additional such steps to be performed in evaluation (i.e., validation or test) episodes.
For the baselines, we tuned whether the cosine classifier of Baseline++ will be used, as opposed to a standard forward pass through linear classification layer. Also, since Chen et al. (2019) added weight normalization (Salimans & Kingma 2016) to their implementation of the cosine classifier layer, we also implemented this and created a hyperparameter choice for whether or not is enabled. This hyperparameter is independent from the one that decides if the cosine classifier is used. Both are applicable to the N--N Basline (for its all-way training classification task) and to the Finetune Baseline (both for its all-way training classification and for its within-episode classification
15

---

Published as a conference paper at ICLR 2020
validation and test times). For the Finetune Baseline, tuned binary hyperparameter deciding if gradient descent or ADAM is used for the within-task optimization. We also tuned the decision of whether all embedding layers finetuned alternatielly, the embedding held fixed and only the final classifier on top of is optimized. Finally, we tuned the number of finetuning steps that will be carried out.
We also tried two different image resolutions: the commonly-used 84x84 and 126x 126. Finally, we tuned the learning rate schedule and weight decay and used ADAM train all of models. All other details, dataset splits and the complete set of best hyperparameters discovered for each model included in the source code.
.5 COMPLETE MAIN RESULTS AND RANK COMPUTATION
Rank computation We rank models by decreasing order of accuracy and handle ties by assigning tied models the average of their ranks. A tie between two models occurs when a 95% confidence interval statistical test on the difference between their mean accuracies is inconclusive rejecting the null hypothesis that this difference is O. Our recommendation is that this test is ran to determine if ties occur. As mentioned earlier, our source code will include this computation.
Complete main tables For completeness, Table 2 also displays confidence intervals and per-dataset ranks computed using the above procedure.
presents more detailed version of Table that
-6 ANALYSIS OF PERFORMANCE ACROSS SHOTS AND WAYS
For completeness, in Figure we show the results the analysis of the robustness to different ways and shots for the variants of the models that were trained on all datasets. We observe the same trends as discussed in our Experiments section for the variants of the models that were trained on ImageNet.
baseline
Finetune baseline
MatchingNet
ProtoNet
Precision
Accuracy
RelationNet
0.6
K-NN baseline
MatchingNet
0.2
Protolk et
RelationNet
fo-Proto-MAMI
10
20
30
40
50
0.0
20
40
60
80
100
Way
(a) Ways Analysis
(b) Shots Analysis
Figure 5: Analysis of performance as a function of the episode's 's way, shots for models whose training source is (the training data of) all datasets. The bands display 95% confidence intervals.
.7 EFFECT OF TRAINING ON ALL DATASETS OVER TRAINING ON ILSVRC-2012 ONLY
For more clearly observing whether training on all datasets leads to improved generalization over training on ImageNet only, & side-to-side the performance of each model trained on IISVVC only vs. datasets. The difference between the performance of the all-dataset trained models versus the ImageNet-only trained ones is also visualized in Figure in the main paper.
As discussed in the main paper, we notice that we not always observe clear generalization advantage in training from wider collection of image datasets. While some of the datasets that were added to the meta-training phase did see an improvement across all models, in particular for Omniglot and Quick Draw, this not true across the board. In fact, in certain cases the performance is slightly worse. We believe that more successfully leveraging diverse sources of data is an interesting open research problem.
16

---

Published as a conference paper at ICLR 2020
Proto-MAMI
19.5311.05 (1.5)
63.37 1..7:113300 1.33 (1)
55.95 (2.5)
68.66 68.66 0.96 (1.5)
06.49 (4)
51.52+ 1.02
39.96 1 1.14
87.15 (1)
1. (3.5)
43.7411.12 (1)
Proo-MAMI
46.52 1.05 (1)
82.69 (2.5)
75.2310..6 (2)
69.88
0..2
66.84+0094
41.99
0.67 (1)
1.
(3)
417441.13
1.85
1.5
62.4311.08 (4) 54.14110.99 (7) 69.88
66.
62.21+0955 (3.5) 68.79 1.01 (1.5) 63.61 (3.5) 49.51 (6.5)
43.30 (4.5)
L.0 (6.5)
76.411:0.99 (I) 69.71 (5.5)
61.75 (3.5)
RelationNer
45.78 1.10 (4) 45.0011.110 (4) 50.50 1.08 (1.5) 45.51 (4) 34.6911..0 (7)
45.35 (6)
40.73 (7)
52.97 (7)
1..55-11004 (7)
68.76
33.67#1.0500
1.11
RelationNer
36.08 1 1.00 (6) 44.50 1.0 (2.5) 37,83 1.01 (4.5) 30.89 0..3 (7)
83.921 (2.5) 86.57 86.57 L0.79 (1)
64. (4.14.1..3 (3.5) 56.56 (7)
32.56+1.000 1.08 (6)
76.08 +0.76 (7)
37.48 (7)
27.41 (7)
6.55
5.8
30.55
56.24 (2.5)
08.04 (1.5)
43.961 1229 (4.5)
52.27 (5) 59.9811 1.55 (2.5) 55.55 54 (4)
32.10E1.10
81.7410.83 (5)
50.93 (2)
35.30 (4)
59.73+
33.54+1.11 (6)
79.9410.84 (6)
42.91 (5) (5)
29.371 1.08 (5)
fo-MAML
fo-MAMI
Method: Accuracy (%) confidence (%) (rank)
Table 2: Few-shot classification results on META- DATASET.
3.7
4.25
MatchingNet Method: Accuracy (%) = confidence (%)
83
Models trained on ILSVRC- 2012 only.
48.97 (S) 53.1011.00 (4)
64.15 (6) 66.56-10.33 (4)
42.87 (4.5) 48. 18.99111.08 (2)
33.97 1.00 (5) 39.71E1.11 (15)
85.27-L0.77 (.5)
47.80+1.114 (3.5) 47.1241.1015
34.991 1.00 (4) 41.00 (2)
Models trained on datasets.
79.56 (4.5)
71.14 86 (3.5)
67.01 (2.5)
65.18 (3.5)
40.26-1.13 (2)
81 00.70 (5) 86.85 +0.71 (2)
39.87=1006 2
ProtoNet
1..9911.011
ProtoNer
16.48-1.000
2.65
14.88=0.85
2,85
64.
18.25.11.01 (4.5)
69. 19.1710.96 (55)
60.81#1003 (35)
80.1310071 (6)
61 61.80 (6)
33.70 (6)
28.7910.96 (5)
MatchingNes
56. 56.40111.00
55
4.65
4.95
80.
28.
60.85 (2.5)
69.05 -0.9 (1.5)
42.60=1.17 (4.5)
85.51 (2.5)
38.55 10.94 (4.5) 43.08 + 0. (2.5)
64. 14.9880..82 ( 72.03 (3.)
63.58 03.58.10.799 (3) 69.14 (15)
38.16-1..044 (35)
57.31 17.3+-122 1 (5)
38 38.20 1.02 (3)
66.79-1.31 (1)
10.97 (4)
71.11 17 (7)
85.28 85.28-0.669 00699 (3)
35.17 3
Finetune
18.69-1.266
Finetune
59.8211.15
47.05=#1.16
66.7441.23
2.9
3.6
34.86
38.
50.13 1.00 (6.5)
$0.3810.99 (6.5)
66.35 10.9520.92222. (2.))
37.12+1.006 006 (3.5)
41.03.1.00 (6)
37.07=1.15 (7)
66.36 06.36 (4)
36. 16 1.02 (4)
83. 0. (4)
1. (6)
74.60 08 (
44 14.8841.05.00 (7)
13.47-10.61 (4)
MSCOCO 29.551009960 (5)
Traffic Signs 10
K-NN
46.81=0.899
32. 1.00
5.7
K-NN
Avg. rank 5.05
Test Source
Quick Draw
VGG Flower
Traffic Signs
MSCOCO
Avg. rank
Test Source
Quick Draw
VGG Flower
ILSVRC
Omniglot
Aireralt
Textures
ILSVRC
Omniglot
Aircraft
Textures
Birds
Fungi
Birds
Fungi
17

---

Published as a conference paper at ICLR 2020
Figure 6: Accuracy on the test datasets, when training on ILSVRC only or All datasets (same results
as shown in the main tables). The bars display 95% confidence intervals.
ILSVRC
Omniglot
Aircraft
CU Birds
Textures
90
80
Model
60
K-NN baseline
Finetune baseline
30
MatchingNet
ProtoNet
10
0
fo-MAML
Quick Draw
Fungi
VGG Flower
Traffic Sign
MSCOCO
RelationNet
fo-Prato-MAML
70
40
Train Source
ILSVRC-2012
20
10
All datasets
.8 EFFECT OF PRE-TRAINING VERSUS TRAINING FROM SCRATCH
For each meta-lcarner, selected the best model (based validation ImageNet's validation split) out of the ones that used the pre-trained initialization, and the best out of the ones that trained from scratch. We then the evaluation of each (the test split of) all datasets in order to quantify how beneficial this pre-trained initialization is. We performed this experiment twice: for the models that are trained ImageNet only and for the models that are trained on (the training splits of) all datasets.
The results this investigation were reported in the main paper in Figure 3a and Figure 3bb for ImageNec-only training and all dataset training, respectively. We show the same results in Figure 7 printed larger facilitate viewing of error bars. For easier comparison, performance of the models that were pre-trained over the ones that weren's 't, in Figures 8a and {bb These figures make easier spot that while using the pre-trained solution usually helps for datasets that are visually not too different from ImageNet, it may hurt for datasets that are significantly different from it, such as Omniglot, Quickdraw (and surprisingly Aircraft). Note that these three datasets the same three that found benefit from training All datasets instead of ImageNet-only. It appears that using the pre-trained solution biases the final solution to specialize on ImageNet-like datasets.
also plot the difference in
9 EFFECT META-LEARNING VERSUS INFERENCE-ONE
Figure 9 ssows the same plots as in Figures 3c and 3d but printed larger to facilitate viewing of error bars. Furthermore, as we have done for visualizing the observed gain of pre-training, we also present in Figures T0a and TOb the gain observed from meta-learning as opposed to training the corresponding inference-ond bascline, explained in the Experiments section of the main paper. This visulization makes clear that while meta-training usually helps on ImageNet (or doesn't 't hurt too much), it sometimes hurts when it is performed all datasets, emphasizing the need for further research into best practices meta-learning across heterogeneous sources.
.10 FINEGRAINEDNESS ANALYSIS
We investigate the hypothesis that finer-grained tasks are more challenging than coarse-grained ones by creating binary ImageNet episodes with the two classes chosen uniformly at random from the DAG's set of leaves. We then define the degree of coarse-grainedness of task as the height of the lowest common ancestor of the two chosen leaves, where the height is defined as the length of the longest path from the lowest common ancestor one of the selected leaves. Larger heights then correspond to coarser- gaanser-graiedd tasks. We present these results in Figure [T] We do not detect a significant trend when performing this analysis the test DAG. The results the training DAG,
18

---

Published as a conference paper at ICLR 2020
ILSVRC
Omniglat
Aircraft
CU Birds
Textures
Model
60-
MatchingMet
50
Proto
20
1o.MAML.
RelationNet
10
Proto.
90
Quick
Fungi
VGG
Traffic Sign
MSCOCO
80-
70-
60
50
Initialization
30
Pre-trained From scratch
20
10
(a) ImageNett
100-
Omniglot
Aircraft
Textures
Model
MatchingNet
60
ProtoiNet
fo-MANL
RelationNet
fo-Proto MAML
100
Quick Draw
Fungi
VGG Flower
Traffic Sign
MSCOCO
Initiallization
Pre-trained
20
From scratch
(b) datasets.
Figure 7: Comparing pre-training to starting from scratch. Same plots Figure 3a and Figure 3b only larger.
Moze
Mood
Prif RAMM
Quick
Purg
VDS Home
Trlls-
R500CU
Outck Druw
VCC Ronm
Trails
MDCOCO
10
(a) The gain from pre-training (ImageNet).
(b) The gain from pre-training (All datasets).
Figure 8: The performance difference of initializing the embedding weights from a pre-trained solution, before episodically training on ImageNet or all datasets, over using random initialization of those weights. The pre-trained weights that we consider are the ones that the k-NN baseline converged to when it was trained on ImageNet. Positive values indicate that this pre-training was beneficial.
19

---

Published as a conference paper at ICLR 2020
ILSVRC
Omniglat
Aircraft
cu Birds
Textures
70
Model MatchingNet
60-
50
40
30
fo-Proto-MAML
90
Fungi
VGG
Traffic Sign
MSCOCO
80
70-
00
50
30
Meta-training Inference-only
20
10
(a) ImageNet.
Omniglat
Textures
80
Model MatchingNet
70
60-
ProtoNet fo Proto MAMIL
20
10
90
Quick Draw
Fungi
VGG Flower
Traffic Sign
MSCOCO
80-
70-
50
40
30
Meta-training Inference-only
20
(b) datasets.
Figure 9: Comparing the meta-trained variant of meta-learners against their inference-only counter- part. Same plots as Figure 3c and Figure 3d only larger.
LSVIc
Artut.
Wodd
Manchnggts Proeeee: In
-10
Fungl
VGG OAEr
Traff 390
MSCOCO
visi ReAer
Traffic
MicoCo
(a) The gain from meta-training on ImageNet.
(b) The gain from meta-training on All datasets.
Figure 10: The performance difference of meta-learning over the corresponding inference-only baseline of each meta-learner. Positive values indicate that meta-learning was beneficial.
0.35
10-
0.85
M-
k- bustine
0.75
Vatchingher
Frscure
07
Moccer ghoct
Fineure tanediee
Procotet
6.6
0.5
0.60
Height Lowest Common Ancestor
Height Loweet Common Ancestor
(a) Fine- sine-grainedness Analysis (on ImageNet's test Fine-grainn meneggaiinedeess Analysis (on ImageNet's train graph)
graph graph)
Figure 11: Analysis of performance function of the degree of fine-grainedness. Larger heights correspond to grained tasks. The bands display 95% confidence intervals.
20

---

Published as a conference paper at ICLR 2020
Table 3: Improvement of fo-MAML when using larger inner learning rate a.
(a) Models trained on ILSVRC-2012 only.
Method: Accuracy (%)
confidence (%)
Test Source
fo-MAML 0.01 (old)
fo-MAML a 0.1
ILSVRC
36.09=1.01
45.51+1.11
Omniglot
38.67+1.39
55.55 + .54
Aircraft
34.50=00.90
56.24+1.11
Birds
49.10=1.18
63.61#1.066
Textures
56.50=10.80
68.04410.81
Quick Draw
27.24=1.224
43.96=#1.29
Fungi
23.50=
32.10=1.110
VGG Flower
66.42.1096
81.7440.83
Traffic Signs
33.23+1.34
50.93+1.51
MSCOCO
27.52+1.111
35.30+1.223
(b) Models trained on all datasets.
Method: Accuracy (%)
confidence (%)
Test Source
fo-MAML 0.01 (old)
fo-MAML a R 0.1
ILSVRC
32.36=11.02
37.8311.001
Omniglot
71.91=1.220
83.92:10.95
Aircraft
52.76=10.90
76.41+0.69
Birds
47.24=1.14
62.431
Textures
56.66-0074
6..8
Quick Draw
50.50=1.119
59.73+11.10
Fungi
21.02:10.99
33.54::1.11
VGG Flower
70.93-10.99
70.94
Traffic Signs 34.18=1.26
42.91=1.31
MSCOCO
24.05=1.10
29.37
though, do seem indicate that our hypothesis holds some extent. We conjecture that this may be due to the richer structure of the training DAG, but we encourage further investigation.
.11 THE IMPORTANCE OF MAML'S INNER-LOOP LEARNING RATE HYPERPARAMETER.
The camera-ready version includes updated results for MAMI and Proto-MAML following external suggestion to experiment with larger values for the inner-loop learning rate of MAML. We found that re-doing our hyperparameter search with a revised range that includes larger values significantly improved fo-MAML's performance on the same change to To-Proto-MAMe and re-ran those experiments too.
For consistency, we applied
We found that the value of this a that performs best for fo-MAML both for training on ImageNet only and training on all datasets is approximately 0.1, which is an order of magnitude larger than our previous best value. Interestingly, being 0.0054 when training on ImageNet only and 0.02 when training on all datasets. Plausibly this difference be attributed to the better initialization of Proto-MAME which requires less aggressive optimization for the adaptation to each new task. This hypothesis is also supported by the fact that fo-Proto-MANN chooses to take fewer adaptation steps than fo-MAML does. The complete set of best discovered hyperparameters is available in our public code.
--PPoto-MMMM does not choose such large value, with best
To emphasize the importance of properly tuning this hyperparameter, Table Bableadssplyys the previous best and the new best fo-MAML results sideeb showcasing the large performance gap when using the appropriate value for a.
21

---

Published as a conference paper at ICLR 2020
.12 THE CHOICE OF META- VALIDATION PROCEDURE FOR SETA-DATASET
The design choice we made, as discussed in the main paper, is to use (the meta-validation set of) ImageNet only for model selection in all of our experiments. In the absence of previous results the topic, this is a reasonable strategy since ImageNet has been known to consitute a useful proxy for performance on other datasets. However, it likely that this is not the optimal choice: there might be certain hyperparameters for given model that work best for held-out ImageNet episodes, but not for held-out episodes of other datasets. An alternative meta-validation scheme would to use the average (across datasets) validation accuracy as an indicator for early stopping and model selection. did not choose this method due to concerns about the reliability this average performance. Notably, taking simple average would over-emphasize larger datasets, or might over-emphasize datasets with natural images (as opposed to Omniglot and Quickdraw). Nevertheless, whether this strategy is beneficial is an interesting empirical question.
.13 ADDITIONAL PER-DA TER-DATASET ANALYSIS SHOTS AND WAYS
In our previous analysis of performance across different shots and ways (Figures 2b 2a and 5). the performance is averaged over all evaluation datasets. this section we further break down those plots by presenting the results separately for each dataset. Figures 12 and 13 show the analysis of performance as a function of ways and shots (respectively) for each evaluation dataset, for models that were trained on ImageNet only. For completeness, Figures 14 and 15 show the same for the models trained on (the training splits of) all datasets.
(a) ILSVRC
(b) Omniglot
(c) Aireraft
(d) Birds
(c) Textures
(f) Quickdraw
(g) Fungi
(h) VGG Flower
K-NN baseline
Finetune baseline
Matching
ProtoNet
fo MAML
RelationNet
fo Ro-Proto-MAMA
(1) Traffic Signs
(j) MSCOCO
Figure 12: The performance across different ways, with 95% confidence intervals, shown separately for each evaluation dataset. All models had been trained on ImageNet-only.
22

---

Published as a conference paper at ICLR 2020
(a) ILSVRC
(b) Omniglot
(c) Aireraft
(d) Birds
Stof
(e) Textures
Quickdraw
(g) Fungi
(h) VGG Flower
K-NN baseline
Finetune baseline
Matching Net
ProtoNet
fo-MAML
RelatonNet
fo-Prato-MAMA
(i) Traffic Signs
(i) MSCOCO
Figure 13: The performance across different shots, with 95% confidence intervals, shown separately for each evaluation dataset. All models had been trained ImageNet-only.
Saa
(a) ILSVRC
(b) Omniglot
(c) Aireraft
(d) Birds
(e) Textures
(f) Quickdraw
(g) Fungi
(h) VGG Flower
K-NN baseline
Finctune baseling
Matching
ProtoNet
fo-MAML
A3
RelationNet
fo Proto MAML
(i) Traffic Signs
(i) MSCOCO
Figure 14: The performance across different ways, with 95% confidence intervals, shown separately for each evaluation dataset. All models had been trained on (the training splits of) all datasets.
23

---

Published as a conference paper at ICLR 2020
(a) ILSVRC
(b) Omniglot
(c) Aircraft
(d) Birds
(c) Textures
(f) Quickdraw
(g) Fungi
(h) VGG Flower
K-NN baseline
Finetune baseline
Matching
ProtoNet
fo MAML
RelationNet
fo-Proto-MAML
(i Traffic Signs
MSCOCO
Figure 15: The performance across different shots, with 95% confidence intervals, shown separately for each evaluation dataset. All models had been trained on (the training splits of) all datasets.
24


Reviews:
[{"review_id": "ry4i8swFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents Meta-Dataset, a benchmark for few-shot classification that combines various image classification data sets, allows the number of classes and examples per class to vary, and considers the relationships between classes. It performs an empirical evaluation of six algorithms from the literature, k-NN, FineTune, Prototypical Networks, Matching Networks, Relation Networks, and MAML. A new approach combining Prototypical Networks and first-order MAML is shown to outperform those algorithms, but there is substantial room for improvement overall.\n\nMany papers in the literature have observed that current algorithms achieve very high performance on existing few-shot classification data sets such as Omniglot, making it difficult to compare them. This paper fills the need for a more challenging data set. This data set is also more realistic in that\n1. The training and test data may come from different data sets. In the real world, we often encounter locations, weather conditions, etc. that are unseen during training, and agents must be able to adapt quickly to these new situations.\n2. Classes are selected based on their semantic structure. This allows control of the difficulty of the task, as it is easier to adapt to a semantically similar class, and thus enables a more nuanced comparison of few-shot classification algorithms.\n\nSuggestions for clarifying the writing:\n1. Section 3 should also discuss choosing the data set to sample from.\n2. In figures 1c and 1d, it would be helpful to include standard error regions, like in the rest of figure 1.\n3. Maybe the paragraph on Proto-MAML should be moved to section 3, as it is not from previous literature. In addition, steps 2b and 2c in section 3.2 overlap in content, and so it may be clearer to combine them.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "rJxECyLMqB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors of this paper construct a new few-shot learning dataset. The whole dataset consists of several data from different sources. The authors test several representative meta-learning models (e.g., matching network, Prototype network, MAML) on this dataset and give the analysis. Furthermore, the authors combine MAML and Prototype network, which achieves the best performance on this new dataset.\n\nPros:\n+ Compared with previous datasets (e.g., miniimagenet, omniglot), the constructed meta-dataset is larger and more realistic, which contains several datasets collected from different sources\n+ Several competitive baselines are compared on this dataset under different scenarios (e.g., different number of shot) with reasonable analysis.\n\nCons:\n- I am familiar with meta-learning, however, it is my first time to review a paper whose main contribution is proposing a new benchmark. The proposed dataset may useful in further meta-learning research. However, I do not feel the construction way is quite difficult. The authors only propose several rules to construct the data set (see 3.2). \n- It would be better if the authors can explain more about Proto-MAML. My understanding of Proto-MAML is to apply the prototype on the last layer and keep the other layers.\n\n-------------------------------------------------------------------------------------------------------------------------------------------------\nAfter rebuttal:\n\nAfter reading the response, I think constructing a new benchmark is important and useful. However, considering the technical contributions of this paper, I finally decide to keep my score.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "ByeE8SYkqS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a really interesting direction for few-shot and meta-learning, the concept of a 'meta-dataset', which can be used for more realistic evaluation of algorithms. The main contributions are:\n\n1) A more realistic environment for training and testing few-shot learners. \n2) Experimental evaluation of popular models\n3) Analyses of whether different models benefit from more data,\n4) A new meta-learner\n\nI think this work is an interesting empirical paper which should be supported by solid experimental results. My concern about this paper in its current form is that the layout/structure of the paper needs to be improved, for example:\n\nConsidering putting some of the key results in the appendix section in the main text\nRemoval of repeating results from the main text by shortening the early sections", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: While the reviewers have some outstanding issues regarding the organization and clarity of the paper, the overall consensus is that the proposed evaluation methods is a useful improvement over current standards for meta-learning.
Decision: Accept (Poster)