Year: 2020

Abstract:
Few-shot classification refers to learning a classifier for new classes given only a few examples. While a plethora of models have emerged to tackle it, we find the procedure and datasets that are used to assess their progress lacking. To address this limitation, we propose META-DATASET: a new benchmark for training and evaluating models that is large-scale, consists of diverse datasets, and presents more realistic tasks. We experiment with popular baselines and meta-learners on META-DATASET, along with a competitive method that we propose. We analyze performance as a function of various characteristics of test tasks and examine the models’ ability to leverage diverse training sources for improving their generalization. We also propose a new set of baselines for quantifying the benefit of meta-learning in META-DATASET. Our extensive experimentation has uncovered important research challenges and we hope to inspire work in these directions.

heading: 1 INTRODUCTION
text: Few-shot learning refers to learning new concepts from few examples, an ability that humans naturally possess, but machines still lack. Improving on this aspect would lead to more efficient algorithms that can flexibly expand their knowledge without requiring large labeled datasets. We focus on few-shot classification: classifying unseen examples into one of N new ‘test’ classes, given only a few reference examples of each. Recent progress in this direction has been made by considering a meta-problem: though we are not interested in learning about any training class in particular, we can exploit the training classes for the purpose of learning to learn new classes from few examples, thus acquiring a learning procedure that can be directly applied to new few-shot learning problems too.
This intuition has inspired numerous models of increasing complexity (see Related Work for some examples). However, we believe that the commonly-used setup for measuring success in this direction is lacking. Specifically, two datasets have emerged as de facto benchmarks for few-shot learning: Omniglot (Lake et al., 2015), and mini-ImageNet (Vinyals et al., 2016), and we believe that both of them are approaching their limit in terms of allowing one to discriminate between the merits of different approaches. Omniglot is a dataset of 1623 handwritten characters from 50 different alphabets and contains 20 examples per class (character). Most recent methods obtain very high accuracy on Omniglot, rendering the comparisons between them mostly uninformative. mini-ImageNet is formed out of 100 ImageNet (Russakovsky et al., 2015) classes (64/16/20 for train/validation/test) and contains 600 examples per class. Albeit harder than Omniglot, it has the same property that most recent methods trained on it present similar accuracy when controlling for model capacity. We advocate that a more challenging and realistic benchmark is required for further progress in this area.
More specifically, current benchmarks: 1) Consider homogeneous learning tasks. In contrast, real-life learning experiences are heterogeneous: they vary in terms of the number of classes and examples per class, and are unbalanced. 2) Measure only within-dataset generalization. However, we are eventually after models that can generalize to entirely new distributions (e.g., datasets). 3) Ignore the relationships between classes when forming episodes. Specifically, the coarse-grained classification of dogs and chairs may present different difficulties than the fine-grained classification of dog breeds, and current benchmarks do not establish a distinction between the two.
META-DATASET aims to improve upon previous benchmarks in the above directions: it is significantly larger-scale and is comprised of multiple datasets of diverse data distributions; its task creation is informed by class structure for ImageNet and Omniglot; it introduces realistic class imbalance; and it varies the number of classes in each task and the size of the training set, thus testing the robustness of models across the spectrum from very-low-shot learning onwards.
The main contributions of this work are: 1) A more realistic, large-scale and diverse environment for training and testing few-shot learners. 2) Experimental evaluation of popular models, and a new set of baselines combining inference algorithms of meta-learners with non-episodic training. 3) Analyses of whether different models benefit from more data, heterogeneous training sources, pre-trained weights, and meta-training. 4) A novel meta-learner that performs strongly on META-DATASET.

heading: 2 FEW-SHOT CLASSIFICATION: TASK FORMULATION AND APPROACHES
text: Task Formulation The end-goal of few-shot classification is to produce a model which, given a new learning episode with N classes and a few labeled examples (kc per class, c ∈ 1, . . . , N ), is able to generalize to unseen examples for that episode. In other words, the model learns from a training (support) set S = {(x1, y1), (x2, y2), . . . , (xK , yK)} (with K = ∑ c kc) and is evaluated on a held-out test (query) setQ = {(x∗1, y∗1), (x∗2, y∗2), . . . , (x∗T , y∗T )}. Each example (x, y) is formed of an input vector x ∈ RD and a class label y ∈ {1, . . . , N}. Episodes with balanced training sets (i.e., kc = k, ∀c) are usually described as ‘N -way, k-shot’ episodes. Evaluation episodes are constructed by sampling their N classes from a larger set Ctest of classes and sampling the desired number of examples per class.
A disjoint set Ctrain of classes is available to train the model; note that this notion of training is distinct from the training that occurs within a few-shot learning episode. Few-shot learning does not prescribe a specific procedure for exploiting Ctrain, but a common approach matches the conditions in which the model is trained and evaluated (Vinyals et al., 2016). In other words, training often (but not always) proceeds in an episodic fashion. Some authors use training and testing to refer to what happens within any given episode, and meta-training and meta-testing to refer to using Ctrain to turn the model into a learner capable of fast adaptation and Ctest for evaluating its success to learn using few shots, respectively. This nomenclature highlights the meta-learning perspective alluded to earlier, but to avoid confusion we will adopt another common nomenclature and refer to the training and test sets of an episode as the support and query sets and to the process of learning from Ctrain simply as training. We use the term ‘meta-learner’ to describe a model that is trained episodically, i.e., learns to learn across multiple tasks that are sampled from the training set Ctrain.
Non-episodic Approaches to Few-shot Classification A natural non-episodic approach simply trains a classifier over all of the training classes Ctrain at once, which can be parameterized by a neural network with a linear layer on top with one output unit per class. After training, this neural network is used as an embedding function g that maps images into a meaningful representation space. The hope of using this model for few-shot learning is that this representation space is useful even for examples of classes that were not included in training. It would then remain to define an algorithm for performing few-shot classification on top of these representations of the images of a task. We consider two choices for this algorithm, yielding the ‘k-NN’ and ‘Finetune’ variants of this baseline.
Given a test episode, the ‘k-NN’ baseline classifies each query example as the class that its ‘closest’ support example belongs to. Closeness is measured by either Euclidean or cosine distance in the learned embedding space; a choice that we treat as a hyperparameter. On the other hand, the ‘Finetune’ baseline uses the support set of the given test episode to train a new ‘output layer’ on top of the embeddings g, and optionally finetune those embedding too (another hyperparameter), for the purpose of classifying between the N new classes of the associated task.
A variant of the ‘Finetune’ baseline has recently become popular: Baseline++ (Chen et al., 2019), originally inspired by Gidaris & Komodakis (2018); Qi et al. (2018). It uses a ‘cosine classifier’ as the final layer (`2-normalizing embeddings and weights before taking the dot product), both during the non-episodic training phase, and for evaluation on test episodes. We incorporate this idea in our codebase by adding a hyperparameter that optionally enables using a cosine classifier for the ‘k-NN’ (training only) and ‘Finetune’ (both phases) baselines.
Meta-Learners for Few-shot Classification In the episodic setting, models are trained end-to-end for the purpose of learning to build classifiers from a few examples. We choose to experiment with Matching Networks (Vinyals et al., 2016), Relation Networks (Sung et al., 2018), Prototypical Networks (Snell et al., 2017) and Model Agnostic Meta-Learning (MAML, Finn et al., 2017) since they cover a diverse set of approaches to few-shot learning. We also introduce a novel meta-learner which is inspired by the last two models.
In each training episode, episodic models compute for each query example x∗ ∈ Q, the distribution for its label p(y∗|x∗,S) conditioned on the support set S and allow to train this differentiablyparameterized conditional distribution end-to-end via gradient descent. The different models are distinguished by the manner in which this conditioning on the support set is realized. In all cases, the performance on the query set drives the update of the meta-learner’s weights, which include (and sometimes consist only of) the embedding weights. We briefly describe each method below.
Prototypical Networks Prototypical Networks construct a prototype for each class and then classify each query example as the class whose prototype is ‘nearest’ to it under Euclidean distance. More concretely, the probability that a query example x∗ belongs to class k is defined as:
p(y∗ = k|x∗,S) = exp(−||g(x ∗)− ck||22)∑
k′∈{1,...,N} exp(−||g(x∗)− ck′ ||22)
where ck is the ‘prototype’ for class k: the average of the embeddings of class k’s support examples.
Matching Networks Matching Networks (in their simplest form) label each query example as a (cosine) distance-weighted linear combination of the support labels:
p(y∗ = k|x∗,S) = |S|∑ i=1 α(x∗,xi)1yi=k,
where 1A is the indicator function and α(x∗,xi) is the cosine similarity between g(x∗) and g(xi), softmax-normalized over all support examples xi, where 1 ≤ i ≤ |S|.
Relation Networks Relation Networks are comprised of an embedding function g as usual, and a ‘relation module’ parameterized by some additional neural network layers. They first embed each support and query using g and create a prototype pc for each class c by averaging its support embeddings. Each prototype pc is concatenated with each embedded query and fed through the relation module which outputs a number in [0, 1] representing the predicted probability that that query belongs to class c. The query loss is then defined as the mean square error of that prediction compared to the (binary) ground truth. Both g and the relation module are trained to minimize this loss.
MAML MAML uses a linear layer parametrized by W and b on top of the embedding function g(·; θ) and classifies a query example as
p(y∗|x∗,S) = softmax(b′ +W′g(x∗; θ′)),
where the output layer parameters W′ and b′ and the embedding function parameters θ′ are obtained by performing a small number of within-episode training steps on the support set S, starting from initial parameter values (b,W, θ). The model is trained by backpropagating the query set loss through the within-episode gradient descent procedure and into (b,W, θ). This normally requires computing second-order gradients, which can be expensive to obtain (both in terms of time and memory). For this reason, an approximation is often used whereby gradients of the within-episode descent steps are ignored. This variant is referred to as first-order MAML (fo-MAML) and was used in our experiments. We did attempt to use the full-order version, but found it to be impractically expensive (e.g., it caused frequent out-of-memory problems).
Moreover, since in our setting the number of ways varies between episodes, b,W are set to zero and are not trained (i.e., b′,W′ are the result of within-episode gradient descent initialized at 0), leaving only θ to be trained. In other words, MAML focuses on learning the within-episode initialization θ of the embedding network so that it can be rapidly adapted for a new task.
Introducing Proto-MAML We introduce a novel meta-learner that combines the complementary strengths of Prototypical Networks and MAML: the former’s simple inductive bias that is evidently effective for very-few-shot learning, and the latter’s flexible adaptation mechanism.
As explained by Snell et al. (2017), Prototypical Networks can be re-interpreted as a linear classifier applied to a learned representation g(x). The use of a squared Euclidean distance means that output logits are expressed as
−||g(x∗)− ck||2 = −g(x∗)T g(x∗) + 2cTk g(x∗)− cTk ck = 2cTk g(x∗)− ||ck||2 + constant
where constant is a class-independent scalar which can be ignored, as it leaves output probabilities unchanged. The k-th unit of the equivalent linear layer therefore has weights Wk,· = 2ck and biases bk = −||ck||2, which are both differentiable with respect to θ as they are a function of g(·; θ). We refer to (fo-)Proto-MAML as the (fo-)MAML model where the task-specific linear layer of each episode is initialized from the Prototypical Network-equivalent weights and bias defined above and subsequently optimized as usual on the given support set. When computing the update for θ, we allow gradients to flow through the Prototypical Network-equivalent linear layer initialization. We show that this simple modification significantly helps the optimization of this model and outperforms vanilla fo-MAML by a large margin on META-DATASET.

heading: 3 META-DATASET: A NEW FEW-SHOT CLASSIFICATION BENCHMARK
text: META-DATASET aims to offer an environment for measuring progress in realistic few-shot classification tasks. Our approach is twofold: 1) changing the data and 2) changing the formulation of the task (i.e., how episodes are generated). The following sections describe these modifications in detail. The code is open source and publicly available1.

heading: 3.1 META-DATASET’S DATA
text: META-DATASET’s data is much larger in size than any previous benchmark, and is comprised of multiple existing datasets. This invites research into how diverse sources of data can be exploited by a meta-learner, and allows us to evaluate a more challenging generalization problem, to new datasets altogether. Specifically, META-DATASET leverages data from the following 10 datasets: ILSVRC-2012 (ImageNet, Russakovsky et al., 2015), Omniglot (Lake et al., 2015), Aircraft (Maji et al., 2013), CUB-200-2011 (Birds, Wah et al., 2011), Describable Textures (Cimpoi et al., 2014), Quick Draw (Jongejan et al., 2016), Fungi (Schroeder & Cui, 2018), VGG Flower (Nilsback & Zisserman, 2008), Traffic Signs (Houben et al., 2013) and MSCOCO (Lin et al., 2014). These datasets were chosen because they are free and easy to obtain, span a variety of visual concepts (natural and human-made) and vary in how fine-grained the class definition is. More information about each of these datasets is provided in the Appendix.
To ensure that episodes correspond to realistic classification problems, each episode generated in META-DATASET uses classes from a single dataset. Moreover, two of these datasets, Traffic Signs and MSCOCO, are fully reserved for evaluation, meaning that no classes from them participate in the training set. The remaining ones contribute some classes to each of the training, validation and test splits of classes, roughly with 70% / 15% / 15% proportions. Two of these datasets, ImageNet and Omniglot, possess a class hierarchy that we exploit in META-DATASET. For each dataset, the composition of splits is available online2.
ImageNet ImageNet is comprised of 82,115 ‘synsets’, i.e., concepts of the WordNet ontology, and it provides ‘is-a’ relationships for its synsets, thus defining a DAG over them. META-DATASET uses the 1K synsets that were chosen for the ILSVRC 2012 classification challenge and defines a new class split for it and a novel procedure for sampling classes from it for episode creation, both informed by its class hierarchy.
Specifically, we construct a sub-graph of the overall DAG whose leaves are the 1K classes of ILSVRC2012. We then ‘cut’ this sub-graph into three pieces, for the training, validation, and test splits,
1github.com/google-research/meta-dataset 2github.com/google-research/meta-dataset/tree/master/meta_dataset/dataset_conversion/splits
such that there is no overlap between the leaves of any of these pieces. For this, we selected the synsets ‘carnivore’ and ‘device’ as the roots of the validation and test sub-graphs, respectively. The leaves that are reachable from ‘carnivore’ and ‘device’ form the sets of the validation and test classes, respectively. All of the remaining leaves constitute the training classes. This method of splitting ensures that the training classes are semantically different from the test classes. We end up with 712 training, 158 validation and 130 test classes, roughly adhering to the standard 70 / 15 / 15 (%) splits.
Omniglot This dataset is one of the established benchmarks for few-shot classification as mentioned earlier. However, contrary to the common setup that flattens and ignores its two-level hierarchy of alphabets and characters, we allow it to influence the episode class selection in META-DATASET, yielding finer-grained tasks. We also use the original splits proposed in Lake et al. (2015): (all characters of) the ‘background’ and ‘evaluation’ alphabets are used for training and testing, respectively. However, we reserve the 5 smallest alphabets from the ‘background’ set for validation.

heading: 3.2 EPISODE SAMPLING
text: In this section we outline META-DATASET’s algorithm for sampling episodes, featuring hierarchicallyaware procedures for sampling classes of ImageNet and Omniglot, and an algorithm that yields realistically imbalanced episodes of variable shots and ways. The steps for sampling an episode for a given split are: Step 0) uniformly sample a dataset D, Step 1) sample a set of classes C from the classes of D assigned to the requested split, and Step 2) sample support and query examples from C.
Step 1: Sampling the episode’s class set This procedure differs depending on which dataset is chosen. For datasets without a known class organization, we sample the ‘way’ uniformly from the range [5,MAX-CLASSES], where MAX-CLASSES is either 50 or as many as there are available. Then we sample ‘way’ many classes uniformly at random from the requested class split of the given dataset. ImageNet and Omniglot use class-structure-aware procedures outlined below.
ImageNet class sampling We adopt a hierarchy-aware sampling procedure: First, we sample an internal (non-leaf) node uniformly from the DAG of the given split. The chosen set of classes is then the set of leaves spanned by that node (or a random subset of it, if more than 50). We prevent nodes that are too close to the root to be selected as the internal node, as explained in more detail in the Appendix. This procedure enables the creation of tasks of varying degrees of fine-grainedness: the larger the height of the internal node, the more coarse-grained the resulting episode.
Omniglot class sampling We sample classes from Omniglot by first sampling an alphabet uniformly at random from the chosen split of alphabets (train, validation or test). Then, the ‘way’ of the episode is sampled uniformly at random using the same restrictions as for the rest of the datasets, but taking care not to sample a larger number than the number of characters that belong to the chosen alphabet. Finally, the prescribed number of characters of that alphabet are randomly sampled. This ensures that each episode presents a within-alphabet fine-grained classification.
Step 2: Sampling the episode’s examples Having already selected a set of classes, the choice of the examples from them that will populate an episode can be broken down into three steps. We provide a high-level description here and elaborate in the Appendix with the accompanying formulas.
Step 2a: Compute the query set size The query set is class-balanced, reflecting the fact that we care equally to perform well on all classes of an episode. The number of query images per class is set to a number such that all chosen classes have enough images to contribute that number and still remain with roughly half on their images to possibly add to the support set (in a later step). This number is capped to 10 images per class.
Step 2b: Compute the support set size We allow each chosen class to contribute to the support set at most 100 of its remaining examples (i.e., excluding the ones added to the query set). We multiply this remaining number by a scalar sampled uniformly from the interval (0, 1] to enable the potential generation of ‘few-shot’ episodes even when multiple images are available, as we are also interested in studying that end of the spectrum. We do enforce, however, that each chosen class has a budget for at least one image in the support set, and we cap the total support set size to 500 examples.
Step 2c: Compute the shot of each class We now discuss how to distribute the total support set size chosen above across the participating classes. The un-normalized proportion of the support set that will be occupied by a given chosen class is a noisy version of the total number of images of that class in the dataset. This design choice is made in the hopes of obtaining realistic class ratios, under the hypothesis that the dataset class statistics are a reasonable approximation of the real-world statistics of appearances of the corresponding classes. We ensure that each class has at least one image in the support set and distribute the rest according to the above rule.
After these steps, we complete the episode creation process by choosing the prescribed number of examples of each chosen class uniformly at random to populate the support and query sets.

heading: 4 RELATED WORK
text: In this work we evaluate four meta-learners on META-DATASET that we believe capture a good diversity of well-established models. Evaluating other few-shot classifiers on META-DATASET is beyond the scope of this paper, but we discuss some additional related models below.
Similarly to MAML, some train a meta-learner for quick adaptation to new tasks (Ravi & Larochelle, 2017; Munkhdalai & Yu, 2017; Rusu et al., 2019; Yoon et al., 2018). Others relate to Prototypical Networks by learning a representation on which differentiable training can be performed on some form of classifier (Bertinetto et al., 2019; Gidaris & Komodakis, 2018; Oreshkin et al., 2018). Others relate to Matching Networks in that they perform comparisons between pairs of support and query examples, using either a graph neural network (Satorras & Estrach, 2018) or an attention mechanism (Mishra et al., 2018). Finally, some make use of memory-augmented recurrent networks (Santoro et al., 2016), some learn to perform data augmentation (Hariharan & Girshick, 2017; Wang et al., 2018) in a low-shot learning setting, and some learn to predict the parameters of a large-shot classifier from the parameters learned in a few-shot setting (Wang & Hebert, 2016; Wang et al., 2017). Of relevance to Proto-MAML is MAML++ (Antoniou et al., 2019), which consists of a collection of adjustments to MAML, such as multiple meta-trained inner loop learning rates and derivative-order annealing. Proto-MAML instead modifies the output weight initialization scheme and could be combined with those adjustments.
Finally, META-DATASET relates to other recent image classification benchmarks. The CVPR 2017 Visual Domain Decathlon Challenge trains a model on 10 different datasets, many of which are included in our benchmark, and measures its ability to generalize to held-out examples for those same datasets but does not measure generalization to new classes (or datasets). Hariharan & Girshick (2017) propose a benchmark where a model is given abundant data from certain base ImageNet classes and is tested on few-shot learning novel ImageNet classes in a way that doesn’t compromise its knowledge of the base classes. Wang et al. (2018) build upon that benchmark and propose a new evaluation protocol for it. Chen et al. (2019) investigate fine-grained few-shot classification using the CUB dataset (Wah et al., 2011, also featured in our benchmark) and crossdomain transfer between mini-ImageNet and CUB. Larger-scale few-shot classification benchmarks were also proposed using CIFAR-100 (Krizhevsky et al., 2009; Bertinetto et al., 2019; Oreshkin et al., 2018), tiered-ImageNet (Ren et al., 2018), and ImageNet-21k (Dhillon et al., 2019). Compared to these, META-DATASET contains the largest set of diverse datasets in the context of few-shot learning and is additionally accompanied by an algorithm for creating learning scenarios from that data that we advocate are more realistic than the previous ones.

heading: 5 EXPERIMENTS
text: Training procedure META-DATASET does not prescribe a procedure for learning from the training data. In these experiments, keeping with the spirit of matching training and testing conditions, we trained the meta-learners via training episodes sampled using the same algorithm as we used for META-DATASET’s evaluation episodes, described above. The choice of the dataset from which to sample the next episode was random uniform. The non-episodic baselines are trained to solve the large classification problem that results from ‘concatenating’ the training classes of all datasets.
Validation Another design choice was to perform validation on (the validation split of) ImageNet only, ignoring the validation sets of the other datasets. The rationale behind this choice is that the
performance on ImageNet has been known to be a good proxy for the performance on different datasets. We used this validation performance to select our hyperparameters, including backbone architectures, image resolutions and model-specific ones. We describe these further in the Appendix.
Pre-training We gave each meta-learner the opportunity to initialize its embedding function from the embedding weights to which the k-NN Baseline model trained on ImageNet converged to. We treated the choice of starting from scratch or starting from this initialization as a hyperparameter. For a fair comparison with the baselines, we allowed the non-episodic models to start from this initialization too. This is especially important for the baselines in the case of training on all datasets since it offers the opportunity to start from ImageNet-pretrained weights.
Main results Table 1 displays the accuracy of each model on the test set of each dataset, after they were trained on ImageNet-only or all datasets. Traffic Signs and MSCOCO are not used for training in either case, as they are reserved for evaluation. We propose to use the average (over the datasets) rank of each method as our metric for comparison, where smaller is better. A method receives rank 1 if it has the highest accuracy, rank 2 if it has the second highest, and so on. If two models share the best accuracy, they both get rank 1.5, and so on. We find that fo-Proto-MAML is the top-performer according to this metric, Prototypical Networks also perform strongly, and the Finetune Baseline notably presents a worthy opponent3. We include more detailed versions of these tables displaying confidence intervals and per-dataset ranks in the Appendix.
Effect of training on all datasets instead of ImageNet only It’s interesting to examine whether training on (the training splits of) all datasets leads to improved generalizaton compared to training on (the training split of) ImageNet only. Specifically, while we might expect that training on more data helps improve generalization, it is an empirical question whether that still holds for heterogeneous data. We can examine this by comparing the performance of each model between the top and bottom sets of results of Table 1, corresponding to the two training sources (ImageNet only and all datasets, respectively). For convenience, Figure 1 visualizes this difference in a barplot. Notably, for Omniglot, Quick Draw and Aircraft we observe a substantial increase across the board from training on all
3We improved MAML’s performance significantly in the time between the acceptance decision and publication, thanks to a suggestion to use a more aggressive inner-loop learning rate. In the Appendix, we present the older MAML results side by side with the new ones, and discuss the importance of this hyperparameter for MAML.
sources. This is reasonable for datasets whose images are significantly different from ImageNet’s: we indeed expect to gain a large benefit from training on some images from (the training classes of) these datasets. Interestingly though, on the remainder of the test sources, we don’t observe a gain from all-dataset training. This result invites research into methods for exploiting heterogeneous data for generalization to unseen classes of diverse sources. Our experiments show that learning ‘naively’ across the training datasets (e.g., by picking the next dataset to use uniformly at random) does not automatically lead to that desired benefit in most cases.
Ways and shots analysis We further study the accuracy as a function of ‘ways’ (Figure 2a) and the class precision as a function of ‘shots’ (Figure 2b). As expected, we found that the difficulty increases as the way increases, and performance degrades. More examples per class, on the other hand, indeed make it easier to correctly classify that class. Interestingly, though, not all models benefit at the same rate from more data: Prototypical Networks and fo-Proto-MAML outshine other models in very-low-shot settings but saturate faster, whereas the Finetune baseline, Matching Networks, and fo-MAML improve at a higher rate when the shot increases. We draw the same conclusions when performing this analysis on all datasets, and include those plots in the Appendix. As discussed in the Appendix, we recommend including this analysis when reporting results on Meta-Dataset, aside from the main table. The rationale is that we’re not only interested in performing well on average, but also in performing well under different specifications of test tasks.
Effect of pre-training In Figures 3a and 3b, we quantify how beneficial it is to initialize the embedding network of meta-learners using the weights of the k-NN baseline pre-trained on ImageNet, as opposed to starting their episodic training from scratch. We find this procedure to often be beneficial, both for ImageNet-only training and for training on all datasets. It seems that this ImageNet-influenced initialization drives the meta-learner towards a solution which yields increased performance on natural image test datasets, especially ILSVRC, Birds, Fungi, Flowers and MSCOCO.
Perhaps unsusprisingly, though, it underperforms on significantly different datasets such as Omniglot and Quick Draw. These findings show that, aside from the choice of the training data source(s) (e.g., ImageNet only or all datasets, as discussed above), the choice of the initialization scheme can also influence to an important degree the final solution and consequently the aptness of applying the resulting meta-learner to different data sources at test time. Finally, an interesting observation is that MAML seems to benefit the most from the pre-trained initialization, which may speak to the difficulty of optimization associated with that model.
Effect of meta-training We propose to disentangle the inference algorithm of each meta-learner from the fact that it is meta-learned, to assess the benefit of meta-learning on META-DATASET. To this end, we propose a new set of baselines: ‘Prototypical Networks Inference’, ‘Matching Networks Inference’, and ‘fo-Proto-MAML Inference’, that are trained non-episodically but evaluated episodically (for validation and testing) using the inference algorithm of the respective meta-learner. This is possible for these meta-learners as they don’t have any additional parameters aside from the embedding function that explicitly need to be learned episodically (as opposed to the relation module of Relation Networks, for example). We compare each Inference-only method to its corresponding meta-learner in Figures 3c and 3d. We find that these baselines are strong: when training on ImageNet only, we can usually observe a small benefit from meta-learning the embedding weights but this benefit often disappears when training on all datasets, in which case meta-learning sometimes actually hurts. We find this result very interesting and we believe it emphasizes the need for research on how to meta-learn across multiple diverse sources, an important challenge that META-DATASET puts forth.
Fine-grainedness analysis We use ILVRC-2012 to investigate the hypothesis that finer-grained tasks are harder than coarse-grained ones. Our findings suggest that while the test sub-graph is not rich enough to exhibit any trend, the performance on the train sub-graph does seem to agree with this hypothesis. We include the experimental setup and results for this analysis in the Appendix.

heading: 6 CONCLUSION
text: We have introduced a new large-scale, diverse, and realistic environment for few-shot classification. We believe that our exploration of various models on META-DATASET has uncovered interesting
directions for future work pertaining to meta-learning across heterogeneous data: it remains unclear what is the best strategy for creating training episodes, the most appropriate validation creation and the most appropriate initialization. Current models don’t always improve when trained on multiple sources and meta-learning is not always beneficial across datasets. Current models are also not robust to the amount of data in test episodes, each excelling in a different part of the spectrum. We believe that addressing these shortcomings consitutes an important research goal moving forward.
AUTHOR CONTRIBUTIONS
Eleni, Hugo, and Kevin came up with the benchmark idea and requirements. Eleni developed the core of the project, and worked on the experiment design and management with Tyler and Kevin, as well as experiment analysis. Carles, Ross, Kelvin, Pascal, Vincent, and Tyler helped extend the benchmark by adding datasets. Eleni, Vincent, and Utku contributed the Prototypical Networks, Matching Networks, and Relation Networks implementations, respectively. Tyler implemented baselines, MAML (with Kevin) and Proto-MAML models, and updated the backbones to support them. Writing was mostly led by Eleni, with contributions by Hugo, Vincent, and Kevin and help from Tyler and Pascal for visualizations. Pascal and Pierre-Antoine worked on code organization, efficiency, and open-sourcing, Pascal and Vincent optimized the efficiency of the data input pipeline. Pierre-Antoine supervised the code development process and reviewed most of the changes, Hugo and Kevin supervised the overall direction of the research.

heading: ACKNOWLEDGMENTS
text: We would like to thank Chelsea Finn for fruitful discussions and advice on tuning fo-MAML and ensuring the correctness of implementation, as well as Zack Nado and Dan Moldovan for the initial dataset code that was adapted, and Cristina Vasconcelos for spotting an issue in the ranking of models. Finally, we’d like to thank John Bronskill for suggesting that we experiment with a larger inner-loop learning rate for MAML which indeed significantly improved our fo-MAML results on META-DATASET.

Review summaries:
The authors of this paper construct a new few-shot learning dataset. The whole dataset consists of several data from different sources. The authors test several representative meta-learning models (e.g., matching network, Prototype network, MAML) on this dataset and give the analysis. Furthermore, the authors combine MAML and Prototype network, which achieves the best performance on this new dataset.

The paper presents Meta-Dataset, a benchmark for few-shot classification that combines various image classification data sets, allows the number of classes and examples per class to vary, and considers the relationships between classes. It performs an empirical evaluation of six algorithms from the literature, k-NN, FineTune, Prototypical Networks, Matching Networks, Relation Networks, and MAML. A new approach combining Prototypical Networks and first-order MAML is shown to outperform those algorithms, but there is substantial room for improvement overall.