Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples

1 INTRODUCTION . Few-shot learning refers to learning new concepts from few examples , an ability that humans naturally possess , but machines still lack . Improving on this aspect would lead to more efficient algorithms that can flexibly expand their knowledge without requiring large labeled datasets . We focus on few-shot classification : classifying unseen examples into one of N new ‘ test ’ classes , given only a few reference examples of each . Recent progress in this direction has been made by considering a meta-problem : though we are not interested in learning about any training class in particular , we can exploit the training classes for the purpose of learning to learn new classes from few examples , thus acquiring a learning procedure that can be directly applied to new few-shot learning problems too . This intuition has inspired numerous models of increasing complexity ( see Related Work for some examples ) . However , we believe that the commonly-used setup for measuring success in this direction is lacking . Specifically , two datasets have emerged as de facto benchmarks for few-shot learning : Omniglot ( Lake et al. , 2015 ) , and mini-ImageNet ( Vinyals et al. , 2016 ) , and we believe that both of them are approaching their limit in terms of allowing one to discriminate between the merits of different approaches . Omniglot is a dataset of 1623 handwritten characters from 50 different alphabets and contains 20 examples per class ( character ) . Most recent methods obtain very high accuracy on Omniglot , rendering the comparisons between them mostly uninformative . mini-ImageNet is formed out of 100 ImageNet ( Russakovsky et al. , 2015 ) classes ( 64/16/20 for train/validation/test ) and contains 600 examples per class . Albeit harder than Omniglot , it has the same property that most recent methods trained on it present similar accuracy when controlling for model capacity . We advocate that a more challenging and realistic benchmark is required for further progress in this area . More specifically , current benchmarks : 1 ) Consider homogeneous learning tasks . In contrast , real-life learning experiences are heterogeneous : they vary in terms of the number of classes and examples per class , and are unbalanced . 2 ) Measure only within-dataset generalization . However , we are eventually after models that can generalize to entirely new distributions ( e.g. , datasets ) . 3 ) Ignore the relationships between classes when forming episodes . Specifically , the coarse-grained classification of dogs and chairs may present different difficulties than the fine-grained classification of dog breeds , and current benchmarks do not establish a distinction between the two . META-DATASET aims to improve upon previous benchmarks in the above directions : it is significantly larger-scale and is comprised of multiple datasets of diverse data distributions ; its task creation is informed by class structure for ImageNet and Omniglot ; it introduces realistic class imbalance ; and it varies the number of classes in each task and the size of the training set , thus testing the robustness of models across the spectrum from very-low-shot learning onwards . The main contributions of this work are : 1 ) A more realistic , large-scale and diverse environment for training and testing few-shot learners . 2 ) Experimental evaluation of popular models , and a new set of baselines combining inference algorithms of meta-learners with non-episodic training . 3 ) Analyses of whether different models benefit from more data , heterogeneous training sources , pre-trained weights , and meta-training . 4 ) A novel meta-learner that performs strongly on META-DATASET . 2 FEW-SHOT CLASSIFICATION : TASK FORMULATION AND APPROACHES . Task Formulation The end-goal of few-shot classification is to produce a model which , given a new learning episode with N classes and a few labeled examples ( kc per class , c ∈ 1 , . . . , N ) , is able to generalize to unseen examples for that episode . In other words , the model learns from a training ( support ) set S = { ( x1 , y1 ) , ( x2 , y2 ) , . . . , ( xK , yK ) } ( with K = ∑ c kc ) and is evaluated on a held-out test ( query ) setQ = { ( x∗1 , y∗1 ) , ( x∗2 , y∗2 ) , . . . , ( x∗T , y∗T ) } . Each example ( x , y ) is formed of an input vector x ∈ RD and a class label y ∈ { 1 , . . . , N } . Episodes with balanced training sets ( i.e. , kc = k , ∀c ) are usually described as ‘ N -way , k-shot ’ episodes . Evaluation episodes are constructed by sampling their N classes from a larger set Ctest of classes and sampling the desired number of examples per class . A disjoint set Ctrain of classes is available to train the model ; note that this notion of training is distinct from the training that occurs within a few-shot learning episode . Few-shot learning does not prescribe a specific procedure for exploiting Ctrain , but a common approach matches the conditions in which the model is trained and evaluated ( Vinyals et al. , 2016 ) . In other words , training often ( but not always ) proceeds in an episodic fashion . Some authors use training and testing to refer to what happens within any given episode , and meta-training and meta-testing to refer to using Ctrain to turn the model into a learner capable of fast adaptation and Ctest for evaluating its success to learn using few shots , respectively . This nomenclature highlights the meta-learning perspective alluded to earlier , but to avoid confusion we will adopt another common nomenclature and refer to the training and test sets of an episode as the support and query sets and to the process of learning from Ctrain simply as training . We use the term ‘ meta-learner ’ to describe a model that is trained episodically , i.e. , learns to learn across multiple tasks that are sampled from the training set Ctrain . Non-episodic Approaches to Few-shot Classification A natural non-episodic approach simply trains a classifier over all of the training classes Ctrain at once , which can be parameterized by a neural network with a linear layer on top with one output unit per class . After training , this neural network is used as an embedding function g that maps images into a meaningful representation space . The hope of using this model for few-shot learning is that this representation space is useful even for examples of classes that were not included in training . It would then remain to define an algorithm for performing few-shot classification on top of these representations of the images of a task . We consider two choices for this algorithm , yielding the ‘ k-NN ’ and ‘ Finetune ’ variants of this baseline . Given a test episode , the ‘ k-NN ’ baseline classifies each query example as the class that its ‘ closest ’ support example belongs to . Closeness is measured by either Euclidean or cosine distance in the learned embedding space ; a choice that we treat as a hyperparameter . On the other hand , the ‘ Finetune ’ baseline uses the support set of the given test episode to train a new ‘ output layer ’ on top of the embeddings g , and optionally finetune those embedding too ( another hyperparameter ) , for the purpose of classifying between the N new classes of the associated task . A variant of the ‘ Finetune ’ baseline has recently become popular : Baseline++ ( Chen et al. , 2019 ) , originally inspired by Gidaris & Komodakis ( 2018 ) ; Qi et al . ( 2018 ) . It uses a ‘ cosine classifier ’ as the final layer ( ` 2-normalizing embeddings and weights before taking the dot product ) , both during the non-episodic training phase , and for evaluation on test episodes . We incorporate this idea in our codebase by adding a hyperparameter that optionally enables using a cosine classifier for the ‘ k-NN ’ ( training only ) and ‘ Finetune ’ ( both phases ) baselines . Meta-Learners for Few-shot Classification In the episodic setting , models are trained end-to-end for the purpose of learning to build classifiers from a few examples . We choose to experiment with Matching Networks ( Vinyals et al. , 2016 ) , Relation Networks ( Sung et al. , 2018 ) , Prototypical Networks ( Snell et al. , 2017 ) and Model Agnostic Meta-Learning ( MAML , Finn et al. , 2017 ) since they cover a diverse set of approaches to few-shot learning . We also introduce a novel meta-learner which is inspired by the last two models . In each training episode , episodic models compute for each query example x∗ ∈ Q , the distribution for its label p ( y∗|x∗ , S ) conditioned on the support set S and allow to train this differentiablyparameterized conditional distribution end-to-end via gradient descent . The different models are distinguished by the manner in which this conditioning on the support set is realized . In all cases , the performance on the query set drives the update of the meta-learner ’ s weights , which include ( and sometimes consist only of ) the embedding weights . We briefly describe each method below . Prototypical Networks Prototypical Networks construct a prototype for each class and then classify each query example as the class whose prototype is ‘ nearest ’ to it under Euclidean distance . More concretely , the probability that a query example x∗ belongs to class k is defined as : p ( y∗ = k|x∗ , S ) = exp ( −||g ( x ∗ ) − ck||22 ) ∑ k′∈ { 1 , ... , N } exp ( −||g ( x∗ ) − ck′ ||22 ) where ck is the ‘ prototype ’ for class k : the average of the embeddings of class k ’ s support examples . Matching Networks Matching Networks ( in their simplest form ) label each query example as a ( cosine ) distance-weighted linear combination of the support labels : p ( y∗ = k|x∗ , S ) = |S|∑ i=1 α ( x∗ , xi ) 1yi=k , where 1A is the indicator function and α ( x∗ , xi ) is the cosine similarity between g ( x∗ ) and g ( xi ) , softmax-normalized over all support examples xi , where 1 ≤ i ≤ |S| . Relation Networks Relation Networks are comprised of an embedding function g as usual , and a ‘ relation module ’ parameterized by some additional neural network layers . They first embed each support and query using g and create a prototype pc for each class c by averaging its support embeddings . Each prototype pc is concatenated with each embedded query and fed through the relation module which outputs a number in [ 0 , 1 ] representing the predicted probability that that query belongs to class c. The query loss is then defined as the mean square error of that prediction compared to the ( binary ) ground truth . Both g and the relation module are trained to minimize this loss . MAML MAML uses a linear layer parametrized by W and b on top of the embedding function g ( · ; θ ) and classifies a query example as p ( y∗|x∗ , S ) = softmax ( b′ +W′g ( x∗ ; θ′ ) ) , where the output layer parameters W′ and b′ and the embedding function parameters θ′ are obtained by performing a small number of within-episode training steps on the support set S , starting from initial parameter values ( b , W , θ ) . The model is trained by backpropagating the query set loss through the within-episode gradient descent procedure and into ( b , W , θ ) . This normally requires computing second-order gradients , which can be expensive to obtain ( both in terms of time and memory ) . For this reason , an approximation is often used whereby gradients of the within-episode descent steps are ignored . This variant is referred to as first-order MAML ( fo-MAML ) and was used in our experiments . We did attempt to use the full-order version , but found it to be impractically expensive ( e.g. , it caused frequent out-of-memory problems ) . Moreover , since in our setting the number of ways varies between episodes , b , W are set to zero and are not trained ( i.e. , b′ , W′ are the result of within-episode gradient descent initialized at 0 ) , leaving only θ to be trained . In other words , MAML focuses on learning the within-episode initialization θ of the embedding network so that it can be rapidly adapted for a new task . Introducing Proto-MAML We introduce a novel meta-learner that combines the complementary strengths of Prototypical Networks and MAML : the former ’ s simple inductive bias that is evidently effective for very-few-shot learning , and the latter ’ s flexible adaptation mechanism . As explained by Snell et al . ( 2017 ) , Prototypical Networks can be re-interpreted as a linear classifier applied to a learned representation g ( x ) . The use of a squared Euclidean distance means that output logits are expressed as −||g ( x∗ ) − ck||2 = −g ( x∗ ) T g ( x∗ ) + 2cTk g ( x∗ ) − cTk ck = 2cTk g ( x∗ ) − ||ck||2 + constant where constant is a class-independent scalar which can be ignored , as it leaves output probabilities unchanged . The k-th unit of the equivalent linear layer therefore has weights Wk , · = 2ck and biases bk = −||ck||2 , which are both differentiable with respect to θ as they are a function of g ( · ; θ ) . We refer to ( fo- ) Proto-MAML as the ( fo- ) MAML model where the task-specific linear layer of each episode is initialized from the Prototypical Network-equivalent weights and bias defined above and subsequently optimized as usual on the given support set . When computing the update for θ , we allow gradients to flow through the Prototypical Network-equivalent linear layer initialization . We show that this simple modification significantly helps the optimization of this model and outperforms vanilla fo-MAML by a large margin on META-DATASET .

Summary: The paper presents Meta-Dataset, a benchmark for few-shot classification that combines various image classification data sets, allows the number of classes and examples per class to vary, and considers the relationships between classes. It performs an empirical evaluation of six algorithms from the literature, k-NN, FineTune, Prototypical Networks, Matching Networks, Relation Networks, and MAML. A new approach combining Prototypical Networks and first-order MAML is shown to outperform those algorithms, but there is substantial room for improvement overall.