Are Few-shot Learning Benchmarks Too Simple ?

We argue that the widely used Omniglot and miniImageNet benchmarks are too simple because their class semantics do not vary across episodes , which defeats their intended purpose of evaluating few-shot classification methods . The class semantics of Omniglot is invariably “ characters ” and the class semantics of miniImageNet , “ object category ” . Because the class semantics are so similar , we propose a new method called Centroid Networks which can achieve surprisingly high accuracies on Omniglot and miniImageNet without using any labels at metaevaluation time . Our results suggest that those benchmarks are not adapted for supervised few-shot classification since the supervision itself is not necessary during meta-evaluation . The Meta-Dataset , a collection of 10 datasets , was recently proposed as a harder few-shot classification benchmark . Using our method , we derive a new metric , the Class Semantics Consistency Criterion , and use it to quantify the difficulty of Meta-Dataset . Finally , under some restrictive assumptions , we show that Centroid Networks is faster and more accurate than a state-of-the-art learning-to-cluster method ( Hsu et al. , 2018 ) . 1 INTRODUCTION . Supervised few-shot classification , sometimes simply called few-shot learning , consists in learning a classifier from a small number of examples . Being able to quickly learn new classes from a small number of labeled examples is desirable from a practical perspective because it removes the need to label large datasets . Typically , supervised few-shot classification is formulated as meta-learning on episodes , where each episode corresponds to two small sets of labeled examples called support and query sets . The goal is to train a classifier on the support set and to classify the query set with maximum accuracy . The Omniglot ( Lake et al. , 2011 ) and miniImageNet ( Vinyals et al. , 2016 ; Ravi & Larochelle , 2017 ) benchmarks have been heavily used to evaluate and compare supervised few-shot classification methods in the last few years ( Vinyals et al. , 2016 ; Ravi & Larochelle , 2017 ; Snell et al. , 2017 ; Finn et al. , 2017 ; Sung et al. , 2018 ) . Despite their popularity and their important role in pioneering the few-shot learning field , we argue that the Omniglot and miniImageNet benchmarks should not be taken as gold standards for evaluating supervised few-shot classification because they rely on consistent class semantics across episodes . Specifically , Omniglot classes always correspond to alphabet characters , while miniImageNet classes always correspond to object categories as defined by the WordNet taxonomy ( Miller , 1995 ; Russakovsky et al. , 2015 ) . One consequence is that benchmarks with consistent class semantics have similar class semantics between meta-training and meta-evaluation1 . Therefore , they are too “ easy ” because they do not test the ability of supervised few-shot classification methods to adapt to new class semantics . From an applications perspective , being able to adapt to changing class semantics is a desirable feature . For instance , if the application is to organize users ’ personal photo gallery , different users might want to sort their personal photo gallery according to the different semantics , such as person identity , place or time . 1We will use the term meta-evaluation to refer to either meta-validation or meta-testing , i.e . evaluation on the meta-learning validation set or test set , respectively . From a methodological perspective , we argue that supervised few-shot classification becomes an awkward task in the ideal case where the class semantics are perfectly consistent . Indeed , if the end goal of every episode is to classify the query set according to the same class semantics , do we even need the support set to define the classes , once the semantics are learned ? Consider the characters below , extracted from the “ Mongolian ” alphabet of Omniglot . How would you group the characters below ? This task is not particularly hard , even if the reader was never shown labeled examples prior to the task , simply because the reader was already familiar with the class semantics of interest ( characters ) , and can generalize them to new classes . This simple observation suggests that when class semantics are consistent , few-shot learning algorithms might not actually need labels during metaevaluation . To show this , we introduce a new learning-to-cluster2 method called Centroid Networks which achieves surprisingly high accuracies on Omniglot and miniImageNet without using any labels at meta-evaluation time.3 The method is very similar to Prototypical Networks ( Snell et al. , 2017 ) , but the key difference is that the labels of the support set can be reliably recovered through clustering whenever the cluster semantics are consistent across tasks . A harder benchmark would involve selecting different cluster semantics across episodes . For example , consider the following set of shapes : In this case , the task remains ambiguous because clustering semantics ( e.g . shape , color , border style ) have not been specified . To classify such a set requires either supervision , such as a labeled support set , or to somehow know the class semantics beforehand . Following that spirit , the Meta-Dataset , a collection of 10 datasets , was recently proposed as a harder and more realistic few-shot classification benchmark ( Triantafillou et al. , 2019 ) . Among other things such as variable numbers of ways and shots , a key difficulty of the Meta-Dataset is that class semantics vary across episodes , since episodes are generated from a randomly selected dataset . We propose to use Centroid Networks to benchmark how hard this dataset is . In particular , we suggest looking at the gap between the performance of Prototypical Networks and Centroid Networks , which we call the class semantics consistency criterion ( CSCC ) . Our contributions : • We first show that Centroid Networks , our proposed approach to perform clustering without labels at meta-evaluation time , can beat a state-of-the-art learning-to-cluster method ( Hsu et al. , 2018 ) in the setting of a known number of equally-sized clusters , while being easier to train and orders of magnitude faster to run . • We show that it is possible to achieve surprisingly high accuracies on Omniglot and miniImageNet without using any labels at meta-evaluation time , using Centroid Networks . This is captured by our proposed metric , class semantics consistency criterion ( CSCC ) , which is the first to quantify how easy a few-shot classification benchmark is . This result highlights the need for harder benchmarks which actually test the ability of supervised few-shot classification methods to adapt to new class semantics • We report CSCC on the recently proposed Meta-Dataset , to assess whether it is indeed a harder benchmark for few-shot classification . 2When no labels are available , few-shot classification is more naturally formulated as a clustering problem , as detailed in Section 4 . 3During meta-training , labeled support set is necessary for learning the class semantics . During metaevaluation , unlabeled support set images are necessary for fair comparison , as detailed in Section 4 . 2 RELATED WORK . Supervised clustering . Supervised clustering is defined in Finley & Joachims ( 2005 ) as “ learning how to cluster future sets of items [ ... ] given sets of items and complete clusterings over these sets ” . They use structured SVM to learn a similarity-metric between pairs of items , then run a fixed clustering algorithm which optimizes the sum of similarities of pairs in the same cluster . In follow-up work ( Finley & Joachims , 2008 ) , they use K-Means as the clustering algorithm . A main difference with our work is that we learn a nonlinear embedding function , whereas they assume linear embeddings . The work of Awasthi & Zadeh ( 2010 ) is also called supervised clustering , although they solve a very different problem . They propose a clustering algorithm which repetitively presents candidate clusterings to a “ teacher ” and actively requests feedback ( supervision ) . Learning to cluster . Recent deep learning literature has preferred the term “ learning to cluster ” to “ supervised clustering ” . Although the task is still the same , the main difference is the learning of a similarity metric using deep networks . Because of this aspect , these works are often classified as falling in the “ metric learning ” literature . Hsu et al . ( 2017 ; 2019 ) propose a Constrained Clustering Network ( CCN ) for learning to cluster based on two distinct steps : learning a similarity metric to predict if two examples are in the same class , and optimizing a neural network to predict cluster assignments which tend to agree with the similarity metric . CCNs obtained the state-of-the-art results when compared against other supervised clustering algorithms , we will thus use CCN as a strong baseline . In our experiments , Centroid Networks improve over CCN on their benchmarks , while being simpler to train and computationally much cheaper . Semi-supervised & constrained clustering . Semi-supervised clustering consists of clustering data with some supervision in the form of “ this pair of points should be/not be in the same cluster ” . Some methods take the pairwise supervision as hard constraints ( Wagstaff et al. , 2001 ) , while others ( including CCN ) learn metrics which tend to satisfy those constraints ( Bilenko et al. , 2004 ) . See the related work sections in Finley & Joachims ( 2005 ) ; Hsu et al . ( 2017 ) . Supervised few-shot classification . For the unsupervised few-shot classification task , our method may be compared to the supervised few-shot classification literature ( Vinyals et al. , 2016 ; Ravi & Larochelle , 2017 ; Finn et al. , 2017 ) . In particular , we have compared with Prototypical Networks ( Snell et al. , 2017 ) , which was a source of inspiration for Centroid Networks . Our work is also related to follow-up work on Semi-Supervised Prototypical Networks ( Ren et al. , 2018 ) , in which the support set contains both labeled and unlabeled examples . In this work , we go beyond by requiring no labels to infer centroids at evaluation time . Sinkhorn K-Means . The idea of formulating clustering as minimizing a Wasserstein distance between empirical distributions has been proposed several times in the past ( Mi et al. , 2018a ) . Canas & Rosasco ( 2012 ) explicit some theoretical links between K-Means and the Wasserstein-2 distance . The most similar work to Sinkhorn K-Means is Regularized Wasserstein-Means ( Mi et al. , 2018b ) , but they use another method for solving optimal transport . Specifically using Sinkhorn distances4 for clustering has even been suggested in Genevay et al . ( 2018 ) . However , as we could not find an explicit description of the Sinkhorn K-Means anywhere in the literature , we coin the name and explicitly state the algorithm in Section 5.1 . To our knowledge , we are the first to use Sinkhorn K-Means in the context of learning to cluster and to scale it up to more complex datasets like miniImageNet . Note that our work should not be confused with Wasserstein K-Means and similar variants , which consist in replacing the squared L2 base-distance in K-Means with a Wasserstein distance . Meta-Learning and Unsupervised Learning . Finally , some recent work has explored combinations of unsupervised learning and meta-learning , to address various other tasks . Metz et al . ( 2018 ) propose a method to meta-train an unsupervised representation learning model that produces useful features for some given task . That is , at evaluation time , their method produces features without requiring labels , much like Centroid Networks produce centroids without requiring labels . The difference with their method thus lies in the addressed task : we focus on clustering , while they consider the task of representation/feature learning . Hsu et al . ( 2018 ) ; Khodadadeh et al . ( 2018 ) also considers the opposite : meta-learning that requires no labels for meta-training but that delivers methods that require labels to be run at evaluation time . Specifically , they propose unsupervised approaches 4A regularized version of the Wasserstein distance . to generate episodes for supervised few-shot classification , while we use supervised data to learn an unsupervised clustering algorithm .

Summary: The paper is concerned with few-shot classification, both its benchmarks and method used to tackle it. The scope of the few-shot classification problem can be set relatively widely, depending on what data is available at what stage. In general few-shot classification is an important ability of intelligent systems and arguably an area in which biological systems outperform current AI systems the most.