Are Few-shot Learning Benchmarks Too Simple ?
Authors: Gabriel Huang
Hugo Larochelle
Simon Lacoste-Julien
Venue:  (2020)
Keywords: few-shot
classification
meta-learning
benchmark
omniglot
miniimagenet
meta-dataset
learning to cluster
learning
cluster
unsupervised

Abstract: We argue that the widely used Omniglot and miniImageNet benchmarks are too simple because their class semantics do not vary across episodes, which defeats their intended purpose of evaluating few-shot classification methods. The class semantics of Omniglot is invariably “characters” and the class semantics of miniImageNet, “object category”. Because the class semantics are so similar, we propose a new method called Centroid Networks which can achieve surprisingly high accuracies on Omniglot and miniImageNet without using any labels at metaevaluation time. Our results suggest that those benchmarks are not adapted for supervised few-shot classification since the supervision itself is not necessary during meta-evaluation. The Meta-Dataset, a collection of 10 datasets, was recently proposed as a harder few-shot classification benchmark. Using our method, we derive a new metric, the Class Semantics Consistency Criterion, and use it to quantify the difficulty of Meta-Dataset. Finally, under some restrictive assumptions, we show that Centroid Networks is faster and more accurate than a state-of-the-art learning-to-cluster method (Hsu et al., 2018). 

Under review as a conference paper at ICLR 2020
ARE FEW-SHOT LEARNING BENCHMARKS TOO SIMPLE ?
A Anony monns authors Paper under double-blind review
ABSTRACT
We argue that the widely used Omniglot and minilmageNet benchmarks are too simple because their class semantics do not vary across episodes, which de- feats their intended purpose of evaluating few-shot classification methods. The class semantics of Omniglot is invariably "characters" and the class semantics of minilmageNet, "object category". Because the class semantics are so similar, we propose a new method called Centroid Networks which can achieve surprisingly high accuracies on Omniglot and minilmageNet without using any labels at meta- evaluation time. Our results suggest that those benchmarks are not adapted for su- pervised few-shot classification since the supervision itself is not necessary during meta-evaluation. The Meta-Datases a collection of 10 datasets, was recently pro- posed as harder few-shot classification benchmark. Using our method, we derive a new metric, the Class Semantics Consistency Criterion, and use to quantify the difficulty of Meta-Dataset. Finally, under some restrictive assumptions, we show that Centroid Networks is faster and more accurate than a state-of-the-art learning- momning-to-cluseee to-cluster method (Hsu al., 2018).
INTRODUCTION
Supervised few-shot classification, sometimes simply called few-shot learning, consists in learning a classifier from a small number of examples. Being able to quickly learn new classes from a small number of labeled examples desirable from a practical perspective because it removes the need to label large datasets. Typically, supervised few-shot classification is formulated as meta-learning on episodes, where each episode corresponds two small sets of labeled examples called support and query sets. The goal is to train classifier on the support set and to classify the query set with maximum accuracy.
The Omniglot (Lake et al., 2011) and minilmageNNet (Vinyals et al., 2016; Ravi & Larochelle, 2017) benchmarks have been heavily used to evaluate and compare supervised few-shot classification methods in the last few years (Vinyals et al., 2016; Ravi & Larochelle, 2017; Snell et al., 2017; Finn et al., 2017; Sung et al., 2018). Despite their popularity and their important role pioneer- ing the few-shot learning field, we argue that the Omniglot and minilmageNet benchmarks should not be taken as gold standards for evaluating supervised few-shot classification because they rely on consistent class semantics across episodes. Specifically, Omniglot classes always correspond to alphabet characters, while minilmageNet classes always correspond to object categories as de- fined by the WordNet taxonomy (Miller, 1995; Russakovsky et al., 2015). One consequence is that benchmarks with consistent class semantics have similar class semantics between meta-training and meta-evaluation!! Therefore, they are too "easy" because they do not test the ability of supervised few-shot classification methods to adapt to new class semantics.
From an applications perspective, being able adapt to changing class semantics is a desirable feature. For instance, if the application is to organize users' personal photo gallery, different users might want to sort their personal photo gallery according to the different semantics, such person identity, place or time.
We will use the term meta-evaluation to refer to either meta-validation meta-testing, i.c. evaluation on the meta-Icarning validation or test set, respectively.

---

Under review as a conference paper at ICLR 2020
From methodological perspective, argue that supervised few-shot classification becomes awkward task in the ideal case where the class semantics are perfectly consistent. Indeed, if the end goal of every episode is to classify the query set according to the same class semantics, do need the support set to define the classes, once the semantics are learned ? Consider the characters below, extracted from the "Mongolian" alphabet of Omniglot. How would you group the characters below?
5 C I
P T
This task is not particularly hard, even if the reader was never shown labeled examples prior to the task, simply because the reader already familiar with the class semanties of interest (charac- ters), and can generalize them to new classes. This simple observation suggests that when class semantics are consistent, few-shot learning algorithms might actually need labels during meta- evaluation. To show this, we introduce new learning-to-clusse which achieves surprisingly high accuracies on Omniglot and minilmageNet without using any la- bels at meta-evaluation 2017), but the key difference is that the labels of the support set can be reliably recovered through clustering whenever the cluster semantics are consistent across tasks.
method called Centroid Networks
time.? The method is very similar to Prototypical Networks (Snell et al.,
harder benchmark would involve selecting different cluster semantics across episodes. For exam- ple, consider the following set of shapes:
In this case, the task remains ambiguous because clustering semantics (e.g. shape, color, border style) have not been specified. To classify such set requires either supervision, such labeled support set, or to somehow know the class semantics beforehand.
Following that spirit, the Meta-Dataset, collection of 10 datasets, recently proposed harder and more realistic few-shot classification benchmark (Triantafillou et al., 2019). Among other things such variable numbers of ways and shots, key difficulty of the Meta-Dataset that class semantics vary across episodes, since episodes are generated from a randomly selected dataset. We propose use Centroid Networks benchmark how hard this dataset is. In particular, we suggest looking at the gap between the performance of Prototypical Networks and Centroid Networks, which we call the class semantics consistency criterion (CSCC).
Our contributions:
We first show that Centroid Networks, our proposed approach to perform clustering without labels at meta-evaluation time, can beat a state-of-in fi-the--rt learning-to-clusto to-cluster method (Hsu et al., 2018) the setting of a known number of equally-sized clusters, while being easier to train and orders of magnitude faster to run.
We show that it possible achieve surprisingly high accuracies on Omniglot and minilmageNet without using any labels at meta- evaluuavalttion time, using Centroid Networks. This is captured by our proposed metric, class semanties consistency criterion (CSCC), which is the first to quantify how easy few-shot classification benchmark is. This re- sult highlights the need for harder benchmarks which actually test the ability of supervised few-shot classification methods to adapt to new class semantics
We report CSCC on the recently proposed Meta-Dataset, to assess whether it is indeed harder benchmark for few-shot classification.
WWhen labels available, few-shot classification is naturally formulated as clustering problem, as detailed Section 4.
During meta-training, labeled support set is necessary for learning the class semantics. During meta- evaluation, unlabeled support set images are necessary for fair comparison, detailed in Section
2

---

Under review as a conference paper at ICLR 2020
2
RELATED WORK
Supervised clustering. Supervised clustering is defined in Finley & Joachims (2005) as "learning how to cluster future sets of items [.-]] given sets of items and complete clusterings over these sets". They use structured SVM to learn a similarity-metrre between pairs of items, then tering algorithm which optimizes the sum of similarities of pairs in the same cluster. In follow-up work (Finley & Joachims, 2008), they use K-Means as the clustering algorithm. A main difference with our work is that we learn nonlinear embedding function, whereas they assume linear embed- dings. The work of Awasthi & Zadeh (2010) is also called supervised clustering. although they solve very different problem. They propose clustering algorithm which repetitively presents candidate clusterings to a "teacher" and actively requests feedback (supervision).
fixed clus-
Learning to cluster. Recent deep learning literature has preferred the term "learning to cluster" to "supervised clustering". Although the task still the same, the main difference is the learning of similarity metric using deep networks. Because of this aspect, these works are often classified as falling in the "metric learning" literature. Hsu et al. (2017: 2019) propose a Constrained Clustering Network (CCN) for learning to cluster based on two distinct steps: learning similarity metric to predict two examples are in the same class, and optimizing a neural network to predict cluster assignments which tend agree with the similarity metric. CCNs obtained the state-of-the-art results when compared against other supervised clustering algorithms, we will thus use CCN as a strong baseline. In our experiments, Centroid Networks improve over CCN on their benchmarks, while being simpler to train and computationally much cheaper.
Semi-supervised & constrained clustering. Semi-supervised clustering consists of clustering data with some supervision in the form of "this pair of points should be/not be in the same cluster". Some methods take the pairwise supervision as hard constraints (Wagstaff et al., 2001), while others (including CCN) learn metrics which tend to satisfy those constraints (Bilenko et al., 2004). See the related work sections in Finley & Joachims (2005); Hsu al. (2017).
Supervised few-shot classification. For the unsupervised few-shot classification task, our method may be compared to the supervised few-shot classification literature (Vinyals et al., 2016; Ravi & Larochelle, 2017; Finn et al., 2017). In particular, we have compared with Prototypical Net- works (Snell et al., 2017), which was a source of inspiration for Centroid Networks. Our work is also related to follow-up work Semi- Supervised Prototypical Networks (Ren et al., 2018), in which the support set contains both labeled and unlabeled examples. In this work, we go beyond by requiring no labels to infer centroids evaluation time.
Sinkhorn K-Means. The idea of formulating clustering as minimizing a Wasserstein distance be- tween empirical distributions has been proposed several times in the past (Mi et al., 2018a). Canas & Rosasco (2012) explicit some theoretical links between K-Means and the Wasserstein-2 distance. The most similar work to Sinkhorn K-Means is Regularized Wasserstein-Means (Mi al., 2018b), but they use another method for solving optimal transport. Specifically using Sinkhorn distances 4 for clustering has even been suggested in Genevay et al. (2018). However, as we could not find an ex- plicit description of the Sinkhorn K-Means anywhere in the literature, we coin the name and explic- itly state the algorithm in Section 5.1. To our knowledge, we are the first to use Sinkhorn K-Means in the context of learning to cluster and to scale it up to more complex datasets like minilmageNet, Note that our work should not be confused with Wasserstein K-Means and similar variants, which consist in replacing the squared L2 base-distance in K-Means with a Wasserstein distance.
Meta-Learning and Unsupervised Learning. Finally, some recent work has explored combina- tions of unsupervised learning and lecanne, address various other tasks. Metz et al. (2018) propose a method to meta- train an unsupervised representation learning model that produces useful features for some given task. That is, at evaluation time, their method produces features without requiring labels, much like Centroid Networks produce centroids without requiring labels. The dif- ference with their method thus lies in the addressed task: we focus on clustering, while they consider the task of representation/featuu learning. Hsu et al. (2018); Khodadadeh et al. (2018) also consid- ers the opposite: meta-learning that requires no labels for meta-training but that delivers methods that require labels to be run at evaluation time. Specifically, they propose unsupervised approaches
*A regularized version of the Wasserstein distance.
3

---

Under review as a conference paper at ICLR 2020
Find Optimal Permutation
Cluster Set
predicted cluster A
true cluster
Compute Accurscy
WRONG J
clustering accuracy 66%
true cluster
predicted cluster B
optimal permutation
OK
WRONG
OK
Ground Truth Clusters cluster semantic: shape
Predicted Clusters
Figure 1: A Few-Shot Clustering Episode.
generate episodes for supervised few-shot classification, while use supervised data to learn an unsupervised clustering algorithm.
TASKS AND EVALUATION METRICS
The main point of this paper is discuss the class semantics consistency of few-shot classification benchmarks. Recall the visual examples from the introduction, where we asked the reader to cluster similar images together. The more consistent the class semantics are across episodes, the easier it should be to cluster them. Therefore, for the purpose of evaluating semantics consistency, we propose to consider additional categorization tasks for existing few-shot classification benchmarks.
3.1 DEFINING OTHER CATEGORIZATION TASKS FROM FEW-SHOT BENCHMARKS
The most common categorization task supervised few-shot classification, where episodes come with small training (support) set S (Xs, Ys) and small validation (query) set Q (Xq. YQ), where Xs, Xq denote images or data, and Ys, Yę the associated labels. The task is to predict labels for validation images Xę and the algorithm has access both to the support set images Xs and labels Ys. Finally, the predicted labels are compared against Yq, and the accuracy is returned. From now on call this metric the supervised accuracy in order to distinguish it from the clustering and unsupervised accuracy introduced below.
Few-Shot Clustering Task The task to cluster the query² images XQ. without access to the support set Xs, Ys. For evaluation, the predicted clusters matched with the ground-truth clus- ters (which can be obtained from Y²) by searching for the one-to-one ground-truth -rrun cluster/predicted cluster mapping (i.e. permutation) which results in the highest accuracy. Finding the optimal permu- tation can be done efficiently using the Hungarian algorithm as described Hsu et al. (2017). The resulting accuracy is called the clustering accuracy. This is common metric used in the literature on learning to cluster. See Figure for an illustration.
Few-shot clustering is the simplest clustering task defined here, and can be seen episodic ver- sion of the learning to cluster task. However, clustering accuracy cannot be meaningfully compared with supervised accuracy. On one hand, few-shot clustering harder than supervised few-shot clas- sification because the support set cannot be used. On the other hand, it may be easier because the query set clustered jointly (vs. independent predictions for supervised few-shot classification). In particular, the 1-shot clustering is trivial because each point already belongs to its own cluster, whereas supervised I-shot classification is not. Therefore, we propose the unsupervised few-shot classification task which is by construction strictly harder than supervised few-shot classification.
Unsupervised Few-Shot Classification Task The task is to cluster the support set images Xs then to associate each query set image æQ with one of the predicted clusters. For evaluation, the optimal permutation between predicted clusters and ground- truth clusters (which can be obtained
$ For this description, we cluster the query set to maintain the parallel with supervised few-shot classification evaluation. However. in practice we cluster the support set instead of the query set so that few-shot clustering and unsupervised few-shot classification can share the support set cluster matching step and be more directly comparable.

---

Under review as a conference paper at ICLR 2020
Find Optimal
Cluster Support Set
Compune Accuracy
true cluster
predicted cluster
3. Claciity Query Set
A->
OK
unsupervised
B-> 1
WRONG
accuracy
cluster
predicted cluster
B-> 1
OK
75%
optimal permutation Support Set
B-
OK
Support Set
Query Set Predicted Labels (Permuted)
Query Set Ground Truth Labels
Ground Truth Clusters
Predicted Clusters
cluster semantic: shope
Figure 2: An Unsupervised Few-Shot Classification Episode.
from Yz) is found in order to maximize the corresponding support set accuracy. Then the unsuper- vised accuracy is computed after relabeling the query predictions and comparing them with Yo- Unsupervised accuracy can be compared with supervised accuracy because unsupervised strietly harder than supervised few-shot classification (support set classes Ys are not available and need to be inferred). See Figure 2 for an illustration. We will use this metric to define our novel measure for the difficulty of few-shot learning benchmarks.
4 BACKGROUND
4.1 PROTOTYPICAL NETWORKS
Prototypical Networks Protonets (Snell et al., 2017) is one of the simplest and most accurate pervised few-shot classification methods. The only learnable component of Protonets is the embed- ding function hg X Z which maps images task T (K, M, N, Staheled, Q) to solve, Protonets compute the average embedding (the prototype) of each class jaj
embedding (feature) space. Given supervised
Σ, hox)) [yf j}} the support set. Each point from the query
set is then classified according the softmax of its squared distance po j j|=;) softmax(-
(1ha(x?) - 1²) Protonets are trained end-to-end by minimizing the log-loss on the query set.
CENTROID NETWORKS
In this section, we deseribe method and explain how it can be applied few-shot clustering and unsupervised few-shot classification. Centroid Networks (or CentroidNets) consist of two modules: trainable embedding module and fixed clustering module. The fact that the only trainable component of Centroid Networks is the embedding function makes implementation and training very simple.
The embedding module is the same as in Prototypical Networks and consists in a neural network ha X' which maps data (images) r to features 2 ha(x) in the embedding space. The clustering module takes as input the embedded data (=))sesN and outputs set of centroids (c))ışisı (representatives of each cluster) as well as the (soft) assignment Pr,j of each point 2, to each centroid. We use the Sinkhorn K-Means algorithm our clustering module (Section 5.1).
5.1 SINKHORN K-MEANS
We propose Sinkhorn K-Means as the clustering module of Centroid Networks. It takes input set of points x, E Rd (typically learned embeddings) and outputs a set of K centroids c E Rd. sim- ilarly to K-Means, which can be used cluster points. Sinkhorn K-Means is based the Sinkhorn distance (or regularized Wasserstein). described more in depth in Appendix A.1. The differences be- tween Sinkhorn and Regular K-Means, and their formulations constrained optimization problems are discussed Appendix B. In particular, we expect Sinkhorn K-Means to improve performance on the considered tasks (Section B.2).
5

---

Under review as a conference paper at ICLR 2020
Step 1: Finding Centroids. We propose Expectation Expectation-Maximizaaa Maximization-sty////// procedure to find the centroids which minimize the Sinkhorn distance (Section A.I) between the empirical distributions respectively defined by the data p(x) * Σ> 6(x-x;; and the centroids g(æ) * Σラ= 6(x- ej). We alternate descent on assignments and centroids. Minimization in the assignments is call to the Sinkhorn algorithm (Algorithm 1), instead of the usual greedy argmin for K-Means. Minimization in the centroids amounts to setting them equal to the weighted average of the points assigned to them. For simplicity, Algorithm describes the procedure in the case where clusters are balanced (e.g. Omniglot and minilmageNet). Typically, we initialize centroids around zero and add a tiny bit of Gaussian noise to break symmetries. When clusters are not balanced but the cluster weights known (e.g. Meta-Dataset), the weights be passed to the Sinkhorn distance (see Algorithm 1). All details can be found in the code.
Algorithm Sinkhorn(1, x c, 7) for Wasserstein-2 distance between empirical distributions.
Input: data (#i) E Rand centroids (c))ssss* E RRd regularization constant ? > O.
Output: optimal transport plan (Pt.g) E R
</1l//)) // Compute exponentiated pairwise squared 1.2-distances Kij t exp(-I|a:; ||2 -
fy // Initialize dual variables (uz) E R
R₂ + 1/n, Cj 1/k // Set data (Ra) R" and centroid (Cz) E R weights to uniform (or
desired distribution) while not converged do
U - R2 Ke5U)); by C 1= Keyt),,
15 S Enforce row (data) marginals
1 < k Enforce column (centroid) marginals
end while Pij + oy isis n, sisk
// Return optimal transport plan.
Algorithm 2 Sinkhorn K-Means c,7)
Input: data (x:) initial centroids (cy)ısıık, regularization constant y >
Output: final centroids (cy)ıεлık, optimal assignment (pt.)) E R N while not converged do (pi.) t Sinkhorn (x, 7) // Compute OT Pr.j between data and centroids Cj EE= Pi, i S k // Update centroids e, to minimize cost
end while Return centroids and assignments p.
Step 2: Clustering Points. Once the centroids are computed, we propose different ways to cluster the data points:
Softmax conditionals. The conditional probability of point being assigned to centroid
is given by softmax on their distance: polcluster j|æ 2x) Σ 1
We add an extra temperature parameter T > 0. Larger temperatures yield more uniform assignments. This is the way points are classified Prototypical Networks.
Sinkhorn conditionals. The conditional probability of point being assigned centroid is given by the optimal transport plan Pi,j computed previously: pa (cluster= = j = x;) ΣØ Pc.s" Although there no temperature parameter to tune, the Sinkhorn algorithm
Pc.
has a regularization parameter 7 > O, which has similar effect the temperature, since using both are equivalent to rescaling the distance matrix [(bo(xz)) 12 Using Sinkhorn conditionals favors balanced clusters, but using Softmax conditionals provides no such guarantees.
5.2 PREDICTION AND META- DETTA-EVALUATION
Given a few-shot clustering or unsupervised few-shot classificaiton episode, we embed the raw data zi ha (xx). Then, cluster the support set in embedding space using Sinkhorn K-Means. Finally,

---

Under review as a conference paper at ICLR 2020
associate query set points with predicted clusters by finding their nearest-centroid in embedding space. We compute the clustering and unsupervised accuracies following Section 3.1.
5.3 TRAINING WITH A SUPERVISED SURROGATE Loss (META- (EETA-TRAINNNG
The most intuitive way to train Centroid Networks would be to train them end-to-end, by backprop- agating through Sinkhorn K-Means, which contains two nested loops. Although this is technically possible after defining smoother versions of the elustering/umsuperviiedd accuracies (by replacing the 0-1 loss with a cross-entropy), we did not have much success with this approach. Instead, we opt for the much simpler approach of training with a supervised surrogate loss. Since we have access to the ground-truth classes during meta-training, we can simply replace the centroids oj with the average of each class μ, M Σ, (x;)) 1 =j}. Then, we classify the query set points using either Softmax or Sinkhorn conditionals. Finally, we compute the log-loss on the query set and min- imize it using gradient descent. 6 The supervised surrogate loss is very simple, as it removes both the need to find the optimal cluster-class permutation and the the need to backpropagate through Sinkhorn K-means.
Center Loss. Additionally to the supervised surrogate loss, we use a center loss penalty (Wen et al., 2016). Center losses have been used in metric-learnigg methods to penalize the variance of each class in embedding space. See for instance Wen et al. (2016) where it is used in addition to the standard log-loss for learning discriminative face embeddings. Using center loss makes sense because there is no obvious reason why the surrogate loss (basically a cross-entropy) by itself would make the classes more compact in embedding space. However, compact clusters is an implicit assumption of K-means and Sinkhorn K-means, which makes it essential for having good validation performance. We find experimentally that center loss helps improve clustering and unsupervised accuracies, at the cost of making supervised accuracy slightly worse (we don't Ion't use it for training Protonets).
5.4 CLASS SEMANTICS CONSISTRNCY CRITERION
As preliminary attempt to quantify how consistent class semantics are across episodes, we define the Class Semantics Consistency Criterion as the following ratio:
unsupervised accuracy (Bayes)
~ CSCC == unsupervised acc. (Centroid Networks)
CSCC ==
supervised accuracy (Bayes)
supervised acc. (Prototypical Networks)
where we define the supervised and unsupervised Bayes accuracies as the highest possible accura- cies on a given supervised few-shot classification task and its associated unsupervised counterpart. Except for degenerate cases, the CSCC always varies between O (classes are totally inconsistent) and (classes are totally consistent). In practice, we approximate the CSCC by replacing the Bayes accuracies with the supervised accuracy of Protonets and the unsupervised accuracy of Centroid Net- works, but with the constraint that their backbone networks have exactly the same architecture. We point out that approximate CSCC is not rigorously defined and can potentially depend significantly on the chosen architecture and hyperparameters. However, we see as a first step towards quanti- fying the difficulty of few-shot learning benchmarks. The motivations for introducing unsupervised Bayes accuracy and CSCC are discussed more in depth in Sections B.5 and B.4.
6 EXPERIMENTS
We first confirm that Centroid Networks is a reasonable approach by comparing it against a state- of-the art few-shot clustering method (Section 6.1). Then, we attempt to use Centroid Networks to evaluate the difficulty (in terms of class semantic variability) of current few-shot learning bench- marks (Sections 6.2 and 6.3). In all cases, we train our method by minimizing the surrogate loss with Softmax conditionals combined with a center loss (both improve accuracies). Our method requires little to no tuning across datasets, and to show this, we run all experiments with the following default hyperparameters: temperature T' = 1, sinkhorn regularization = 1, center loss of 1, and Sinkhorn
'When using softmax conditionals, surrogate loss minimization reduces to the standard training of Proto- typical Networks. In fact, the reason for naming our method Centroid Networks is because they can be seen as replacing the Prototypes (class averages) by the Centroids (weighted cluster averages) during prediction.

---

Under review as a conference paper at ICLR 2020
Omniglot (CCN setting)
20. -way 20-shot Acc.
Few-shot Clustering (Clustering Accuracy)
K-Means (raw features)
21.7%*
CCN (KCL) (Hsu al., 2017)
82.4%
CCN (MCL) (Hsu et al., 2019)
83.3%
Centroid Networks (ours, protomet arch.)
86.8 % 0.6%
Centroid Networks (ours, CCN arch.)
86.6 % + 0.6%
Table I: Top: Centroid Networks vs. K-Means on raw and Protonet features. Bottom: Test clus- tering accuracies on Omniglot evaluation set, using the Constrained Clustering Network splits (Hsu et al., 2017) (much harder than Ravi splits). Numbers with a star* are those reported in (Hsu et al., 2019). We compared both using the Protonet Conv4 architecture and the architecture in (Hsu al., 2017) (CCN), which has more filters. The differences between the two architectures are not sig- nificant, All our accuracy results are averaged over 1000 test episodes with a fixed model, and are reported with 95% confidence intervals.
Omni5- 5-shot Omni 20-way 5-shot Acc.
minilNet 5-way 5-shot
Few-shot Clustering (Clustering Accuracy)
K-Means (raw images)
45.2%
0.5%
30.7%
0.2%
41.4% + 0.4%
K-Means (Protonet features)
83.5% 1: 0.8%
76.8%
0.4%
48.7% 1: 0.5%
Centroid Networks (ours)
99.6% + 0.1%
99.1 % + 0.1%
64.5% + 0.7%
Table 2: Top: Few-shot clustering accuracies for Centroid Networks vs. K-Means on raw data and Protonet features.
conditionals for training. The only exceptions for Omniglot-CCN, where we take center loss weight equal to 0.1, and for the Meta-Dataset, for which we take = 0.1 and a center loss weight of 0.01. Please refer the Appendix B.6 for ablation study on the effect of each trick.
6.1 SOLVING FEW-SHOT CLUSTERING
We start with experiments designed validate that Centroid Networks are competitive approach to learning how to categorize examples without labels (an important assumption behind our proposed CSCC).
[Table 1] For this, we consider the specific task of few-shot clustering and compare with Constrained Clustering Networks (Hsu et al., 2017: 2019), recent state-of-the learning to cluster method, on the same task as them, which we will denote Omniglot-CCN. Omniglot is resized to 32 x 32 and split into 30 alphabets for training (background set) and 20 alphabets for evaluation. The Omniglot-CCr CCN task consists in clustering each alphabet of the evaluation set individually, after training on the background set. This makes a harder task than standard few-shot classification on Omniglot, because characters from the same alphabet are harder to separate (more fine-grained), and because the number of ways varies from 20 to 47 characters per set. We run Centroid Networks with all default hyperparameters, except centroid loss of 0.1. The results given in Table I show that Centroid Networks outperform all "flavors" of CCN by a margin (86.8% vs. 83.3% highest). Furthermore, Centroid Networks also simpler and about 100 times faster than CCN, because they require to embed the data only once, instead of iteratively minimizing a KCL/MCL criterion.
However, wish point out that Centroid Networks are less flexible than CCNs, they require specifying the number clusters and making an assumption on the sizes of the clusters (in our case, equal size). For this reason, CCNs are assumptions cannot be made. That said, Centroid Networks are particularly suited to our CSCC metric for few-shot classification benchmarks, as they are very efficient and otherwise require strictly less information than supervised few-shot learning method would. Note that extending Centroid Networks to be as flexible as CCNs would be a promising direction for developing new learning-to- cluster methods.
appropriate for the general setting where such
We make the choice to apply our method on their task rather than the opposite because their method is much slower and more complicated to run. By solving the same task as them, we can compare directly with the results from their paper.
8

---

Under review as a conference paper at ICLR 2020
Omni 5-Way 5-shot Omni 20-way 5-shot
minilNet 5-way 5-shot
With Labels: Supervised Few-shot Classification (Supervised Accuracy)
Prototypical Networks 99.7%*
98.9%
68.7% 0.5%
No Labels: Unsupervised Few-shot Classification (Unsupervised Accuracy)
Centroid Networks (ours) 99.1% i 0.1%%
98.1%
0.1%%
55.3% 0.5%
Approximate Class Semantics Consistency Criterion
Approximate CSCC
99.4%
99.2%
80.5% 0.9%
Table 3: Using Centroid Networks to solve Omniglot and minilmageNet without using meta-testing labels (unsupervised seo-shot classification). We compare the unsupervised test accuracy of centroid networks with the supervised test accuracy of Protonets. Centroid Networks can solve Omniglot al- most perfectly (CSCC close 100%), which suggests the class semantics extremely consistent, while there is a small gap for minilmageNet (CSCC close to 80%), which suggests the class seman- tics are fairly consistent. Accuracy results are averaged over 1000 test episodes with fixed model, and are reported with 95% confidence intervals.
(Table 2] We also compare Centroid Networks with two baselines on Omniglot and minilmageNet (standard splits, see Appendix A.2). We run K-Means with K-Mcans++ initialization directly on the raw images and show that it performs very poorly even on Omniglot, which confirms the im- portance of learning an embedding function. We also run K-Means on pretrained Protonet features, which is more interesting comparison, since at the highest level, our method could be described as just clustering Protonet embeddings. It turns out that Centroid Networks still outperform K-Means on the embeddings by substantial margin on both Omniglot (99.6% vs. 83.5% for 5-way) and minilmageNet (64.5% vs. 48.7%), which confirms the importance of combining Sinkhorn K-Means and the center loss trick."
6.2 SOLVING OMNIGLOT AND MINIIMAGENET WITHOUT USING META-TEEE TESTING LABELS
We now come to the main contribution of this work, which is to assess the difficulty of current few-shot learning benchmarks, using CSCC.
[Table 3] We report the performance of CentroidNets on unsupervised few-shot classification tasks on Omniglot and minilmageNet. We also report the performance of Prototypical Networks for the standard supervised few-shot classification tasks. This comparison between the models is fair (Sec- tion B.3) even if they solve different tasks, because unsupervised few-shot classification is strictly harder than supervised few-shot classification (Section 3.1). Data splits and architectures are the same as in Protonets, and can be found Section A.2 of the Appendix." 9 For Omniglot, Centroid- Nets achieves nearly the same accuracy as Prototypical Networks despite using none of the labels of the support set. The CSCCs of 0.99 are nearly equal the maximum, which supports our hypothesis that Omniglot has nearly perfect class semantics consistency. For minilmageNet CentroidNets can still achieve an unsupervised accuracy of 55.3%, which of the same order as the supervised accu- racy of 68.7%, despite not using any labels from the support set. The CSCC of 0.80 is not high as Omniglot but still suggests that there is still a fairly high amount of class semantics consistency.
6.3 USING CSCC TO EVALUATE THE DIFFICULTY OF META-DATASED
[abble 4] We use approximate CSCC to evaluate the difficulty of Meta-Dataset, under the two set- tings presented in Triantafillou et al. (2019): meta-train on ILSVRC and meta-train on all datasets. Traffic Sign and MSCOCO are ovaluation-only datasets which are excluded from the meta- meta-training. Meta-cvaluation is done on all datasets. We use the same Resnet-18 architecture and hyperparam-
'Interestingly, on Onniglot 20 20-way 5-shot, the clustering accuracy of Centroid Networks is actually a bit higher than the supervised accuracy of Protonets (99.1% vs. 98.9%) despite using no labels from the support set. Although impressive, this result not paradoxical and only confirms that clustering accuracies cannot be directly compared with supervised accuracies (Section 3.1).
"Note that we exclude the I-shot setting from our experiments because it is trivial in our case. For unsuper- vised few-shot classification, the centroids would be equal to the prototypes up permutation, and Centroid Networks would reduce to Prototypical network (for evaluation).

---

Under review as a conference paper at ICLR 2020
Train only on ILSVRC
Train on All Datasets
Test Source
ProtoNets
CentroidNets
ApproxCSCC
PeotoNets
CentroidNets
ApproxCSCC
ILSVRC
44.12::1.08
0..40
59.832247
41.79:FT004
23.84:0.3 0.82
37.06
Omniglot
53,40+1.33
36,834
68.96+2.84
81.9310.7
66,25=1.12
80.86=1.66
Aircraft
45.29110.91
24.15
53.33 1.91
69.43 69.43110.8
57.50 1.01
82.82
1
Birds
63.59.10.94
1..08.1..05
64.61
64.7310.97
43,56 1.03
67.29
07.29-1889
Textures
61.78+0.77
39.63++0.70
64.15+1.39
66.35++0.73
43.50=0.76
65.57 1.35
QuickDraw
4..58.11.06
31.04.00.99
62.60
6..74-10.94
46,96
69.32
1.81
Fungi
35.2741.006
18.11 =0.71
2.553
38.94
1.08
2.76
55,89
VGG Flower
78.09 F0.82
47.98
1.996
01.44-11.30
84,45
0.74
55.11
65.26
1.26
Traftic Sign
11.08
22.0310.66
47.82
49.91
1.01
22.71
45.50
1.61
MSCOCO
35.63 1.03
F8.19
51.05
2.42
36.64
1.04
17.60
0.77
48.04
48.04+2.50
Table 4: Using Centroid Networks to solve Meta-Dataset without using meta-testing labels (unsu- pervised few-shot classification) under the two originally proposed settings: training on ILSVRC, and training on all datasets except Traffic Sign and MSCOCO. We report supervised test accuracy for Prototypical Networks (reproduced from the official implementation). unsupervised test accuracy for Centroid Networks (ours), and approximate CSCCs (their ratio). All numbers are in percentages, all accuracies are averaged over 600 test episodes.
eters for Protonets and CentroidNets, but CentroidNets is trained with an additional of center loss of 0.001 during meta-training. We pretrain on ILSVRC for the all datasets setting. The first ob- servation is that supervised/unsuperiied accuracies and approximate CSCCs, 10 are higher when training on all datasets instead of training on ILSVRC only, except for ILSVRC (since it is used in both trainings), Traffic Sign and MSCOCO. The fact that CSCC for Traffic Sign and MSCOCO is actually lower when training on more datasets either means that training on ILSVRC alone can sometimes be better for transfer learning, or is consequence that the sampling scheme is not op- timal (Triantafillou et al., 2019). Aircraft (53%) and Omniglot (69%) are the ones that benefit the most from training on all datasets in terms of CSCCs. We compare approximate CSCCs inside each sub-table." II High CSCCs in the all datasets sub-table suggest that those datasets have very self-consistent class semanties Omniglot and Aircraft both have very high CSCCs (more than 80%), while ILSVRC (57%) and Fungi (56%) have the lowest ones. It is less clear how to interpret the CSCCs in the only ILSVRC sub-table, but High CSCCs might suggest that those datasets have very similar class semantics with ILSVRC. Except Omniglot (69%), most datasets have fairly low CSCCs. It is interesting to note that some datasets higher CSCC than ILSVRC itself. We leave to future work to determine whether it means that ILSVRC is so inconsistent that it is easier to adapt to other datasets, or it is shortcoming of our metrie.
7 CONCLUSION
We proposed Centroid Networks for performing clustering without labels at meta-evaluation time, and with the idea of using it to assess the difficulty of few-shot classification benchmarks. First, we validate our method by beating state-of-the-ar few-shot clustering method (Hsu al., 2018) in the setting of a known number of equally-sizer clusters, with the advantage that our method is easier to train and orders of magnitude faster to run. Then, we define the CSCC metric from the unsupervised accuracy of Centroid Networks, and use it for quantifying the difficulty of current few-shot learning benchmarks in terms of class semantics consistency. We find that Omniglot has extremely consistent class semantics (CSCC close to 1), and that minilmageNee has fairly high CSCC as well (CSCC close to 0.8), which backs the intuition that its class semantics invariably correspond to object categorics. Our results the Meta-Datase benchmark show that it has much lower CSCCs than Omniglot in all settings, and lower CSCCs than minilmageNet in the ILSVRC only setting, which confirms that
Note that Meta-Dater Dataset has the particularity that meta- training and meta-testing consist of different groups of datasets, therefore CSCC does not only correspond to the class semantics consistency across episodes, but also accounts for the class semantics shift hetween training and testing.
II is unclear whether the values of the Deta-Dataset approximate CSCCs can be directly compared to Omniglot and miniImageNet because its number of ways and shots are variahle, and because Meta-Dataset has discrepancy between meta-training and meta-evaluation distributions which not the case for those datascts.
10

---

Under review as a conference paper at ICLR 2020
Meta-Dataset has harder and more diverse class semantics, As future work, would like to improve the CSCC by making more interpretable and less dependent on the backbone architectures.
REFERENCES
Pranjal Awasthi and Reza B Zadeh. Supervised clustering. In Neural Information Processing Sys- tems, 2010.
Mikhail Bilenko, Sugato Basu, and Raymond J Mooney. Integrating constraints and metric learning in semi-supervised clustering. In International Conference Machine Learning, 2004.
Guillermo Canas and Lorenzo Rosasco. Learning probability measures with respect to optimal transport metrics. In Neural Information Processing Systems, 2012.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In Neural Infor- mation Processing Systems, 2013.
Marco Cuturi and Arnaud Doucet. Fast computation of wasserstein barycenters. In International Conference on Machine Learning, 2014.
Thomas Finley and Thorsten Joachims. Supervised clustering with support vector machines. In International Conference on Machine Learning, 2005.
Thomas Finley and Thorsten Joachims. Supervised k-means clustering. Cornell Computing and Information Science Technical Report, 2008.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaminn meta-learning for fast adaptation of deep networks. In International Conference Machine Learning, 2017.
Aude Genevay, Gabriel Peyre, and Marco Cuturi. Learning generative models with sinkhorn diver- gences. In International Conference on Artificial Intelligence and Statistics, 2018.
Kyle Hsu, Sergey Levine, and Chelsea Finn. Unsupervised learning via meta-learning arXiv preprint 1810.02334, 2018.
Yen-Chang Hsu, Zhaoyang Lv, and Zsolt Kira. Learning cluster in order to transfer across do- mains and tasks. arXiv preprint 1711. 2017.
Yen-Chang Hsu, Zhaoyang Lv, Joel Schlosser, Phillip Odom, and Zsolt Kira. Multi-class classi- fication without multi-class labels. In International Conference on Learning Representattinns, 2019.
Siavash Khodadadeh, Ladislau Bölöni, and Mubarak Shah. Unsupervised meta-learning for few- shot image and video classification. arXiv preprint 1811.11819, 2018.
Brenden Lake, Ruslan Salakhutdinov, Jason Gross, and Joshua Tenenbaum. One shot learning of simple visual concepts. Proceedings of the Annual Meeting of the Cognitive Science Society, volume 33, 2011.
Luke Metz, Niru Maheswaranathan, Brian Cheung, and Jascha Sohl-Dickstein. Learning unsuper- vised learning rules. arXiv preprint /804.00222, 2018.
Liang Mi, Wen Zhang, Xianfeng Gu, and Yalin Wang. Variational wasserstein clustering. In Euro- pean Conference on Computer Vision, 2018a
Liang Mi, Wen Zhang, and Yalin Wang. Regularized wasserstein means based on variational trans- portation. arXiv preprint 1812.00338, 2018b.
George A Miller. Wordnet: a lexical database for english. Communications of the ACM, 38(11); 39-41, 1995.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. In International Conference Learning Representations, 2017.
11

---

Under review as a conference paper at ICLR 2020
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenenbaum, Hugo Larochelle, and Richard S Zemel. Meta-learning for semi semi-supervised few-shot classifica- tion. arXiv preprint 1803.00676, 2018.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-255 2015.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Neural Information Processing Systems, 2017.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE Con- ference Computer Vision and Pattern Recognition, pp. 1199-1208, 2018.
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Kelvin Xu, Ross Goroshin, Car- les Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle. Meta-dataset: A dataset of datasets for learning learn from few examples. arXiv preprint arXiv:1903.03000 2019.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Neural Information Processing Systems, 2016.
Kiri Wagstaff, Claire Cardie, Seth Rogers, Stefan Schrödl, et al. Constrained k-means clustering with background knowledge. In International Conference Machine Learning. 2001.
Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. A discriminative feature learning approach for deep face recognition. In European Conference on Computer Vision, 2016.
12

---

Under review as a conference paper at ICLR 2020
A APPENDIX: BACKGROUND AND IMPLEMENTATION DETAILS
A.1 SINKHORN DISTANCES
The Wasserstein-2 distance is distance between two probability masses p and q. Given a base distance d(x, x'), we define the cost of transporting one unit of mass from x to x' as d(x, z²)² The Wasserstein-2 distance is defined as the cheapest cost for transporting all mass from p to q. When the transportation plan is regularized to have large entropy, we obtain Sinkhorn distances, which can be computed very efficiently for discrete distributions (Cuturi, 2013; Cuturi Doucet, 2014) (entropy-regularizatie makes the problem strongly convex). Sinkhorn distances are the basis of the Sinkhorn K-Means algorithm, which is the main component of Centroid Networks. In Algorithm 1, describe the Sinkhorn algorithm in the particular case where we want to transport mass from the weighted data points (2,, R2) to the weighted centroids (cy, Cý), where Rg and Cj are the weights of the data points and centroids, respectively. In practice, leverage the log-sum-exp trick in the to avoid numerical underflows.
A.2 DATA SPLITS AND ARCHITECTURE FOR OMNIGLOT AND MinİIMAGENET EXPERIMENTS
For the embedding network for the Omniglot and minilmageNet, we reuse exactly the same simple convolutional architecture in Prototypical Networks (Snell et al., 2017), which consists of four stacked blocks (2D convolution with x 3 kernel and stride I, BatchNorm, ReLU, and max-pooling), the output of which is flattened. This results in a 64-dimensional embedding for Omniglot and 3000-dimessional embedding for miniImageNet. For minilmageNet, we pretrain the embedding function using prototypical networks to solve 30-way problems instead of 5, which is the recommended trick in the paper (Snell et al., 2017). 7). For the other settings, we train from scratch.
Omniglot (Lake al., 2011) consists of total of 1623 classes of handwritten characters from 50 alphabets, with 20 examples per class. Images are grayscale with size 28 x 28. We follow the same protocol in Prototypical Networks and use the "Vinyals" train/validation/test splits. We consider 5-way 5-shot and 20-way 5-shot settings (15 query points per class).
minimmageeet (Vinyals et al., 2016) consists of 100 classes, each containing 600 color images of size 84 x 84. We follow the "Ravi" splits: 64 classes for training. 16 for validation, and 20 for testing. We consider the 5-way 5-shot setting (15 query points per class).
B APPENDIX: : QUESTIONS AND ANSWERS
B. WHAT THE DIFFERENCE BETWEEN SINKHORN K-MEANS VS. REGULAR K-MEANS
We present our version of the Sinkhorn K-Means optimization problem, and compare it with regular K-Means. Both of them can be formulated as a joint minimization in the centroids ej Rd (real vectors) and the assignments pi.j 2 0 (scalars) which specify how much of each point z; is assigned to centroid oj:
K-Means. Note that compared to the usual convention, have normalized assignments Pig so that they sum up
K Σ
minimize
min 1=1 1=1
K
subject
Σ pt. = N' j=1
iE
Pij E {0, 1
1EI:N, jEl:K
13

---

Under review as a conference paper at ICLR 2020
Sinkhorn K-Means.
minimize
min
Σ
Ph.j
H(p)
subject to
Σ
Pt.)
1E1:N
N
j=1
N
Σ
pt.
jEL:K
K'
20
iEI:N, jel:K
where Hpp) - Pt.j Pi, Pi.j is the entropy of the assignments, and y 2 0 is parameter tuning the entropy penalty term.
Sinkhorn vs. Regular K-Means. The first difference is that K-Means only allows hard assign- ments Pi,j E {0, N) that is, each point v; assigned exactly one cluster oj. On the contrary, the Sinkhorn K-Means formulation allows soft assignments Pi.j E [o, I. but with the additional con- straint that the clusters have to be balanced, i.e., the same amount of points cluster >; Pi.j K. The second difference is the penalty term H (p) which encourages solutions of high-entropy, i.e., points will tend be assigned more uniformly over clusters, and clusters more uniformly over points. Adding vngropy-reguularizatoo allows us to compute Pt.j very efficiently us- ing the work of Cuturi (2013). Note that removing the balancing constraint >; Pi.j = π in the Sinkhorn K-Means objective would yield a regularized K-Means objective with coordinate update steps identical to EM in mixture of Gaussians (with pij updated using softmax conditionals).
soft-assigned each
B.2 WHY IS SINKHORN K- K-MEANS EXPECTED TO IMPROVE PERFORMANCE ?
The ablation study in Section B.6 shows that using Sinkhorn K-Means instead of K-Means is the most decisive factor in improving performance. There are mainly two possible explanations
1. Sinkhorn K-Means is particularly well adapted to the few-shot clustering and unsupervised few-shot classification problems because it strictly enforces the fact that the classes have to follow a given distribution (e.g. balanced), whereas K-Means does not.
2. Sinkhorn K-Means is likely to converge better than K-means due the entropy- regularization factor of the Sinkhorn distance.
To illustrate the second point, consider the limit case where the regularization factor of Sinkhorn distance goes to infinity (7 + oo). Then, the assignments in Sinkhorn K-Means become uniform (each cluster is assigned equally to all points), and all the centroids converge in one step - to the average of all the points, reaching global minimum. This is by no means rigorous proof, but the limit case suggests that Sinkhorn K-Means converges well for large enough This behavior to be contrasted with K-means, for which convergence is well known depend largely on the initialization.
B.3 WHAT THE EFFECT USING WEIGHTED UNWEIGHTED AVERAGES ?
One could argue that comparing CentroidNets with ProtoNets is unfair because using Sinkhorn K- Means leads to centroids which are weighted averages, whereas ProtoNet prototypes are restricted to be unweighted averages. Therefore, run Centroid Networks on minilmagenet, but under the constraint that centroids to be unweighted averages of the data points. To do so, starting from the soft weights, we reassign each data point only to its closest centroid, and compute the unweighted averages. The comparison between ProtoNets and CentroidNets now fair in the sense that both prototypes and centroids use unweighted averages.
Unsupervised accuracy minilmagenet is 0.5508 + 0.0072 for weighted average and 0.5497 + 0.0072 for unweighted average. The difference not significant.
Clustering accuracy on minilmagenet is 6642140.0064 for weighted average and 0.64174 0.0069 for unweighted average. The difference also significant.
14

---

Under review as a conference paper at ICLR 2020
This experiment suggests that using weighted averages does not bring an unfair advantage, and therefore does not invalidate our comparison. More generally, instead of trying to tune ProtoNets and CentroidNets as well as possible, we try to make ProtoNets and CentroidNets more comparable by using the same architectures and representation.
B.4 WHAT DOES UNSUPERVISED BAYES ACCURACY QUANTIFY
We define the unsupervised Bayes accuracy an unsupervised few-shot classification task distribu- tion as the highest achievable unsupervised accuracy. Just like the usual Bayes error is limited by label noise, the unsupervised Bayes accuracy limited by cluster-semantic noise of a task.
For illustration, consider the following unsupervised fhw-shot classification task distribution
1. Uniformly sample a random dimension j I (hidden to the algorithm)
2. Sample (iid, with probability 1/2) random binary vectors (ai)ısigD of dimension D (shown to the algorithm) and split them between support and query set.
3. Assign binary labels y = 2i to each vector (a:) (hidden to algorithm).
4. The goal is to cluster the support set and associate query set points with the support clusters.
Because the algorithm does not know which dimension j was sampled (i.e. the class semantic), does not know how to cluster the support set. Therefore, is just as good to make random predictions on the query set. Therefore the unsupervised Bayes accuracy is 0.5.
Now, consider the same task distribution, except the dimension index j is always fixed to 1. Af- ter meta-training, the algorithm can learn a representation mapping each vector to the value of its first dimension only. The support set can be clustered by grouping all Is together, and all Os to- gether. Each query point can then be unambiguously assigned to one of the clusters. The resulting unsupervised Bayes accuracy is 1.
Both task distributions would become equivalent if the algorithm had access to the class seman- ties j. Therefore, the two unsupervised few-shot tasks differ in difficulty only because of the un- certainty/variability on class semantics, and this is reflected in the difference in unsupervised Bayes accuracy.
B.5 IS CSCC A GOOD PROXY FOR BENCHMARK DIFFICULTY
CSCC attempts to quantify the importance of the supervision information, which is not directly related to the difficulty of few-shot learning problem. Indeed, the difficulty of few-shot learning problems come from many aspects, including but not limited to
visual difficulty (how hard is it to train a classifier on all the classes at the same time)
class semantic consistency (how much do the class semantics vary)
the goal to design meaningful benchmarks for supervised few-shot classification methods, important to understand which aspects make those benchmarks difficult. For instance, consider the limit case of a supervised few-shot classification task in which the same 5 classes are sampled over and over again. The visual difficulty might be extremely high (c.g. very fine-graines classification), which might lead people to believe that is a good benchmark (because it is hard and all methods achieve low accuracies). However, because there is variability at all in the class semantic con- sistency, such a benchmark does not evaluate at all the capacity few-shot methods of adapting to new tasks.
Our intent is not to introduce CSCC as a proxy for task difficulty (supervised accuracy of SOTA models might be fine for that purpose). Rather, introduce the CSCC attempt decouple the different axes of difficulty. Dividing the unsupervised Bayes accuracy by the supervised Bayes accuracy is rough way of normalizing away the visual difficulty (which affects both supervised and unsupervised accuracies) and focusing on the supervision information only.
15

---

Under review as a conference paper at ICLR 2020
Clustering
Train Cond
Eval Cond.
Center Loss ClusteringAce +
UnsuperAcc +/-
01
K-means++
Softmax
Softmax
No
83.80%
0.80%
85.20%
0.90%
02
K-means++
Softmax
Sinkhorn
No
87.32%
0.80%
0.80% 84.00%
03
K-means++
Sinkhorn
Sinkhorn
Yes
89.80%
0.70%
0.90% 86.00%
Sinkhorn K-means
Sinkhorn
Sinkhorn
No
99.20%
0.20%
98.50%
0.20%
05
Sinkhorn K-means
Softmax
Softmax
No
99.40%
0.16%
98.90%
0.20%
Sinkhorn K-means
Softmax
Softmax
99.50%
0.10%
99.00%
0.10%
07
Sinkhorn K-means
Softmax
Sinkhorn
Yes
99.50%
0.10%
99.00%
0.10%
Sinkhorn K-means
Sinkhorn
Sinkhorn
99%60%
0.10%
99.10%
0.10%
Figure 3: Omniglot 5 5-way 5-shot Ablation Study
Clustering
Train
Center Loss
ClusteringAcc
UnsuperAcc
M1
K-means++
Softmax
Softmax
No
50.50%
0.50%
39.40%
0.70%
M2
K-means+
Softmax
Sinkhom
No
57.40%
0.60%
38.60%
0.70%
M3
K-means**
Softmax
Softmax
Yes
50 60%
0.30%
39.50%
0.40%
M4
K-means++
Saftmax
Sinkhorn
Yes
56 98.90%
0.30%
38.70%
0.40%
K-means+*
Sinkhom
Softmax
60%
0.60%
36.70%
0.70%
M6
K-means++
Sinkhom
Sinkhorn
Yes
56. 60%
0.60%
35.80%
0.70%
M7
Sinkhom K-means
Sinkhom
Sinkhom
No
62. 30%
0.70%
52.50%
0.80%
M8
Sinkhorn K-means
Softmax
Softmax
No
83.70%
0.70%
54.80%
0.80%
M9
Sinkhomn K-means
Saftmax
Softmax
Yes
64.50%
0.80%
55.30%
0.80%
M10
Sinkhorn K-means
Seftmax
Sinkhorn
64.50%
0.80%
55.40%
0.80%
M11
Sinkhorn K-means
Sinkhorn
Sinkhorn
Yes
63%10%
0.70%
53.20%
0.80%
Figure 4: minilmageNet 5-way 5-shot Ablation Study
B.6
ABLATION STUDY
[Figures 3,4] We conduct an ablation study on Omniglot (5-way 5-shot) and minilmageNer (5-way 5-shot) determine the effect and importance of the various proposed tricks and components:
K-Means vs. Sinkhorn K-Means. From comparing O3 to O4, O1 to O5, M6 to M7, M1 to M8, it appears that using Sinkhorn K-Means instead of K-Means++ is the most beneficial and important factor.
Center Loss. From comparing 02 to O3, 05 to 06, 04 to O8, M7 to MI1, M8 to M9, center loss seems to be beneficial (although the significance is at the limit of the confidence intervals). It is the second most influential factor.
Softmax vs. Sinkhorn conditionals (at meta-training and meta-evaluation time). For train- ing, is clear whether using Sinkhorn or Softmax conditionals beneficial or not. For evaluation, from comparing M1 to M2, M3 to M4, M5 to M6, it seems that Sinkhorn con- ditionals are better if the metric clustering accuracy, while Softmax conditionals might be better if the metric is unsupervised accuracy, although the effect seems to be negligible (see how the color patterns are inverted).
16


Reviews:
[{"review_id": "rkeIbjh3tr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a new method for learning to cluster without labels at meta-evaluation time and show that this method does as well as supervised methods on benchmarks with consistent class semantics. The authors propose a new metric for measuring the simplicity of a few-shot learning benchmark and demonstrate that it is possible to achieve high performance on Omniglot and miniImageNet with their unsupervised method, resulting in a high value of this criterion, whereas the Meta-Dataset is much more difficult.\n\nThe paper is well written and generally very clear. I appreciate that the authors have highlighted the limitations of both their clustering method (that it requires more assumptions than CCNs) and their benchmark. The centroid method itself seems to draw heavily on pre-existing work, but uses a new similarity metric that improves performance beyond the current state-of-the-art on few-shot clustering tasks.\n\nThe authors acknowledge that the approximate CSCC metric they define is not consistent across architectures and hyperparameters. It is also a fairly simple metric, but nonetheless represents a novel contribution.\n\nOverall I feel that the paper introduces a well-defined problem and makes a step toward quantifying and resolving it. The experiments do a thorough job supporting the arguments of the authors.\n\n\nI have only minor issues that could help with the clarity of this paper: \n\nI wasn’t sure what was meant by “relabeling” the query set predictions in the text below Figure 2. \n\nI would have appreciated some discussion as to why Sinkhorn distance might be expected to improve performance .", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SyxI703nYr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is concerned with few-shot classification, both its benchmarks and method used to tackle it. The scope of the few-shot classification problem can be set relatively widely, depending on what data is available at what stage. In general few-shot classification is an important ability of intelligent systems and arguably an area in which biological systems outperform current AI systems the most.\n\nThe paper makes a number of contributions. (1) It suggests an approach to do a specific type of clustering and compares it favorably to the existing literature. In a specific sense the approach does not use supervised labels (“without labels at meta-evaluation time”). (2) It applies that approach to currently existing datasets and achieves “surprisingly high accuracies” in that setting, with the implication that this shows a weakness in these datasets when used for benchmarking (“too easy”). (3) It further suggests a metric, dubbed “class semantics consistency criterion”, that aims to quantify this shortcoming of current benchmarks on these datasets. (4) It assesses a specific meta-dataset using that metric, confirming it is harder in this sense, at least in specific settings.\n\nMy assessment of the paper is mildly negative; however this is an assessment with low confidence given that I am no expert on few-shot classification or related areas.\n\nWhile the authors first example (the “Mongolian” alphabet of the Omniglot dataset and geometric shapes falling into different categories) illustrates the problem space well and is indeed quite intuitive, the same cannot be said about either the specific setting they consider nor the metric they propose. It’s not immediately clear that the other approaches from the literature they compare their method to were conceived for the setting considered here, or indeed optimized for it. The authors do show good accuracy on clustering Omniglot characters without using labels and thus indeed demonstrate a high amount of class semantics consistency for that dataset. The results on miniImageNet are less clear-cut, and the results of the evaluation of the meta-dataset appear to depend on the specific setting considered. This makes it unclear to what extent the proposed metric is general and predictive. To their credit, the authors state that in future work they are looking to make their metric “more interpretable and less dependent on the backbone architectures”.\n\nI believe the paper might benefit from being given additional attention. A streamlined and more accessible version might well be an important contribution in the future.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "HklF3-A6tS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "<Paper summary>\nThe authors argue that the popular benchmark datasets, Omniglot and miniImageNet, are too simple to evaluate supervised few-shot classification methods due to their insufficient variety of class semantics. To validate this, the authors proposed clustering-based meta-learning method, called Centroid Network. Although it does not utilize supervision information during meta-evaluation, it can achieve high accuracies on Omniglot and miniImageNet. The authors also proposed a new metric to quantify the difficulty of meta-learning for few-shot classification.\n\n<Review summary>\nAlthough the main claim of this paper seems correct, it is not sufficiently supported by theory or experiments. My score is actually on the border line, but I currently vote for ``weak reject, because some points in the paper are ambiguous yet. Given clarifications in an author response, I would be willing to increase the score.\n\n<Details>\n* Strength\n + The paper is well-organized. Especially, the examples shown in the introduction greatly help understanding of what the authors argue in this paper.\n + A novel study on quantifying the difficulty of meta-learning.\n + The proposed CentroidNet performs well in the experiments.\n \n* Weakness and concerns\n - Does CentroidNet really work without labels during ``meta-validation\"? As far as I understand, ground truth clusters of the support set defined by the labels are required to compute the accuracies. Therefore, the labels seem to be required to validate the performance of the model. I think it should be ``meta-test.\"\n - The authors state ``The most intuitive way to train ..., we did not have much success with this approach\" in 5.3, but it is counter-intuitive. If the class semantics are similar among episodes, ``the most intuitive way\" should work, because it can learn the common semantics via meta-training. Further discussion about why it does not work is required.\n - The high performance of CentroidNet does not support the claim on the insufficient variety of the class semantics. According to ablation study, adopting Sinkhorn K-means is the most important factor to improve the performance. It means that adopting weighted average like in [R1] can also improve the performance of ProtoNet, which results in substantial difference in the performance between ProtoNet and CentroidNet that can deny the claim. \n - The definition of CSCC is not convincing. First, I could not get the meaning of ``unsupervised Bayes accuracy\" (supervised Bayes accuracy means 1 - Bayes error rate, right?). Second, CSCC seems to mainly quantify the importance of the supervision information during meta-learning, which is not directly related to the difficulty of few-shot learning problem. Intuitively, difficult few-shot learning problems should lead to lower supervised Bayes accuracy, which does not necessarily decrease CSCC. Third, what we can induce via comparing CSCC is not clarified in theory. The discussion in 6.3 is too subjective and specific for the case of training with ILSVRC/all datasets.\n - This paper lacks citing some closely related works [R1, R2].\n\n[R1] ``Infinite Mixture Prototypes for Few-Shot Learning,\" ICML2019\n[R2] ``A Closer Look at Few-shot Classification,\" ICLR2019\n\n* Minor concerns that do not have an impact on the score\n - Another arXiv paper related to this work: ``Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML\"\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is interested in assessing the difficulty of popular few-shot classification benchmarks (Omniglot and miniImageNet). A clustering-based meta-learning method is proposed (called Centroid Network), on which a metric is built (gap between the performance of Prototypical Networks and Centroid Networks). As noted by several reviewers, the proposed metric (critical for the paper) is however not motivated enough, nor convincing enough - after discussion, the logic in the metric reasoning seems to remain flawed.

Decision: Reject