The Role of Global Labels in Few-Shot Classification and How to Infer Them
Authors: Ruohan Wang
Massimiliano Pontil
Carlo Ciliberto
Venue: NeurIPS 2021 Poster (2021)
Keywords: Meta-Learning
Few-Shot Learning

Abstract: Few-shot learning is a central problem in meta-learning, where learners must quickly adapt to new tasks given limited training data. Recently, feature pre-training has become a ubiquitous component in state-of-the-art meta-learning methods and is shown to provide significant performance improvement. However, there is limited theoretical understanding of the connection between pre-training and meta-learning. Further, pre-training requires global labels shared across tasks, which may be unavailable in practice. In this paper, we show why exploiting pre-training is theoretically advantageous for meta-learning, and in particular the critical role of global labels. This motivates us to propose Meta Label Learning (MeLa), a novel meta-learning framework that automatically infers global labels to obtains robust few-shot models. Empirically, we demonstrate that MeLa is competitive with existing methods and provide extensive ablation experiments to highlight its key properties.

The Role of Global Labels in Few-Shot Classification and How to Infer Them
Ruohan Wang
Massimiliano Pontil
University College London Institute for Infocomm Research, A STSR wang_ ruohan@r0.n www.aaaniir..assaar..oo edu. sg
Isitituto Italiano di Tecnologia
University College London massimiliano pontil@iit it
Carlo Ciliberto University College London c. cciliberto@ucl.ac ac.uk
Abstract
Few-shot learning is central problem in meta-learning, where learners must quickly adapt to new tasks given limited training data. Recently, feature pre- training has become ubiquitous component in state-of-the-art art meta-learning methods and is shown to provide significant performance improvement. However, there limited theoretical understanding of the connection between pre-training and meta-learning Further, pre-training requires global labels shared across tasks, which may be unavailable practice. In this paper, we show why exploiting pre-training is theoretically advantageous for meta-learning, and in particular the critical role of global labels. This motivates us to propose Meta Label Learning (MeLa), novel meta-learning framework that automatically infers global labels to obtains robust few-shot models. Empirically, we demonstrate that MeLa is competitive with existing methods and provide extensive ablation experiments highlight its key properties.
Introduction
A central problem in meta-learning is few-shot learning (FSL), where new tasks must be learned quickly given limited amount of training data. FSL has drawn increasing attention recently due to the high cost of collecting and annotating large datasets. To tackle the challenge of model generalization in FSL, meta-learning leverages past experiences of solving related tasks by directly learning transferable knowledge over collection of FSL tasks. A diverse range of meta-learning methods tailored for FSL have been proposed, including optimization based [e.g. 1, 5, 27], metric learning [c.g. 20, 21, 25], and model-based methods [e.g. 7, 14, 19]. The diversity of the existing strategies raises a natural question: do these methods share any common lessons for improving model generalization and for designing future methods?
Several papers addressed the above question. Chen et al. [2] identified that data augmentation and deeper network architecture significantly improve generalization performance across several meta-learning methods. On the other hand, Tian et al. [23] investigated a simple yet competitive approach: a linear model on top of input embeddings learned via feature pre-training. This approach ignores task structures from meta-lcarning and merges all tasks into "flat" dataset of labeled samples. The desired input embedding is then learned by classifying all classes of the flat dataset.
Extensive empirical evidences supports the efficacy of feature pre-training FSL. Pre-training alone already outperforms various meta-learning algorithms [23]. Recently, it is used as ubiquitous
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Global Label
Local Label
Global Classifier
Local Classifier
Flower
Dog
Dog
Dog
Cat
Cat
Cat
3
Flower
Flower
Task A
4/9
Ship
xba
Ship
Ship
Flower
Car
Car
Dog
W
W W
Task
Y (Dog, Can, Ship)
(a) Global vs. Local labels
(b) Global to Local Classifier
Figure I: (a) Colored squares represent samples. Tasks A and B can be "merged" meaningfully using global labels, but not local ones. (b) A global classifier can be used as local classifiers given the indices of the intended classes to predict.
pre- pre-processing step in state-of-the-ar meta-learning methods [e.g. 18, 26, 27]. In particular, [27] reported that pre-training also significantly boosted earlier methods (see Fig. 2).
Despite the significant empirical improvement, there is limited theoretical understanding of why feature pre-training provides significant performance gain for meta-learning. On the other hand, pre- training requires task merging to construct the flat dataset, which implicitly assumes access to some notion of global labels consistently adopted across tasks. However, global labels may exist are inaccessible, such as when each task is annotated independently with only local labels. This renders direct task merging and consequently pre-training impossible (see Fig. Ia). Independent annotation captures realistic scenarios when tasks are collected organically (e.g from different users), rather than generated synthetically from benchmark datasets (e.g. miniMAGENET)) Possible scenarios include non-descript task labels (e.g. tasks with numerical labels) or even concept overlaps among labels across tasks (e.g. sea animals mammals).
In this paper, we address the issues raised above with the following contributions:
In Sec. 3, we show that feature pre-training directly relates to meta-leaming as a loss upper bound. In particular, pre-training induces a conditional meta-learning formulation [3, 26], which provides a principled explanation for the empirical performance gains.
In Sec. 4, we propose Meta Label Learning (MeLa), novel framework that automatically infers some notion of latent global labels consistent with local task constraints. The inferred labels enable to leverage pre-training to improve meta-learners' generalization performance, and bridges the gap between experiment settings with or without acesss to global labels.
In Sec. 5, we demonstrate empirically the competitive performance of MeLa over suite of benchmark datasets. We also present ablation studies highlight its key properties.
In Sec. 2, we first review the key notions of meta- meta-learning and FSL. The supplementary material contains proofs for the theoretical analysis, additional empirical results, and experiment setup.
2 Background
We formalize FSL in the context of meta-learning, followed by reviewing feature pre-training initially investigated in [23].
Meta-learning, FSL [4] considers a meta-training set of tasks T = (S;, Q0) with support set nh
S, [Zxj "r) and query set Q; {z1,, 31 w;) ) sampled from the same distribution. Typically, S, and Qi each contains small number of samples n, and ng respectively (fixed for all tasks for simplicity). We denote by D the space of datasets of the form S, or Q.-
FSL aims to find the best inner learning algorithm (referred to as base learner in the following) Alg(0,-) + F that maps supports sets to predictors f Alg(0, S), such that x + f(x) generalizes well on the corresponding query sets. The base learner is meta- veta-parametrized by 0 e e.

---

Formally, the meta-leariing problem for FSL is
min (SSQ)ET (AAg((O,S), Q) BEB
(1)
where ((S.Q)ET [7| ((S.))E denotes the empirical distribution over the meta-training set. The task loss L: : F x D is the empirical risk of the learner over query sets, according to an inner loss L
where y is the space of labels
D) (())))D [e(f(z),
(2)
Designing effective base learners is key focus in meta-learning literature and various strategies have been explored. We focus on broad class of methods that we call meta-representation learning [l, 6. 10, 15], which is remarkably effective in practice and closely related to feature pre-training. Meta-representation learning infers suitable embedding for base learners by solving
min BE
(( (((((V)()))) 1ho (Q))
where dig -
RD is feature embedding model and (D) & {(bo(a)) ()) y) D} the embedded dataset. In [1], the ridge regression estimator is chosen the base learner Alg(0, D) w(bra
w (bb) = argmin B(z.))EV)()) | - OneHot(y)) + Ai [[W]]
(4)
W
where d1 is a constant and OneHot(y) denotes the one-hot encoding of the label y. Among different base learner designs, ridge regression is often favored for its differentiable closed-form solution and the associated computational efficiency in optimizing (3).
Feature Pre-training. Feature pre-training has been widely used in meta-learning. It was investigated in-depth in [23]. Given the meta-training set T, "flat" dataset Dglobal constructed by merging all tasks in T:
Dglobal = D(7) =
= U (SUQ).
(5)
(8,Q)ET
We then learn emhedding function the Dglobal using the standard cross-entropy loss toe for multi-class classification:
argmin E( (We tho (x:) v)]. 8.W
(6)
After obtaining vee novel task may be solved in the embedded space by base learner. In [23], the authors recommended the regularized logistic regression estimator
Wee D) argmin
Lee (W, D) A2 |
w
for its empirical performance. Here X2 is regularization constant.
[23] demonstrated that feature pre-training standalone method already outperforms many sophis- ticated meta-learning strategies. has also become a ubiquitous component in most state-of-the-ar methods [27, 28, 29]. Furthermore, [27] demonstrated empirically that pre-training provides similar performance gain to earlier methods, making them (mostly) competitive with the state of the art. In Sec. 3, we show that feature pre-training directly relates to meta-learning as loss upper bound, and explains why pre-training contributes to improved performance.
3 Feature Pre-training as Meta-learning
In this section, we show how feature pre-training relates meta-learning as loss upper bound. More precisely, we show that the pre-training induces a special base learner with the corresponding meta-learning loss upper bounded by the cross-entropy loss (Eq. (6)). Consequently, minimizing the cross-entropy loss also indirectly solves the induced meta-learning problem. Further, we observe that the special base learner is conditional meta-learning formulation, which provides principled explanation for the improved performance.
3

---

Let T- = {(S,, Q01 be a meta tettattraning set. We denote the collection of query sets as Q = {Q.) = ((X,, Yilr= where we write Q, = (X,, Y2) as a tuple of input samples X, و- and their corresponding labels Y, = /yjeijn 12g For simplicity, we assume that the query sets are disjoint. namely Qi n Qu = 0 for any t * t'. We merge all query sets into a flat dataset D(2) (II.. = Ut. with N n.J.
Proposition 1. With the notation and assumptions introduced above, les C be the total number of classes in ())), and W ε RC the global classifter. Denote by W the sub-matrix with rows indexed by the sorted unique values' from Y. Then, for any embedding th X + R
E Ex (WI)), (1b < (EG de
(W (1).
(8)
We outline the proof strategy here and defer the details to the supplementary material. We first observe that ((2) has the same collection of samples as Q. Secondly, each matrix W is fact a task classifier for the query set (X, Y). Crucially, the likelihood of query samples p(Y|X) estimated by task classifiers W are no smaller than their likelihood estimated by the global classifier W, which forms the inequality in (8). Combining the two observations yields Prop. 1.
The key implication of Prop. is that, if global labels are available, we can use special base learner
Wglnbal (D) = -yglobal (X, W[Y],
for any dataset of the form D = (X,Y). This base learner only depends the intended classes (i.e. unique values in Y) to produce task classifiers and is independent of specific inputs from D (illustrated in Fig. 1b). We note that for any task (S, Q) E T, Wglobal (S) W'global (Q), since the support and query sets share the same class labels. These observations directly imply the following,
Lemma 2. For meta-training set T where tasks are annotated with global labels, we have
E(S.Q)ET ( vo v01) = Exxxne [Coo (oo (vs())))))
(10)
The left-hand side of (10) thus the meta-learning loss associated with base learner Wglobal (-). Hence, the combination of Prop. 1 and Lemma 2 implies that the empirical risk associated with this meta-learning loss upper bounded by the standard cross-entropy loss, namely the right-hand side of
(8). The connection shows that feature pre-training also solves meta-learning problem when we design the base learner to be (.)oball.).
Remark 1. The bound in Prop. 1 is tight when the standard cross-entropy loss is O.
Remark shows that all task classifiers W[Y] and the embedding function the are optimal, when standard cross-entropy is O. In practice, Remark is achievable since over-parametrized neural networks could obtain zero empirical loss.
We highlight two important properties for the base learner (global()). Firstly, Wglobal is not the base learner used during meta-testing since it cannot generalize to novel classes. In [23], Wglobal was simply replaced with Eq. (7) during meta-testing, while other works choose fine-tune the pre-trained embedding model dh with a new base learner intended for meta-testing using episodic training. The advantage of the fine-tuning strategy is clear since it optimizes vo for the actual base learner used during test time. This observation is well supported by existing empirical results, with many state-of-the-art methods [e.g. 28, 29] adopting the fine-tuning strategy and surpassing standalone pre-training.
Secondly, observe that (-)obai(-) describes conditional meta-learning problem: the global labels Y provide additional side information to facilitating model learning. Specifically, global labels directly reveal how task samples relate to one another and Wglobal could simply map global labels to task classifiers via W[Y]. In contrast, unconditional base learners (e.g. (4) and (7)) have to learn classifiers based on support sets, without access to task relations provided by global labels.
Global labels offer significant advantages to learning the embedding model vho. Denevi et al. [3] proved that conditional meta-learning is advantageous over unconditional formulation by incurring smaller excess risk, especially when the meta-distribution of tasks is organized into distant clusters
E.g. Y (6,6,4, 4, maps (4, 6, 9t rows of W. Also Fig, lb for illustration.

---

(see [3] for further discussion). In practice, global labels cluster task samples for free and improve regularization by enforcing each cluster (denoted by global label y) share vector W [yy] for all task classifiers. The above analysis explains why pre-training yields significantly more robust the than many (unconditional) meta-learning methods.
Algorithm 1: MeLa
80
Pretrain
ProtoNet
MatchNot
ProtoNet-
Input: meta-training set 7 Qi) <gg ,, (( (((w((s((S) the (Q))))
Accuracy
70
MatchNet Pretrain
Global clusters G LeanLabbelr T)
0g = MetaLearn(G;
55
Return 4/
1-shot
5-shot
Figure 2: Effect of Pre-training miniIMAGENET
In Fig. 2, plot several existing results from [2, 27] highlight the contribution of the pre- training. For miniMMAGENET, pre-training accounted for on average 9% improvement ProtoNet and MatchNet. In addition, pre-training alone yields a performance mostly competitive with both ProtoNet and MatchNet, suggesting that the pre-training contributes far more towards generalization performance compared to the specific meta-learning strategies deployed. Similar trends are observed for other methods and datasets, both in the previous works and our experiments. The extensive empirical results consistent with the theoretical advantages of conditional meta-learning.
Meta Label Learning
In Sec. 3, we analyzed the theoretical advantages of feature pre-training in connection to meta- learning, well as the critical role of global labels in learning robust embedding model. However, we argue that leveraging global labels are problematic for meta-learning. Firstly, global labels may be unavailable or inaccessible in practical applications, when meta-training tasks are collected and annotated independently discussed in Sec. I, rendering pre-training inapplicable. Secondly, global labels oversimplify meta-learning: they directly reveal how input samples relate to one another across tasks, while the goal of meta-learning is precisely learning such cross-task relations and extracting transferable knowledge. As highlighted in Sec. 3, pre-training contributes more towards test performance than the meta-learning strategies employed, making it difficult to assess the relative merits of different strategies.
To address these issues, we present a novel meta meta-learning framework that does not require access to global labels. In particular, our framework automatically infers some notion of latent global labels across tasks, therefore bridging the experiment settings with and without global labels.
Alg. 1 outlines our proposed strategy. We first meta-learn embedding function vi as tool to measure sample similarity. Secondly, we introduce a labeling algorithm for clustering task samples while enforcing local task constraints in the training data. The resulting clusters are used as inferred global labels. Lastly, we may apply any existing meta-learning strategy capable of leveraging global labels to obtain the final model?.
We focus on the labeling algorithm since other components of MeLa are standard procedures. The labeling algorithm takes a meta- meta-training set input and outputs set of clusters to represent global labels. The algorithm consists of clustering step for updating centroids and a pruning step for merging small clusters. The algorithm is presented in Alg. 2.
Clustering Step. The procedure exploits local labels from each task to guide sample assignments. For each task, local labels are used to enforce two constraints: samples with the same local label should be assigned the same global label, while samples from different local classes should be assigned different global ones. Formally, given set of cluster centroids G {gg ) we assign all
We note that [9] is a loosely similar strategy: they use fully unsupervised feature representation for clustering to infer auxiliary labels, which then used to refine the representation.
5

---

Algorithm 2 LearnLabeler
Input: embedding model vig. meta-training set T (St, Q+)r=) number of classes in task K Initialization: sample tasks from T to initialize clusters G {g)})=1-
While [G] has not converged:
N₂ 1 for each De E E
For (S, Q) E T:
Match global clusters V 18 via (11) for task (X,Y) SUQ
If V has K unique clusters
Update cluster for each E via (12)
+ (g E G, No z threshold in(13)
Return G
samples {m:)/m sharing the same local label within a task to single global cluster as follows,
0-
argmin
(2.)) go
(1I)
We apply (11) to each class of samples in task, matching K clusters in total. For simplicity, discard tasks which multiple local classes map to the same cluster. Otherwise, we update each matched cluster Je- and sample counts No- with
Ny 9u- + (v)(a)))) and N₂- = N₂- 11,
9o-
(12)
N
Pruning Step. We present simple pruning strategy to regulate the number clusters. Under the mild assumption that each cluster is equally likely appear in task, a cluster v is sampled with probability J for each K- K-way classification task. The number of samples N, assigned to cluster thus follows a binomial distribution with N, ( p) where we recall T the size of the meta-training set. We remove any cluster below the threshold
N₂ No QV Var(
(13)
where N₂ is the mean of N,, Var(NV, the variance, and q hyper-parameter controlling the the aggressiveness of the pruning process.
Alg. initializes a large number of clusters and populates the centroids with mean class embeddings computed from random tasks in T. For J initial clusters, [ tasks are needed since each task contributes K embeddings. The algorithm alternates between clustering and pruning to refine the centroids and estimate the number of clusters. When the number of existing clusters longer changes, the algorithm terminates and returns the current centroids G. Given the clusters G, all samples from the meta-training set can be assigned global labels.
We comment on two important points about Alg. 2. Firstly, the two hyperparameters, initial cluster count and pruning threshold, only necessary when global labels unavailable, since they determine the appropriate number clusters. In contrast, access to global labels implies that the number of clusters and even the number of samples for each cluster is known. This further shows how global labels could oversimplify meta-learning as discussed earlier. Secondly, Alg. 2 differs from the classical K-mean algorithm [11] by exploiting local information to guide the clustering process, while K-mean algorithm is fully unsupervised. We will show empirically that enforcing local constraints is necessary for learning robust models.
Meta-Learning with Inferred Labels. After obtaining the inferred labels, we may apply a wide range of meta-learning algorithms (such as [27, 28] that exploit global labels) to obtain the final model. To highlight the efficacy of pre-training and the robustness of the proposed labeling algorithm, choose [23] without additional fine-tuning in this work.
5 Experiments
We evaluate our proposed method on several benchmark datasets, including ImageNet variants, CIFAR variants, and a subset of MetaDataset [24]. As discussed earlier, we adopt independent

---

annotation with local labels only for all experiments. Model details and hyperparameter values are included in the supplementary material. Due to space constraint, CIFAR experiments and some ablation studies are also presented in the supplementary material.
The experiments aim address the following questions: 1) How does MeL. compare to existing algorithms? 2) How does pre-training affect the performance of meta-learning algorithms? 3) Does McLa learn meaningful clusters?
Experiments on ImageNet Variants. We compare MeLa to a representative set of meta-learning algorithms miniMMAGENET [25] and fieredIMAGENET [17]. For completeness, include methods requiring access to global labels. However, we emphasize that these methods are not directly comparable to MeLa, since access to global labels provide significantly more information to meta- learners as discussed previously. These methods are intended to demonstrate the effect of pre-training on generalization performance. We also include self-supervised methods in the comparison.
Table I: Classification accuracy of meta-learning models minilMAGENET and tieredd
MiniLMAGENET
MAGEe 5-shot
1 1-shot
5-shot
1-shot
Global Labels
LEO [19]
61.7 + 0.7
77.6 + 0.4
66.3 + 0.7
81.4 0.6
RFS [23]
62.0 + 0.4
79.6 0.3
69.4
84.4
FEAT [28]
66.7 = 0.2
82.0 0.1
7
84.8 0.2
FRN [27]
66.4 + 0.2
82.8 0.1
71.2
86.0 0.2
Local Labels
MAML [5]
4.
0.9
1.8
70.3 + 0.8 72.7
ProtoNet [20]
19.4 0.8
6.72
0.9
R2D2 [1]
51.9 0.2
6.2
MetaOptNet [10]
62.6 0.6
0.5
81.5 6.70
Shot-free [16]
n/a
7//
6/a
8/.
Initial Embedding (Eq. (3))
60.2 - 0.3
0.5
78.9 0.5
MeLa (ours)
62.0 0.4
79.6 0.3
69.1 0.5
84.2
No Labels (Self-Supervised)
MoCo [8] (reported in [23])
54.2 + 0.9
73.0 + 0.6
CMC [22] (reported in [23])
56.1 = 0.9
73.9 + 0.7
Similar to Fig. 2, Tab. clearly shows the significant advantages of having access global labels: all methods exploiting pre- vee-trainnn achieves noticeably higher generalization performance compared to methods without global labels. The results consistent with our theoretical analysis that conditional meta-learning is more advantageous. Further, we observe that global labels not only enable pre- training but also flexible task sampling, including practical heuristies such sampling more shots and more classes per task during meta-traing meta-training [10, 27, 28]. All of the above contribute to generalization performance, and it is clear that global labels should be used when available.
In our experiment setting of "local labels" only, MeLa outperforms all baselines three out of four settings, and obtains performance comparable to [10] in the remaining one. We highlight the comparison between the initial embedding ei obtained via (3) and the final embedding viğ obtained by MeLa as they share identical experimental setups. It is thus easy to attribute the performance improvements to the proposed algorithm and the effect of pre- pre-training. the average test performance by over 2% in addition, we observe that MeLa obtains performance comparable to RFS [23], which is the oracle setting (i.e. access to ground truth global labels) for MeLa.
particular, MeLa improves and over 4% in tieredMMAGENeT. In
While FEAT and FRN outperforms MeLa in Tab. 1, we reiterate that methods exploiting global labels are not directly comparable, as discussed earlier. In addition, the performance of Mel.a could be readily improved via more sophisticated data augmentation fine-tuning. We explore one such variant for McLa in Appendix B.2.
7

---

Table 2: The effects of initial embedding function on the labeling algorithm
Dataset Replacement
minilMAGENET
MAGENNT
Yes
No
Yes
No
Percentage of Tasks Clustered (%)
100
98.6
99.9
89.5
Clustering Ace (%)
100
99.5
96.4
96.4
1-shot Acc (%)
62.0 = 0.4
61.8 + 0.5
69.1 0.5
69.1 + 0.5
5-shot Acc (%)
79.6 0.3
79.4 0.4 0.4
84.2 + 0.3
84.1 0.3
Table 3: Comparison between Alg. and K K-mean Clustering
minlMAGENET 1-shot
IieredIMAGENET 1-shot
Cluster Alg.
Cluster Acc
5-shot
Cluster Ace
5-shot
Alg. (McLa)
100
62.0 = 0.4
79.6 0.3 76.9 0.3
96.4 28.2
69.0 = 0.5
8..1
K-mean
84.9
60.7 + 0.5
64.8 + 0.6
78.8
In [23], the authors also adapted two state-of ahr- self-supervised methods MoCo [8] and CMC [22] for FSL. In the adapted methods, labels are not explicitly provided as input, but used to construct contrasting samples for learning self-supervised embedding. MeLa also outperforms them in Tab. 1.
Robustness of the Labeling Algorithm. A potentially trivial solution for clustering samples without any learning is by looking for identical samples: identical sample appearing in multiple tasks would allow several local classes to be assigned to the same cluster. To demonstrate that Alg. is not trivially matching identical samples across task, we introduce more challenging experiment setting: each sample only appears once in the meta-training set. This implementation constructs the meta-training set by sampling from flat dataset without replacement'. Consequently, Alg. 2 must only rely on the initial embedding function vg for estimating sample similarity. We evaluate MeLa under this setting and report the results in Tab. 2.
In Tab. 2, clustering accuracy is computed by assigning the most frequent ground truth label each cluster as the desired target. In addition, percentage of tasks clustered refers to the tasks that map to K unique clusters by Alg. 2. The clustered tasks satisfy both constraints imposed by local labels and are used for pre-training.
The results suggest that Alg. 2 is robust in inferring accurate global labels, even when samples do not repeat across tasks. The no-replacement setting also has negligible impact on test performance. In particular, we note that minIMAAGENET is particularly challenging under the new setting, with only 384 tasks in the meta-training set. In contrast, the typical saaple--wthrrepacemmee setting has access to unlimited number of tasks for training.
The high clustering accuracy implies that the meta-distribution underlying the meta-training set is near perfectly recovered. MeLa thus effectively bridge the gap between experiment settings with and without access to global labels respectively. Using the inferred labels, we may apply wide range of meta-learning algorithms to obtain the final model. We may also adopt flexible task sampling, such as sampling more shots and classes per task [10, 27, 28], for better generalization performance.
The Importance of Local Constraints. The clustering process enforces consistent assignment of task samples given their local labels. To understand the importance of enforcing these constraints, we consider an ablation study where Alg. 2 is replaced with standard K K-mean algorithm, while other components of MeLa remain unchanged. K Kmmean algorithm is fully unsupervised and ignores any local constraints. We initialize the K-mean algorithm with 64 clusters for MiGEIMAGENET and 351 clusters for rieredIMAGENES the actual numbers of classes in respective datasets.
Tab. 3 indicates that enforcing local constraints is critical in accurately inferring the global labels, as measured by clustering accuracy. addition, lower clustering accuracy directly translates to lower test accuracy during meta-testing, suggesting that sensible task merging is important prerequisite for feature pre-training. In particular, test accuracy drops by over 5% for tiGEedI when K K-mean algorithm ignores local task constraints.
For instance, miniIMAGENET (38400 training samples) can be split into 384 tasks of 100 samples in this
setting.
8

---

Table 4: Meta-learning performance on MetaDataset subset (Aircraft, CUB and VGG)
Algorithm
1-shot
5-shot
FEAT (no pre-train)
60.9 + 0.7
75.0 + 0.5
FRN (no pre-train)
63.1 + 0.7
79.7 = 0.5
Initial Embedding Eq. (3)
61.9 0.5
78.6 0.4
MeLa
66.3 = 0.5
82.4 0.3
Between the two local constraints, we note that (11) is more important for accurately inferring global labels. Specifically, the clustering step improves accuracy by averaging the votes from all samples sharing the same local label in a task. On the other hand, the constraint on matching K unique clusters are satisfied by almost all tasks empirically (see Tab. 2).
Experiment on MetaDataset. We further evaluate MeLa on MetaDataset [24], a collection of inde- pendently annotated datasets designed for meta-learning. MetaDataset presents more challenging setting by including more diverse samples from different image domains.
We choose Aircraft, CUB and VGG flower datasets from MetaDataset for the experiment. The chosen datasets are all intended for fine-grained classification in aircraft models, bird species and flower species respectively. We compare MeLa against FEAT and FRN, two saate-of-the-ee methods. Since only local labels are used, all models are trained without pre-training For meta- tittattraniigg all models are trained on tasks sampled from the three datasets. For meta-testings we sample 1500 tasks from each dataset and report the average aceuracy. Test accuracy on individual datasets are included in Appendix B.4.
Tab. 4 shows that Mel.a outperforms both FEAT and FRN when pre-training becomes inapplicable in the "local label" setting. In particular. FEAT performed relatively poorly since it is not designed train from random initialization. In addition, MeLa improves upon the initial embedding by over 4%, similar to the performance gain observed in tieredI The results further validate the efficacy of pre-training even when the labels are inferred, consistent with our theoretical analysis. Lastly, we report that the clustering accuracy in this experiment is 79.3%, with a 94.7 percent of the tasks clustered. The results suggest that MeLa's performance is robust to noise in the inferred labels.
6 Discussion
In this paper, studied the effect of pre-training and the critical role of global labels in meta-learning. We showed that pre-training closely relates to meta-lcarning as a loss upper bound, and induces a conditional meta-learning formulation that explains the improved empirical performance. The effect of pre-training is consistently demonstrated in existing results and our experiments. The connection between meta-learning and pre-training opens up new opportunities of directly applying existing techniques from supervised learning towards meta-learning. For instance, model distillation [23] has been shown to further improve pre- nee-rrainnng performance.
We also proposed a practical framework to infer global labels, for settings when they are unavailable. We demonstrate that Mela is robust and accurate in inferring global labels, and achieves generalization performance competitive with state-of-the-an methods. In the ablation studics, we observed that meta-learning methods implicitly learn to cluster samples across tasks, even when samples do not repeat. In addition, explicitly enforcing the local constraints critical for accurately inferring global labels and learning robust few-shot models.
Limitations and Future Work. We close by discussing some limitations of our work and directions of future research. In this paper, we focused on understanding the connection between pre-training and meta-lea and evaluated MeLa on benchmarks with globally disjoint classes. the future, we intend to extend our method to settings where global labels are ill-defined, such when classes are overlapping or hierarchical. One possible approach is to assign soft labels or multiple labels to samples, thus capturing more complex relationship between classes.

---

Broader Impact
Meta-learning aims construct learning models capable of learning from experiences, Its intended users are thus primarily non-experts who require automated machine learning services, which may occur in a wide range of potential applications such as recommender systems and autoML. The authors do not expect the work to address or introduce any societal or ethical issues.
Acknowledgments and Disclosure of Funding
The authors would like to thank the anonymous reviewers for their comments and suggestions. This work was supported in part by Career Development Fund (grant C2108 12045) from A*STAR Singapore, SAP SE, Royal Society (grant SPREM RGSIRI 149) and Amazon Research Award (ARA).
References
[1] L. Bertinetto, J. F. Henriques, P. H. Torr, and A. Vedaldi. Meta-leamingg with differentiable closed-form solvers. International conference on learning representations, 2019.
[2] W.-Y. Chen, Y.-C. Liu, Z. Kira, Y.-C. F. Wang, and J.-B. Huang. A closer look at few-shot classification. In International Conference on Learning Representations, 2018.
[3] G. Denevi, M. Pontil, and C. Ciliberto. The advantage conditional meta-learning for biased regularization and fine-tuning, In Advances in Neural Information Processing Systems, 2020.
[4] L. Fei-Fei, R. Fergus, and P. Perona. One-shot learning of object categories. IEEE transactions on pattern analysis and machine intelligence, 28(4), 2006.
[5] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference Machine Learning- -ailiine
70. JMLR. org, 2017.
[6] L. Franceschi, P. Frasconi, S. Salzo, Grazzi, and M. Pontil. Bilevel programming for hyperparameter optimization and meta-learning. In International Conference on Machine Learning, pages 1568-1577, 2018.
[7] D. Ha, A. Dai, and Q. V. Le. Hypemetworks, arXiv preprint 1rX9v:160900910 2016.
[8] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 9729-9738, 2020.
[9] K. Hsu, S. Levine, and C. Finn. Unsupervised learning via meta-learning. In International Conference on Learning Representations, 2018.
[10] K. Lee, Maji, A. Ravichandran, and S. Soatto. Meta-learning with differentiable convex optimization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10657-10665, 2019.
[11] S. Lloyd. Least squares quantization in pem. IEEE transactions information theory, 28(2):
129-137, 1982.
[12] Mangla, N. Kumari, A. Sinha, M. Singh, B. Krishnamurthy, and V. N. Balasubramanian. Charting the right manifold: Manifold mixup for few-shot learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages
2020.
[13] B. Oreshkin, P. R. Lópe7, and A. Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. In Advances Neural Information Processing Systems, pages 721-731, 2018.
[14] H. Qi, M. Brown, and D. G. Lowe. Low-shot learning with imprinted weights. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 5822-5830. 2018.
[15] A. Raghu, M. Raghu, S. Bengio, and O. Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of maml. arXiv preprint arXiv: 190.:099990095,, 2019.
[16] A. Ravichandran, R. Bhotika, and S. Soatto. Few-shot learning with embedded class models and shot-free meta training. Proceedings of the IEEF/CVE International Conference on Computer Vision, pages 331-339, 2019.
10

---

[17] M. Ren, E. Triantafillou, S. Ravi, Snell, K. Swersky, J. B. Tenenbaum, H. Larochelle, and R. S. Zemel. Meta-learning for supervised few-shot classification. International conference on learning representations, 2018.
[18] P. Rodríguez, I. Laradji, A. Drouin, and A. Lacoste. Embedding propagation: Smoother manifold for few-shot classification. In European Conference on Computer Vision, pages 121-138. Springer, 2020.
[19] A. A. Rusu, D. Rao, J. Sygnowski, O. Vinyals, R. Pascanu, S. Osindero, and R. Hadsell. Meta-learning with latent embedding optimization. International conference learning representations, 2019.
[20] Snell, K. Swersky, and R. Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, pages 4077-4087 2017.
[21] F. Sung, Y. Yang. L. Zhang. T. Xiang, P. H. Torr, and T. M. Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1199-1208, 2018.
[22] Y. Tian, D. Krishnan, and P. Isola. Contrastive multiview coding. arXiv preprint arXiv: 190.v:9906.05844 2019.
[23] Y. Tian, Y. Wang. D. Krishnan, J. B. Tenenbaum, and P. Isola. Rethinking few-shot image classification: a good embedding is all you need? European Conference on Computer Vision, 2020.
[24] E. Triantafillou, T. Zhu, V. Dumoulin, P. Lamblin, U. Evci, K. Xu, R. Goroshin, C. Gelada, K. Swersky, P.-A. Manzagol, and H. Larochelle. Meta-dataset: A dataset of datasets for learning to learn from few examples. In International Conference on Learning Representations, 2019.
[25] O. Vinyals, C. Blundell, T. Lillicrap. D. Wierstra, et al. Matching networks for one shot learning.
In Advances in neural information processing systems, 2016.
[26] R. Wang, Y. Demiris, and C. Ciliberto. Structured prediction for conditional meta-learning. Advances in Neural Information Processing Systems, 2020.
[27] D. Wertheimer, L. Tang, and B. Hariharan. Few-shot classification with feature map reconstruc- tion networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8012 8021, 2021.
[28] H.J. Ye, H. Hu, D.-C. Zhan, and F. Sha. Few-shot learning via embedding adaptation with set-to-set functions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8808-8817, 2020.
[29] C. Zhang, Y. Cai, G. Lin, and C. Shen. Deepemd: Few-shot image classification with differ- entiable earth mover's ' distance and structured classifiers. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 12203-12213, 2020.
[30] H. Zhang, M. Cisse, N. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk mini- mization. In International Conference on Learning Representations, 2018


Reviews:
[{"review_id": "sXxV1WG2wq", "reviewer": "Reviewer_VigG", "summary": "Authors analyze the role of global pretraining in few-shot classification, establishing classification loss over all classes as an upper bound on the classification losses from particular tasks. Authors then consider the case where task-based labels are available but are not easily collated into a global set of labels. They introduce a task-aware clustering-based method for assigning global labels to embeddings from a task-pretrained feature extractor, allowing them to achieve near-oracle performance in settings where global labels are absent. The method improves upon task-based pretraining alone by a substantial margin. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "STRENGTHS:\n\nPaper tackles a practical, (mostly) realistic, and difficult problem setting. It addresses a fundamental question in few-shot learning (the nature and role of global pretraining). The proposed clustering mechanism is elegant. Behavior is well-explored and analyzed beyond simple accuracy comparisons to prior work. Paper is organized and presented well, and limitations are addressed. \n\nWEAKNESSES:\n\n- I question somewhat the motivation of the paper. Is access to global labels at train time really that unrealistic? This may be opening an unproductive debate around epistemological differences (which came first, the flat dataset or the task collection?), but I was under the impression that a large, globally-labelled dataset at training time is an explicit assumption of few-shot learning, especially given the existence of ImageNet as a permanent fallback option. I can envision a practical application where we would like to merge datasets with label sets that overlap but don’t match exactly – but how often does this actually occur in practice, and how often to the degree envisioned in this paper, where every task is labeled independently, despite substantial image overlap between tasks? Perhaps a more concrete real-world scenario would help here. \n\n- Theoretical analysis in section 3 is somewhat oversimplified. The upper-bound analysis only holds for one particular local classifier: W[Y]. While it is true that W[Y] is _a_ local classifier, it is not _the_ local classifier from eq.3. In fact, the ridge-regression classifier from eq.3 is almost certain to have a higher cross-entropy loss than W[Y], as it is minimizing a different loss (mean-squared-error) and experiencing a heavy small-sample bias at the same time. Thus contrary to what’s stated, we cannot actually be sure that global loss is an upper bound on episodic loss more broadly, though the intuition does make sense. \n\n- Results are somewhat misleading. The choice to avoid models with data augmentation in Table 1 means that all baselines are from 2019 or earlier. MeLa underperforms when more recent models are included – in the global category, [1], [2], and [3] all outperform MeLa, and in the local category, [3] without global pretraining (Table 9) does the same. Basic data augmentation (random crop, flip, color jitter) and global pre-training have become popular for mini-ImageNet and tiered-ImageNet since 2019 ([4] from the paper, [1,2,3,4]), and so while the organizational reasoning for Table 1 makes sense, it also conveniently removes all of these latest models simply because they don’t report numbers without data augmentation. I recommend adding a version of your model with basic augmentation to supplementary, as an apples-to-apples comparison to more up-to-date baselines. Without this, MeLa only outperforms prior work on a technicality. \n\n- Based on section 5.3, the number of tasks is always large relative to the number of global classes. The class distribution is also balanced. I worry that in this setting, class overlap between tasks becomes substantial, and so clustering becomes easy. MeLa approaches oracle performance on the chosen benchmarks but does it generalize to harder, more realistic settings with less overlap and/or class imbalance? Lines 316-320 do discuss these kinds of limitations, though, so perhaps simply add these scenarios to that discussion.  \n\n[1] Zhang et al. CVPR 2020, DeepEMD: Few-Shot Image Classification with Differentiable Earth Mover’s Distance and Structured Classifiers\n\n[2] Ye et al. CVPR 2020, Few-Shot Learning via Embedding Adaptation with Set-to-Set Functions\n\n[3] Wertheimer et al. CVPR 2021, Few-Shot Classification with Feature Map Reconstruction Networks\n\n[4] Wang et al. 2019, Simpleshot: Revisiting nearest-neighbor classification for few-shot learning\n\nPOST-REBUTTAL:\n\nThank you for your detailed response, this is quite helpful. On reflection I was overly harsh in my initial assessment and most of my concerns have been addressed (1-4, 6). Regarding 5: my concern is that the proposed clustering problem may be artificially easy for mini-ImageNet and tiered-ImageNet, because classes are balanced within tasks, class assignments to tasks are IID, there is zero domain shift between tasks, and each individual class appears throughout a range of different tasks and class combinations. All of these could easily not be the case in a less contrived benchmark. Authors rightly point out that this is an understudied problem for few-shot learning in general, and partially address some of these assumptions in the paper (5.3) and rebuttal (4). However, as things stand, MeLa is evaluated primarily on toy benchmarks naively derived from a setting with obvious ground truth global labels (a concern shared by reviewer Cshp), and so it is still not clear to me on a practical level to what degree this is a first step into a new and interesting area of few-shot research, or merely proof of concept (or both). ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors discuss technical limitations (though note weakness #4 above), but do not address societal impacts. I suggest adding this discussion to the main paper or supplementary, even if just to briefly acknowledge that the risks broadly associated with few-shot classification also apply in this case. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "fyaIX8xa4M_", "reviewer": "Reviewer_CWrb", "summary": "This paper tackles few-shot image classification problem which has two main categories of solution, meta-learning based methods and global classification based methods. The authors provide analysis to show that the few-shot result for global classification based method is the result upper bound for meta-learning based method. Due to the unavailability of global labels in global classification in real applications, the paper proposes one algorithm to convert local labels to global labels for global classification. Results show the effectiveness of the proposed method on miniIMAGENET and tieredIMAGENET datasets.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper focuses on converting local labels to global labels, and then relies on global classification. The proposed Mela algorithm isolates the classification weights in meta-training and global classification, while reusing the weights as initialization weights for the other stage could potentially further improve the few-shot classification results. The training pipeline in Mela algorithm needs further examination.\n\nThe Algorithm 2 LearnLabeler, is essentially a clustering based approach with centroid assignment, update and pruning stages. Each stage has its defined metrics. Though this paper provides comparison with Self-Supervised methods (MoCo etc.) and  K-mean Clustering, as well as ablation studies for the effects of pruning threshold. However, the effects of some design details are still unclear. E.g. what if without the pruning stage? The effect of the metric selection (Equation 11 / Equation 12) in centroid assignment and update stages. How did you set the hyper-parameters in \"K-mean Clustering\" method?\n\n\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The proposed method for converting local labels to global labels is only limited to be used in few-shot learning task.\n\nThe few-shot training pipeline is not carefully examined, with disjoint two stages in terms of classification weights.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "Umy1bgLq7q9", "reviewer": "Reviewer_Cshp", "summary": "Stemming from recent observations that standard supervised training (called \"global classification\") empirically works better than local meta-learners, authors propose to infer labels from a trained meta-learner in order to fine-tune the feature-extractor. leveraging the benefits of global classification without requiring absolute labels.\n\nIn more details, authors first draw a connection between global classification risk and local classification risk (Proposition 1). This proposition essentially formalises the simple idea that solving a global classification problem (i.e over the full set of classes) is harder than solving a local one (i.e with a strictly smaller set of classes).\n\nSecond, authors propose an algorithm to infer global labels for each sample in a meta-training set by leveraging the features produced by a trained local meta-learner. The method relies on alternating between clustering extracted features and pruning the centroids with low number of assigned points. During the clustering step, local constraints force samples with the same local label to share the same global assignment. This assignment is decided by a majority vote (i.e the centroid that minimises the average squared distance to all samples with same local label). Once assignments have been made, centroids are updated following standard incremental rules.\n    Finally, authors compare their method on popular benchmarks mini-ImageNet and Tiered-Imagenet against recent meta-learners, and perform some ablation studies on the clustering hyper-parameters and quality.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## 1. Theoretical part\n\nOverall, I find the idea of linking global classification to local episode-wise classification interesting. Below some comments/issues on this part.\n\n### 1.1 Link global classification - Local task minimisation\n\nAuthors justify the success of global classification by saying it implicitly minimises a meta-learning loss (l 140-142). However, I do not see why minimising this meta-learning loss would actually be a good thing itself ? In fact, I would even reverse the argument, global classification works better precisely because it's not equivalent to minimising the local meta-learning loss. Solving global classification is simply a more challenging task, as captured by the very fact that the global loss is always higher than the meta-learning loss for the same model. In fact, it seems the discussion that follows (l. 148-160) goes along the lines of my latter argument, although borrowing different types of arguments (e.g conditional meta-learning).\n\n### 1.2 More \"concrete\" intuition\n\nI do appreciate the high level intuition of the discussion (l. 148-160), but I think a more \"lower level\" explanation could have been a real plus to making the argument more convincing. For instance, from an optimisation perspective, global classification produces a richer learning signal, i.e a given sample with label y has to be at the same time \"similar\" (e.g according to dot-product similarity) to the corresponding class' weight W[y] in the embedding space, while being dissimilar to all others classes' weights. On the other hand, solving local problems is easier and only pushes the sample away from a significantly lower number of class weights (e.g 4 other classes in the typical 5-way setting).  In other, words, the more classes the global problem has, the more challenging it becomes, the better the resulting feature extractor will be. On the other hand, local problems (at least as designed in common settings) always keep the same local classes, regardless of the global problem. Note that this \"metric learning\" interpretation would align with your observations that gains over local meta-learners are significantly higher on tiered-Imagenet with 351 classes than on mini-Imagenet with only 64 classes, while both are variants of the same ImageNet dataset.\n\n## 2. Proposed method:\n\nThe method proposed to infer global labels is relatively simple to understand and to implement. However, I have two concerns.\n\n### 2.1 Class imbalance\n\nThe first one is about the handling of class imbalance. From my understanding, the current method would tend to prune out clusters that do not contain enough samples (in comparison to others). Therefore, in the case of class imbalance, the (legitimate) small clusters would tend to disappear at the profit of prominent classes, which may hurt the performance. Unfortunately, both datasets authors experimented with are perfectly balance, hence such problem can hardly be visible. \n\n### 2.2 What if global labels are not well defined ?\n\nThe second is perhaps a deeper concern. I'm wondering how would such method perform when \"global\" labels do not actually exist, and instead only relative labels really make sense. For instance, think about a few-shot problem where a local class is simply formed of different augmentations of the same sample. Here, trying to find global labels would hardly make sense. Even without going into such extreme case, how would the method perform in the case where global labels are simply not clearly defined, e.g in the motivating example of dataset merging.\n\n\n## 3. Experiments\n\nExperiments are not convincing enough in their current form. See details in points below:\n\n### 3.1 Comparisons\n\nSelf-supervised learning methods are indeed a natural comparison and good initiative from the authors. However, I'm quite puzzled and surprised by the sentence \"A critical difference between MeLa and self-supervised methods is that they are not designed to leverage any information from the local labels\" (216-218). Indeed, those methods (at least CMC that I know more of) can quite trivially leverage the local labels (positive pairs = samples with same local labels / negative pairs = samples with different local labels). Therefore, that makes me suspicious whether the comparisons were fairly executed, and it's impossible to tell from the complete absence of experimental details on this.\n\nI acknowledge that Few-shot literature is expanding very fast and it becomes increasingly hard to keep up-to-date with latest method. However, 2020 state-of-the art for inductive methods is already significantly higher than what is reported in Table 1. One example would be [2], where authors report for Resnet-12: 65.9 and 82.4 for mini-ImageNet (~ +5 % w.r.t current paper), and 71.2 and 86.0 on tiered-Imagenet (~ +2 % w.r.t current paper). Now, it's hard to disentangle which part is due to data augmentation (which authors chose not use for a reason I simply do not understand l.227-228) and which part is due to the methods, but the related work probably needs some update.\n\n### 3.2 Narrowness of experiments\n\nChosen datasets: Experiments are carried out on two datasets (both variants of the same dataset) where global labels are well defined. As previously stated, when such global labels exist (even if not given to the learner), then the problem is indeed well defined. It would have been however more interesting to showcase the method in conditions where such are not that clearly defined anymore. The META-DATASET (cited by the authors) would have been ideal to demonstrate this. Obtaining better results than local meta-learners in that scenario would have been more convincing with respect to the initial motivation of the problem.\n\nAlso, experiments only carried out on a single network. Given current standard in few-shot classification, one would expect more than a single network to demonstrate the applicability of the method. Especially if the method introduces new hyperparameters such as the pruning threshold or initial number of clusters. Currently, it is hard to assess how easily such hyperparameters would transfer, and thus to assess how realistically applicable the method is\n\n### 3.3 Computational overhead\n\n The method proposed uses an additional training stage (once global labels have been infered) over the baseline meta-learner, which inevitably incurs a computational overhead that is never really discussed (Appendix C.4 gives training time, but those are not compared to the baseline). The additional cost may be small, but worth discussing in a few sentences.\n\n## Summary and score\n\nOverall I find this paper has merits in some insights it brings on global vs episodic risk minimisation.  However, I think that the experimental part is just not at the level of such conference. Without repeating the points made in 3.*, it overall fails to target scenarios of significant interest to the method (e.g where global labels are not available because they simply are not easy to define) while these could have been readily available by using META-DATASET. Hence, I cannot recommend acceptance of the paper in its current form.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See my review above for limitations.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "7u4qwWDxRuH", "reviewer": "Reviewer_yjsY", "summary": "This paper aims to reveal the effectiveness of using global labels in FSL from a theoretical perspective, and proposes a meta label learning (MeLa) method to infer the global labels by using a clustering algorithm. Overall, the investigated problem, i.e., whether to use global labels, is interesting and important. Also, the proposed theoretical analysis will be of interest to the community. However, the proposed clustering algorithm and the experimental part are not very surprising.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Strengths\n1. The investigated problem is interesting and important, which is of interest to the community. \n2. This paper is well written and well organized. \n3. The proposed theoretical analysis is important and inspiring.\n\n*Weaknesses\nI mainly have the following concerns about this paper:\n1. It seems that some latest works are not reviewed and compared in this paper, e.g., FEAT [1] and DC [2]. In addition, some recent FSL works have paid attention to local representations, e.g., DN4 [3] and DeepEMD [4]. Are the theoretical analysis and conclusions still valid for these methods?\n2. Why the data augmentation is avoided in the experiments? Since the experiments are mainly conducted on the image datasets, the reason of discarding data augmentation is not very clear.\n3. How to construct the meta-training set?  It seems that the meta-training set is fixed in the paper. Do you randomly sample a series of meta-training tasks from the auxiliary set of the miniImageNet in advance and then fix it?\n4. It will be interesting to explain whether the proposed theoretical analysis still works on the domain-shifted settings. As we know, the meta-training set has a somewhat different label space (even a different domain) with the meta-test set. \n\n[1] Few-Shot Learning via Embedding Adaptation with Set-to-Set Functions. CVPR 2020.\n\n[2] Free lunch for Few-shot Learning: Distribution Calibration. ICLR 2021.\n\n[3] Revisiting Local Descriptor based Image-to-Class Measure for Few-shot Learning. CVPR 2019.\n\n[4] DeepEMD: Differentiable Earth Mover's Distance for Few-Shot Learning. CVPR 2020.\n\n\n================\n\nThanks for the authors' response. I would like to maintain my original rating of 6, and recommend the authors to further enhance the experiments according to the comments of all reviewers.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The main issues amongst reviewers were 1. A lack of compelling scenarios without global labels, 2. Experiments are only on small benchmarks (in today’s environment) in which global labels are all well-defined. The main suggestion was to apply the technique to the meta-dataset benchmark, and the authors have provided some initial results that are indeed promising, however very preliminary. Other issues like performance with data augmentation, class imbalance (robustness in general) have been satisfactorily addressed in the discussion period.

The reviewers tend to agree that a more in-depth look at the case with no global labels is a topic for future research. Even still, there is quite a bit of work to be done to incorporate the new results/feedback from the discussion period, and the reviewers remain neutral-to-positive. I do think that the hypothesis and associated performance is compelling enough to be of interest to the community. As a last suggestion, I think it’s worth briefly citing and discussing the paper “Unsupervised Learning Via Meta-Learning” by Hsu et al., ICLR 2019. The approaches are somewhat different, but related enough to be worth mentioning.
Decision: Accept (Poster)