The Role of Global Labels in Few-Shot Classification and How to Infer Them
Year: 2022

Abstract:
Few-shot learning is a central problem in meta-learning, where learners must quickly adapt to new tasks given limited training data. Recently, feature pretraining has become a ubiquitous component in state-of-the-art meta-learning methods and is shown to provide significant performance improvement. However, there is limited theoretical understanding of the connection between pre-training and meta-learning. Further, pre-training requires global labels shared across tasks, which may be unavailable in practice. In this paper, we show why exploiting pre-training is theoretically advantageous for meta-learning, and in particular the critical role of global labels. This motivates us to propose Meta Label Learning (MeLa), a novel meta-learning framework that automatically infers global labels to obtains robust few-shot models. Empirically, we demonstrate that MeLa is competitive with existing methods and provide extensive ablation experiments to highlight its key properties.

heading: 1 Introduction
text: A central problem in meta-learning is few-shot learning (FSL), where new tasks must be learned quickly given limited amount of training data. FSL has drawn increasing attention recently due to the high cost of collecting and annotating large datasets. To tackle the challenge of model generalization in FSL, meta-learning leverages past experiences of solving related tasks by directly learning transferable knowledge over a collection of FSL tasks. A diverse range of meta-learning methods tailored for FSL have been proposed, including optimization-based [e.g. 1, 5, 27], metric learning [e.g. 20, 21, 25], and model-based methods [e.g. 7, 14, 19]. The diversity of the existing strategies raises a natural question: do these methods share any common lessons for improving model generalization and for designing future methods?
Several papers addressed the above question. Chen et al. [2] identified that data augmentation and deeper network architecture significantly improve generalization performance across several meta-learning methods. On the other hand, Tian et al. [23] investigated a simple yet competitive approach: a linear model on top of input embeddings learned via feature pre-training. This approach ignores task structures from meta-learning and merges all tasks into a “flat” dataset of labeled samples. The desired input embedding is then learned by classifying all classes of the flat dataset.
Extensive empirical evidences supports the efficacy of feature pre-training in FSL. Pre-training alone already outperforms various meta-learning algorithms [23]. Recently, it is used as a ubiquitous
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
pre-processing step in state-of-the-art meta-learning methods [e.g. 18, 26, 27]. In particular, [27] reported that pre-training also significantly boosted earlier methods (see Fig. 2).
Despite the significant empirical improvement, there is limited theoretical understanding of why feature pre-training provides significant performance gain for meta-learning. On the other hand, pretraining requires task merging to construct the flat dataset, which implicitly assumes access to some notion of global labels consistently adopted across tasks. However, global labels may not exist or are inaccessible, such as when each task is annotated independently with only local labels. This renders direct task merging and consequently pre-training impossible (see Fig. 1a). Independent annotation captures realistic scenarios when tasks are collected organically (e.g from different users), rather than generated synthetically from benchmark datasets (e.g. miniIMAGENET). Possible scenarios include non-descript task labels (e.g. tasks with numerical labels) or even concept overlaps among labels across tasks (e.g. sea animals vs mammals).
In this paper, we address the issues raised above with the following contributions:
• In Sec. 3, we show that feature pre-training directly relates to meta-learning as a loss upper bound. In particular, pre-training induces a conditional meta-learning formulation [3, 26], which provides a principled explanation for the empirical performance gains.
• In Sec. 4, we propose Meta Label Learning (MeLa), a novel framework that automatically infers some notion of latent global labels consistent with local task constraints. The inferred labels enable us to leverage pre-training to improve meta-learners’ generalization performance, and bridges the gap between experiment settings with or without acesss to global labels.
• In Sec. 5, we demonstrate empirically the competitive performance of MeLa over a suite of benchmark datasets. We also present ablation studies to highlight its key properties.
In Sec. 2, we first review the key notions of meta-learning and FSL. The supplementary material contains proofs for the theoretical analysis, additional empirical results, and experiment setup.

heading: 2 Background
text: We formalize FSL in the context of meta-learning, followed by reviewing feature pre-training initially investigated in [23].
Meta-learning. FSL [4] considers a meta-training set of tasks T = {(St, Qt)}Tt=1, with support set St = {(xj , yj)}nsj=1 and query setQt = {(xj , yj)} nq j=1 sampled from the same distribution. Typically, St and Qt each contains a small number of samples ns and nq respectively (fixed for all tasks for simplicity). We denote by D the space of datasets of the form St or Qt. FSL aims to find the best inner learning algorithm (referred to as base learner in the following) Alg(θ, ·) : D → F that maps supports sets S to predictors f = Alg(θ, S), such that x 7→ f(x) generalizes well on the corresponding query sets. The base learner is meta-parametrized by θ ∈ Θ.
Formally, the meta-learning problem for FSL is
min θ∈Θ
E(S,Q)∈T L ( Alg(θ, S), Q ) , (1)
where E(S,Q)∈T , 1|T | ∑
(S,Q)∈T denotes the empirical distribution over the meta-training set. The task loss L : F ×D → R is the empirical risk of the learner over query sets, according to an inner loss ` : Y × Y → R, where Y is the space of labels
L(f,D) = E(x,y)∈D [`(f(x), y)]. (2)
Designing effective base learners is a key focus in meta-learning literature and various strategies have been explored. We focus on a broad class of methods that we call meta-representation learning [1, 6, 10, 15], which is remarkably effective in practice and closely related to feature pre-training. Meta-representation learning infers a suitable embedding for base learners by solving
min θ∈Θ
E(S,Q)∈T [L(w(ψθ(S)), ψθ(Q)] (3)
where ψθ : X → Rm is a feature embedding model and ψθ(D) , {(ψθ(x), y)|(x, y) ∈ D} the embedded dataset. In [1], the ridge regression estimator is chosen as the base learner Alg(θ,D) = w(ψθ(D))
w(ψθ(D)) = argmin W
E(x,y)∈ψθ(D) ‖Wx− OneHot(y)‖ 2 + λ1 ‖W‖2 , (4)
where λ1 is a constant and OneHot(y) denotes the one-hot encoding of the label y. Among different base learner designs, ridge regression is often favored for its differentiable closed-form solution and the associated computational efficiency in optimizing (3).
Feature Pre-training. Feature pre-training has been widely used in meta-learning. It was investigated in-depth in [23]. Given the meta-training set T , a “flat” dataset Dglobal is constructed by merging all tasks in T :
Dglobal = D(T ) = {(xi, yi)}Ni=1 = ⋃
(S,Q)∈T
(S ∪Q). (5)
We then learn a embedding function ψθ on Dglobal using the standard cross-entropy loss `ce for multi-class classification:
argmin θ,W E(x,y)∈Dglobal [`ce(Wψθ(x), y)]. (6)
After obtaining ψθ, a novel task may be solved in the embedded space by a base learner. In [23], the authors recommended the regularized logistic regression estimator
wce(D) = argmin W
Lce(W,D) + λ2 ‖W‖2 , (7)
for its empirical performance. Here λ2 is a regularization constant.
[23] demonstrated that feature pre-training as a standalone method already outperforms many sophisticated meta-learning strategies. It has also become a ubiquitous component in most state-of-the-art methods [27, 28, 29]. Furthermore, [27] demonstrated empirically that pre-training provides similar performance gain to earlier methods, making them (mostly) competitive with the state of the art. In Sec. 3, we show that feature pre-training directly relates to meta-learning as a loss upper bound, and explains why pre-training contributes to improved performance.

heading: 3 Feature Pre-training as Meta-learning
text: In this section, we show how feature pre-training relates to meta-learning as a loss upper bound. More precisely, we show that the pre-training induces a special base learner with the corresponding meta-learning loss upper bounded by the cross-entropy loss (Eq. (6)). Consequently, minimizing the cross-entropy loss also indirectly solves the induced meta-learning problem. Further, we observe that the special base learner is a conditional meta-learning formulation, which provides a principled explanation for the improved performance.
Let T = {(St, Qt)}Tt=1 be a meta-training set. We denote the collection of query sets as Q = {Qt}Tt=1 = {(Xt, Yt)}Tt=1 where we writeQt = (Xt, Yt) as a tuple of input samplesXt = {xjt} nq j=1 and their corresponding labels Yt = {yjt} nq j=1. For simplicity, we assume that the query sets are disjoint, namely Qt ∩ Qt′ = ∅ for any t 6= t′. We merge all query sets into a flat dataset D(Q) = {(xi, yi)}Ni=1 = ∪Tt=1Qt, with N = nqT . Proposition 1. With the notation and assumptions introduced above, let C be the total number of classes in D(Q), and W ∈ RC×m the global classifier. Denote by W [Y ] the sub-matrix with rows indexed by the sorted unique values1 from Y . Then, for any embedding ψθ : X → Rm
E(X,Y )∈Q [ Lce ( W [Y ], (ψθ(X), Y ) )] ≤ E(x,y)∈D(Q) [`ce(Wψθ(x), y)] . (8)
We outline the proof strategy here and defer the details to the supplementary material. We first observe that D(Q) has the same collection of samples as Q. Secondly, each matrix W [Y ] is in fact a task classifier for the query set (X,Y ). Crucially, the likelihood of query samples p(Y |X), estimated by task classifiers W [Y ], are no smaller than their likelihood estimated by the global classifier W , which forms the inequality in (8). Combining the two observations yields Prop. 1.
The key implication of Prop. 1 is that, if global labels are available, we can use a special base learner
wglobal(D) = wglobal(X,Y ) = W [Y ], (9)
for any dataset of the form D = (X,Y ). This base learner only depends on the intended classes (i.e. unique values in Y ) to produce task classifiers and is independent of specific inputs from D (illustrated in Fig. 1b). We note that for any task (S,Q) ∈ T , wglobal(S) = wglobal(Q), since the support and query sets share the same class labels. These observations directly imply the following,
Lemma 2. For meta-training set T where tasks are annotated with global labels, we have E(S,Q)∈T [ Lce ( wglobal(S), ψθ(Q) )] = E(X,Y )∈Q [ Lce ( W [Y ], (ψθ(X), Y ) )] . (10)
The left-hand side of (10) is thus the meta-learning loss associated with base learner wglobal(·). Hence, the combination of Prop. 1 and Lemma 2 implies that the empirical risk associated with this meta-learning loss is upper bounded by the standard cross-entropy loss, namely the right-hand side of (8). The connection shows that feature pre-training also solves a meta-learning problem when we design the base learner to be wglobal(·). Remark 1. The bound in Prop. 1 is tight when the standard cross-entropy loss is 0.
Remark 1 shows that all task classifiers W [Y ] and the embedding function ψθ are optimal, when standard cross-entropy is 0. In practice, Remark 1 is achievable since over-parametrized neural networks could obtain zero empirical loss.
We highlight two important properties for the base learner wglobal(·). Firstly, wglobal is not the base learner used during meta-testing since it cannot generalize to novel classes. In [23], wglobal was simply replaced with Eq. (7) during meta-testing, while other works choose to fine-tune the pre-trained embedding model ψθ with a new base learner intended for meta-testing using episodic training. The advantage of the fine-tuning strategy is clear since it optimizes ψθ for the actual base learner used during test time. This observation is well supported by existing empirical results, with many state-of-the-art methods [e.g. 28, 29] adopting the fine-tuning strategy and surpassing standalone pre-training.
Secondly, we observe that wglobal(·) describes a conditional meta-learning problem: the global labels Y provide additional side information to facilitating model learning. Specifically, global labels directly reveal how task samples relate to one another and wglobal could simply map global labels to task classifiers via W [Y ]. In contrast, unconditional base learners (e.g. (4) and (7)) have to learn classifiers based on support sets, without access to task relations provided by global labels.
Global labels offer significant advantages to learning the embedding model ψθ. Denevi et al. [3] proved that conditional meta-learning is advantageous over unconditional formulation by incurring a smaller excess risk, especially when the meta-distribution of tasks is organized into distant clusters
1E.g. Y = {6, 6, 4, 9} maps to {4, 6, 9}-th rows of W . Also see Fig. 1b for an illustration.
(see [3] for further discussion). In practice, global labels cluster task samples for free and improve regularization by enforcing each cluster (denoted by global label y) to share vector W [y] for all task classifiers. The above analysis explains why pre-training yields significantly more robust ψθ than many (unconditional) meta-learning methods.
Algorithm 1: MeLa
Input: meta-training set T = {Si, Qi}Ti=1 ψ0θ = argminψθ E(S,Q)∈T [L(w(ψθ(S), ψθ(Q))]) Global clusters G = LearnLabeler(ψ0θ , T ) ψ∗θ = MetaLearn(G, T ) Return ψ∗θ
Figure 2: Effect of Pre-training on miniIMAGENET
In Fig. 2, we plot several existing results from [2, 27] to highlight the contribution of the pretraining. For miniIMAGENET, pre-training accounted for on average 9% improvement in ProtoNet and MatchNet. In addition, pre-training alone yields a performance mostly competitive with both ProtoNet and MatchNet, suggesting that the pre-training contributes far more towards generalization performance compared to the specific meta-learning strategies deployed. Similar trends are observed for other methods and datasets, both in the previous works and our experiments. The extensive empirical results are consistent with the theoretical advantages of conditional meta-learning.

heading: 4 Meta Label Learning
text: In Sec. 3, we analyzed the theoretical advantages of feature pre-training in connection to metalearning, as well as the critical role of global labels in learning robust embedding model. However, we argue that leveraging global labels are problematic for meta-learning. Firstly, global labels may be unavailable or inaccessible in practical applications, when meta-training tasks are collected and annotated independently as discussed in Sec. 1, rendering pre-training inapplicable. Secondly, global labels oversimplify meta-learning: they directly reveal how input samples relate to one another across tasks, while the goal of meta-learning is precisely learning such cross-task relations and extracting transferable knowledge. As highlighted in Sec. 3, pre-training contributes more towards test performance than the meta-learning strategies employed, making it difficult to assess the relative merits of different strategies.
To address these issues, we present a novel meta-learning framework that does not require access to global labels. In particular, our framework automatically infers some notion of latent global labels across tasks, therefore bridging the experiment settings with and without global labels.
Alg. 1 outlines our proposed strategy. We first meta-learn an embedding function ψ0θ as a tool to measure sample similarity. Secondly, we introduce a labeling algorithm for clustering task samples while enforcing local task constraints in the training data. The resulting clusters are used as inferred global labels. Lastly, we may apply any existing meta-learning strategy capable of leveraging global labels to obtain the final model2.
We focus on the labeling algorithm since other components of MeLa are standard procedures. The labeling algorithm takes a meta-training set as input and outputs a set of clusters to represent global labels. The algorithm consists of a clustering step for updating centroids and a pruning step for merging small clusters. The algorithm is presented in Alg. 2.
Clustering Step. The procedure exploits local labels from each task to guide sample assignments. For each task, local labels are used to enforce two constraints: samples with the same local label should be assigned the same global label, while samples from different local classes should be assigned different global ones. Formally, given a set of cluster centroids G = {gj}Jj=1, we assign all
2We note that [9] is a loosely similar strategy: they use a fully unsupervised feature representation for clustering to infer auxiliary labels, which are then used to refine the representation.
Algorithm 2 LearnLabeler
Input: embedding model ψ0θ , meta-training set T = {St, Qt}Tt=1, number of classes in a task K Initialization: sample tasks from T to initialize clusters G = {gj}Jj=1,
While |G| has not converged: Nv = 1 for each gv ∈ G For (S,Q) ∈ T :
Match global clusters V = {vq}Kq=1 via (11) for task (X,Y ) = S ∪Q If V has K unique clusters
Update cluster v for each v ∈ V via (12) G← {gv|gv ∈ G,Nv ≥ threshold in(13)}
Return G
samples {xi}Ii=1 sharing the same local label within a task to a single global cluster as follows,
v∗ = argmin v ∥∥∥∥∥1I I∑ i=1 ψ0θ(xi)− gv ∥∥∥∥∥ 2 . (11)
We apply (11) to each class of samples in a task, matching K clusters in total. For simplicity, we discard tasks in which multiple local classes map to the same cluster. Otherwise, we update each matched cluster gv∗ and sample counts Nv∗ with
gv∗ = Nv∗gv∗ +
∑I i=1(ψ 0 θ(xi))
Nv∗ + I and Nv∗ = Nv∗ + I, (12)
Pruning Step. We present a simple pruning strategy to regulate the number of clusters. Under the mild assumption that each cluster is equally likely to appear in a task, a cluster v is sampled with probability p = KJ for each K-way classification task. The number of samples Nv assigned to cluster v thus follows a binomial distribution with Nv ∝ B(T, p) where we recall T as the size of the meta-training set. We remove any cluster below the threshold
Nv < N̄v − q √ Var(Nv) (13)
where N̄v is the mean of Nv, Var(Nv) the variance, and q a hyper-parameter controlling the the aggressiveness of the pruning process.
Alg. 2 initializes a large number of clusters and populates the centroids with mean class embeddings computed from random tasks in T . For J initial clusters, d JK e tasks are needed since each task contributes K embeddings. The algorithm alternates between clustering and pruning to refine the centroids and estimate the number of clusters. When the number of existing clusters no longer changes, the algorithm terminates and returns the current centroids G. Given the clusters G, all samples from the meta-training set can be assigned global labels.
We comment on two important points about Alg. 2. Firstly, the two hyperparameters, initial cluster count and pruning threshold, are only necessary when global labels are unavailable, since they determine the appropriate number of clusters. In contrast, access to global labels implies that the number of clusters and even the number of samples for each cluster is known. This further shows how global labels could oversimplify meta-learning as discussed earlier. Secondly, Alg. 2 differs from the classical K-mean algorithm [11] by exploiting local information to guide the clustering process, while K-mean algorithm is fully unsupervised. We will show empirically that enforcing local constraints is necessary for learning robust models.
Meta-Learning with Inferred Labels. After obtaining the inferred labels, we may apply a wide range of meta-learning algorithms (such as [27, 28] that exploit global labels) to obtain the final model. To highlight the efficacy of pre-training and the robustness of the proposed labeling algorithm, we choose [23] without additional fine-tuning in this work.

heading: 5 Experiments
text: We evaluate our proposed method on several benchmark datasets, including ImageNet variants, CIFAR variants, and a subset of MetaDataset [24]. As discussed earlier, we adopt independent
annotation with local labels only for all experiments. Model details and hyperparameter values are included in the supplementary material. Due to space constraint, CIFAR experiments and some ablation studies are also presented in the supplementary material.
The experiments aim to address the following questions: 1) How does MeLa compare to existing algorithms? 2) How does pre-training affect the performance of meta-learning algorithms? 3) Does MeLa learn meaningful clusters?
Experiments on ImageNet Variants. We compare MeLa to a representative set of meta-learning algorithms on miniIMAGENET [25] and tieredIMAGENET [17]. For completeness, we include methods requiring access to global labels. However, we emphasize that these methods are not directly comparable to MeLa, since access to global labels provide significantly more information to metalearners as discussed previously. These methods are intended to demonstrate the effect of pre-training on generalization performance. We also include self-supervised methods in the comparison.
Similar to Fig. 2, Tab. 1 clearly shows the significant advantages of having access to global labels: all methods exploiting pre-training achieves noticeably higher generalization performance compared to methods without global labels. The results are consistent with our theoretical analysis that conditional meta-learning is more advantageous. Further, we observe that global labels not only enable pretraining but also flexible task sampling, including practical heuristics such as sampling more shots and more classes per task during meta-training [10, 27, 28]. All of the above contribute to generalization performance, and it is clear that global labels should be used when available.
In our experiment setting of “local labels” only, MeLa outperforms all baselines in three out of four settings, and obtains performance comparable to [10] in the remaining one. We highlight the comparison between the initial embedding ψ0θ obtained via (3) and the final embedding ψ ∗ θ obtained by MeLa as they share identical experimental setups. It is thus easy to attribute the performance improvements to the proposed algorithm and the effect of pre-training. In particular, MeLa improves the average test performance by over 2% in miniIMAGENET and over 4% in tieredIMAGENET. In addition, we observe that MeLa obtains performance comparable to RFS [23], which is the oracle setting (i.e. access to ground truth global labels) for MeLa.
While FEAT and FRN outperforms MeLa in Tab. 1, we reiterate that methods exploiting global labels are not directly comparable, as discussed earlier. In addition, the performance of MeLa could be readily improved via more sophisticated data augmentation or fine-tuning. We explore one such variant for MeLa in Appendix B.2.
In [23], the authors also adapted two state-of-the-art self-supervised methods MoCo [8] and CMC [22] for FSL. In the adapted methods, labels are not explicitly provided as input, but used to construct contrasting samples for learning self-supervised embedding. MeLa also outperforms them in Tab. 1.
Robustness of the Labeling Algorithm. A potentially trivial solution for clustering samples without any learning is by looking for identical samples: an identical sample appearing in multiple tasks would allow several local classes to be assigned to the same cluster. To demonstrate that Alg. 2 is not trivially matching identical samples across task, we introduce a more challenging experiment setting: each sample only appears once in the meta-training set. This implementation constructs the meta-training set by sampling from a flat dataset without replacement3. Consequently, Alg. 2 must only rely on the initial embedding function ψ0θ for estimating sample similarity. We evaluate MeLa under this setting and report the results in Tab. 2.
In Tab. 2, clustering accuracy is computed by assigning the most frequent ground truth label in each cluster as the desired target. In addition, percentage of tasks clustered refers to the tasks that map to K unique clusters by Alg. 2. The clustered tasks satisfy both constraints imposed by local labels and are used for pre-training.
The results suggest that Alg. 2 is robust in inferring accurate global labels, even when samples do not repeat across tasks. The no-replacement setting also has negligible impact on test performance. In particular, we note that miniIMAGENET is particularly challenging under the new setting, with only 384 tasks in the meta-training set. In contrast, the typical sample-with-replacement setting has access to unlimited number of tasks for training.
The high clustering accuracy implies that the meta-distribution underlying the meta-training set is near perfectly recovered. MeLa thus effectively bridge the gap between experiment settings with and without access to global labels respectively. Using the inferred labels, we may apply a wide range of meta-learning algorithms to obtain the final model. We may also adopt flexible task sampling, such as sampling more shots and classes per task [10, 27, 28], for better generalization performance.
The Importance of Local Constraints. The clustering process enforces consistent assignment of task samples given their local labels. To understand the importance of enforcing these constraints, we consider an ablation study where Alg. 2 is replaced with standard K-mean algorithm, while other components of MeLa remain unchanged. K-mean algorithm is fully unsupervised and ignores any local constraints. We initialize the K-mean algorithm with 64 clusters for miniIMAGENET and 351 clusters for tieredIMAGENET, the actual numbers of classes in respective datasets.
Tab. 3 indicates that enforcing local constraints is critical in accurately inferring the global labels, as measured by clustering accuracy. In addition, lower clustering accuracy directly translates to lower test accuracy during meta-testing, suggesting that sensible task merging is an important prerequisite for feature pre-training. In particular, test accuracy drops by over 5% for tieredIMAGENET, when K-mean algorithm ignores local task constraints.
3For instance, miniIMAGENET (38400 training samples) can be split into 384 tasks of 100 samples in this setting.
Between the two local constraints, we note that (11) is more important for accurately inferring global labels. Specifically, the clustering step improves accuracy by averaging the votes from all samples sharing the same local label in a task. On the other hand, the constraint on matching K unique clusters are satisfied by almost all tasks empirically (see Tab. 2).
Experiment on MetaDataset. We further evaluate MeLa on MetaDataset [24], a collection of independently annotated datasets designed for meta-learning. MetaDataset presents a more challenging setting by including more diverse samples from different image domains.
We choose Aircraft, CUB and VGG flower datasets from MetaDataset for the experiment. The chosen datasets are all intended for fine-grained classification in aircraft models, bird species and flower species respectively. We compare MeLa against FEAT and FRN, two state-of-the-art methods. Since only local labels are used, all models are trained without pre-training. For meta-training, all models are trained on tasks sampled from the three datasets. For meta-testing, we sample 1500 tasks from each dataset and report the average accuracy. Test accuracy on individual datasets are included in Appendix B.4.
Tab. 4 shows that MeLa outperforms both FEAT and FRN when pre-training becomes inapplicable in the “local label” setting. In particular, FEAT performed relatively poorly since it is not designed to train from random initialization. In addition, MeLa improves upon the initial embedding by over 4%, similar to the performance gain observed in tieredIMAGENET. The results further validate the efficacy of pre-training even when the labels are inferred, consistent with our theoretical analysis. Lastly, we report that the clustering accuracy in this experiment is 79.3%, with a 94.7 percent of the tasks clustered. The results suggest that MeLa’s performance is robust to noise in the inferred labels.

heading: 6 Discussion
text: In this paper, we studied the effect of pre-training and the critical role of global labels in meta-learning. We showed that pre-training closely relates to meta-learning as a loss upper bound, and induces a conditional meta-learning formulation that explains the improved empirical performance. The effect of pre-training is consistently demonstrated in existing results and our experiments. The connection between meta-learning and pre-training opens up new opportunities of directly applying existing techniques from supervised learning towards meta-learning. For instance, model distillation [23] has been shown to further improve pre-training performance.
We also proposed a practical framework to infer global labels, for settings when they are unavailable. We demonstrate that MeLa is robust and accurate in inferring global labels, and achieves generalization performance competitive with state-of-the-art methods. In the ablation studies, we observed that meta-learning methods implicitly learn to cluster samples across tasks, even when samples do not repeat. In addition, explicitly enforcing the local constraints is critical for accurately inferring global labels and learning robust few-shot models.
Limitations and Future Work. We close by discussing some limitations of our work and directions of future research. In this paper, we focused on understanding the connection between pre-training and meta-learning, and evaluated MeLa on benchmarks with globally disjoint classes. In the future, we intend to extend our method to settings where global labels are ill-defined, such as when classes are overlapping or hierarchical. One possible approach is to assign soft labels or multiple labels to samples, thus capturing more complex relationship between classes.
Broader Impact
Meta-learning aims to construct learning models capable of learning from experiences, Its intended users are thus primarily non-experts who require automated machine learning services, which may occur in a wide range of potential applications such as recommender systems and autoML. The authors do not expect the work to address or introduce any societal or ethical issues.
Acknowledgments and Disclosure of Funding
The authors would like to thank the anonymous reviewers for their comments and suggestions. This work was supported in part by Career Development Fund (grant C210812045) from A*STAR Singapore, SAP SE, Royal Society (grant SPREM RGS\R1\201149) and Amazon Research Award (ARA).

Review summaries:
This paper tackles few-shot image classification problem which has two main categories of solution, meta-learning based methods and global classification based methods. The authors provide analysis to show that the few-shot result for global classification based method is the result upper bound for meta-learning based method. Due to the unavailability of global labels in global classification in real applications, the paper proposes one algorithm to convert local labels to global labels for global classification. Results show the effectiveness of the proposed method on miniIMAGENET and tieredIMAGENET datasets.

Stemming from recent observations that standard supervised training (called "global classification") empirically works better than local meta-learners, authors propose to infer labels from a trained meta-learner in order to fine-tune the feature-extractor. leveraging the benefits of global classification without requiring absolute labels.  In more details, authors first draw a connection between global classification risk and local classification risk (Proposition 1). This proposition essentially formalises the simple idea that solving a global classification problem (i.e over the full set of classes) is harder than solving a local one (i.e with a strictly smaller set of classes).  Second, authors propose an algorithm to infer global labels for each sample in a meta-training set by leveraging the features produced by a trained local meta-learner. The method relies on alternating between clustering extracted features and pruning the centroids with low number of assigned points. During the clustering step, local constraints force samples with the same local label to share the same global assignment. This assignment is decided by a majority vote (i.e the centroid that minimises the average squared distance to all samples with same local label). Once assignments have been made, centroids are updated following standard incremental rules.     Finally, authors compare their method on popular benchmarks mini-ImageNet and Tiered-Imagenet against recent meta-learners, and perform some ablation studies on the clustering hyper-parameters and quality. 

This paper aims to reveal the effectiveness of using global labels in FSL from a theoretical perspective, and proposes a meta label learning (MeLa) method to infer the global labels by using a clustering algorithm. Overall, the investigated problem, i.e., whether to use global labels, is interesting and important. Also, the proposed theoretical analysis will be of interest to the community. However, the proposed clustering algorithm and the experimental part are not very surprising.

Authors analyze the role of global pretraining in few-shot classification, establishing classification loss over all classes as an upper bound on the classification losses from particular tasks. Authors then consider the case where task-based labels are available but are not easily collated into a global set of labels. They introduce a task-aware clustering-based method for assigning global labels to embeddings from a task-pretrained feature extractor, allowing them to achieve near-oracle performance in settings where global labels are absent. The method improves upon task-based pretraining alone by a substantial margin.