The Role of Global Labels in Few-Shot Classification and How to Infer Them

1 Introduction . A central problem in meta-learning is few-shot learning ( FSL ) , where new tasks must be learned quickly given limited amount of training data . FSL has drawn increasing attention recently due to the high cost of collecting and annotating large datasets . To tackle the challenge of model generalization in FSL , meta-learning leverages past experiences of solving related tasks by directly learning transferable knowledge over a collection of FSL tasks . A diverse range of meta-learning methods tailored for FSL have been proposed , including optimization-based [ e.g . 1 , 5 , 27 ] , metric learning [ e.g . 20 , 21 , 25 ] , and model-based methods [ e.g . 7 , 14 , 19 ] . The diversity of the existing strategies raises a natural question : do these methods share any common lessons for improving model generalization and for designing future methods ? Several papers addressed the above question . Chen et al . [ 2 ] identified that data augmentation and deeper network architecture significantly improve generalization performance across several meta-learning methods . On the other hand , Tian et al . [ 23 ] investigated a simple yet competitive approach : a linear model on top of input embeddings learned via feature pre-training . This approach ignores task structures from meta-learning and merges all tasks into a “ flat ” dataset of labeled samples . The desired input embedding is then learned by classifying all classes of the flat dataset . Extensive empirical evidences supports the efficacy of feature pre-training in FSL . Pre-training alone already outperforms various meta-learning algorithms [ 23 ] . Recently , it is used as a ubiquitous 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . pre-processing step in state-of-the-art meta-learning methods [ e.g . 18 , 26 , 27 ] . In particular , [ 27 ] reported that pre-training also significantly boosted earlier methods ( see Fig . 2 ) . Despite the significant empirical improvement , there is limited theoretical understanding of why feature pre-training provides significant performance gain for meta-learning . On the other hand , pretraining requires task merging to construct the flat dataset , which implicitly assumes access to some notion of global labels consistently adopted across tasks . However , global labels may not exist or are inaccessible , such as when each task is annotated independently with only local labels . This renders direct task merging and consequently pre-training impossible ( see Fig . 1a ) . Independent annotation captures realistic scenarios when tasks are collected organically ( e.g from different users ) , rather than generated synthetically from benchmark datasets ( e.g . miniIMAGENET ) . Possible scenarios include non-descript task labels ( e.g . tasks with numerical labels ) or even concept overlaps among labels across tasks ( e.g . sea animals vs mammals ) . In this paper , we address the issues raised above with the following contributions : • In Sec . 3 , we show that feature pre-training directly relates to meta-learning as a loss upper bound . In particular , pre-training induces a conditional meta-learning formulation [ 3 , 26 ] , which provides a principled explanation for the empirical performance gains . • In Sec . 4 , we propose Meta Label Learning ( MeLa ) , a novel framework that automatically infers some notion of latent global labels consistent with local task constraints . The inferred labels enable us to leverage pre-training to improve meta-learners ’ generalization performance , and bridges the gap between experiment settings with or without acesss to global labels . • In Sec . 5 , we demonstrate empirically the competitive performance of MeLa over a suite of benchmark datasets . We also present ablation studies to highlight its key properties . In Sec . 2 , we first review the key notions of meta-learning and FSL . The supplementary material contains proofs for the theoretical analysis , additional empirical results , and experiment setup . 2 Background . We formalize FSL in the context of meta-learning , followed by reviewing feature pre-training initially investigated in [ 23 ] . Meta-learning . FSL [ 4 ] considers a meta-training set of tasks T = { ( St , Qt ) } Tt=1 , with support set St = { ( xj , yj ) } nsj=1 and query setQt = { ( xj , yj ) } nq j=1 sampled from the same distribution . Typically , St and Qt each contains a small number of samples ns and nq respectively ( fixed for all tasks for simplicity ) . We denote by D the space of datasets of the form St or Qt . FSL aims to find the best inner learning algorithm ( referred to as base learner in the following ) Alg ( θ , · ) : D → F that maps supports sets S to predictors f = Alg ( θ , S ) , such that x 7→ f ( x ) generalizes well on the corresponding query sets . The base learner is meta-parametrized by θ ∈ Θ . Formally , the meta-learning problem for FSL is min θ∈Θ E ( S , Q ) ∈T L ( Alg ( θ , S ) , Q ) , ( 1 ) where E ( S , Q ) ∈T , 1|T | ∑ ( S , Q ) ∈T denotes the empirical distribution over the meta-training set . The task loss L : F ×D → R is the empirical risk of the learner over query sets , according to an inner loss ` : Y × Y → R , where Y is the space of labels L ( f , D ) = E ( x , y ) ∈D [ ` ( f ( x ) , y ) ] . ( 2 ) Designing effective base learners is a key focus in meta-learning literature and various strategies have been explored . We focus on a broad class of methods that we call meta-representation learning [ 1 , 6 , 10 , 15 ] , which is remarkably effective in practice and closely related to feature pre-training . Meta-representation learning infers a suitable embedding for base learners by solving min θ∈Θ E ( S , Q ) ∈T [ L ( w ( ψθ ( S ) ) , ψθ ( Q ) ] ( 3 ) where ψθ : X → Rm is a feature embedding model and ψθ ( D ) , { ( ψθ ( x ) , y ) | ( x , y ) ∈ D } the embedded dataset . In [ 1 ] , the ridge regression estimator is chosen as the base learner Alg ( θ , D ) = w ( ψθ ( D ) ) w ( ψθ ( D ) ) = argmin W E ( x , y ) ∈ψθ ( D ) ‖Wx− OneHot ( y ) ‖ 2 + λ1 ‖W‖2 , ( 4 ) where λ1 is a constant and OneHot ( y ) denotes the one-hot encoding of the label y . Among different base learner designs , ridge regression is often favored for its differentiable closed-form solution and the associated computational efficiency in optimizing ( 3 ) . Feature Pre-training . Feature pre-training has been widely used in meta-learning . It was investigated in-depth in [ 23 ] . Given the meta-training set T , a “ flat ” dataset Dglobal is constructed by merging all tasks in T : Dglobal = D ( T ) = { ( xi , yi ) } Ni=1 = ⋃ ( S , Q ) ∈T ( S ∪Q ) . ( 5 ) We then learn a embedding function ψθ on Dglobal using the standard cross-entropy loss ` ce for multi-class classification : argmin θ , W E ( x , y ) ∈Dglobal [ ` ce ( Wψθ ( x ) , y ) ] . ( 6 ) After obtaining ψθ , a novel task may be solved in the embedded space by a base learner . In [ 23 ] , the authors recommended the regularized logistic regression estimator wce ( D ) = argmin W Lce ( W , D ) + λ2 ‖W‖2 , ( 7 ) for its empirical performance . Here λ2 is a regularization constant . [ 23 ] demonstrated that feature pre-training as a standalone method already outperforms many sophisticated meta-learning strategies . It has also become a ubiquitous component in most state-of-the-art methods [ 27 , 28 , 29 ] . Furthermore , [ 27 ] demonstrated empirically that pre-training provides similar performance gain to earlier methods , making them ( mostly ) competitive with the state of the art . In Sec . 3 , we show that feature pre-training directly relates to meta-learning as a loss upper bound , and explains why pre-training contributes to improved performance . 3 Feature Pre-training as Meta-learning . In this section , we show how feature pre-training relates to meta-learning as a loss upper bound . More precisely , we show that the pre-training induces a special base learner with the corresponding meta-learning loss upper bounded by the cross-entropy loss ( Eq . ( 6 ) ) . Consequently , minimizing the cross-entropy loss also indirectly solves the induced meta-learning problem . Further , we observe that the special base learner is a conditional meta-learning formulation , which provides a principled explanation for the improved performance . Let T = { ( St , Qt ) } Tt=1 be a meta-training set . We denote the collection of query sets as Q = { Qt } Tt=1 = { ( Xt , Yt ) } Tt=1 where we writeQt = ( Xt , Yt ) as a tuple of input samplesXt = { xjt } nq j=1 and their corresponding labels Yt = { yjt } nq j=1 . For simplicity , we assume that the query sets are disjoint , namely Qt ∩ Qt′ = ∅ for any t 6= t′ . We merge all query sets into a flat dataset D ( Q ) = { ( xi , yi ) } Ni=1 = ∪Tt=1Qt , with N = nqT . Proposition 1 . With the notation and assumptions introduced above , let C be the total number of classes in D ( Q ) , and W ∈ RC×m the global classifier . Denote by W [ Y ] the sub-matrix with rows indexed by the sorted unique values1 from Y . Then , for any embedding ψθ : X → Rm E ( X , Y ) ∈Q [ Lce ( W [ Y ] , ( ψθ ( X ) , Y ) ) ] ≤ E ( x , y ) ∈D ( Q ) [ ` ce ( Wψθ ( x ) , y ) ] . ( 8 ) We outline the proof strategy here and defer the details to the supplementary material . We first observe that D ( Q ) has the same collection of samples as Q. Secondly , each matrix W [ Y ] is in fact a task classifier for the query set ( X , Y ) . Crucially , the likelihood of query samples p ( Y |X ) , estimated by task classifiers W [ Y ] , are no smaller than their likelihood estimated by the global classifier W , which forms the inequality in ( 8 ) . Combining the two observations yields Prop . 1 . The key implication of Prop . 1 is that , if global labels are available , we can use a special base learner wglobal ( D ) = wglobal ( X , Y ) = W [ Y ] , ( 9 ) for any dataset of the form D = ( X , Y ) . This base learner only depends on the intended classes ( i.e . unique values in Y ) to produce task classifiers and is independent of specific inputs from D ( illustrated in Fig . 1b ) . We note that for any task ( S , Q ) ∈ T , wglobal ( S ) = wglobal ( Q ) , since the support and query sets share the same class labels . These observations directly imply the following , Lemma 2 . For meta-training set T where tasks are annotated with global labels , we have E ( S , Q ) ∈T [ Lce ( wglobal ( S ) , ψθ ( Q ) ) ] = E ( X , Y ) ∈Q [ Lce ( W [ Y ] , ( ψθ ( X ) , Y ) ) ] . ( 10 ) The left-hand side of ( 10 ) is thus the meta-learning loss associated with base learner wglobal ( · ) . Hence , the combination of Prop . 1 and Lemma 2 implies that the empirical risk associated with this meta-learning loss is upper bounded by the standard cross-entropy loss , namely the right-hand side of ( 8 ) . The connection shows that feature pre-training also solves a meta-learning problem when we design the base learner to be wglobal ( · ) . Remark 1 . The bound in Prop . 1 is tight when the standard cross-entropy loss is 0 . Remark 1 shows that all task classifiers W [ Y ] and the embedding function ψθ are optimal , when standard cross-entropy is 0 . In practice , Remark 1 is achievable since over-parametrized neural networks could obtain zero empirical loss . We highlight two important properties for the base learner wglobal ( · ) . Firstly , wglobal is not the base learner used during meta-testing since it can not generalize to novel classes . In [ 23 ] , wglobal was simply replaced with Eq . ( 7 ) during meta-testing , while other works choose to fine-tune the pre-trained embedding model ψθ with a new base learner intended for meta-testing using episodic training . The advantage of the fine-tuning strategy is clear since it optimizes ψθ for the actual base learner used during test time . This observation is well supported by existing empirical results , with many state-of-the-art methods [ e.g . 28 , 29 ] adopting the fine-tuning strategy and surpassing standalone pre-training . Secondly , we observe that wglobal ( · ) describes a conditional meta-learning problem : the global labels Y provide additional side information to facilitating model learning . Specifically , global labels directly reveal how task samples relate to one another and wglobal could simply map global labels to task classifiers via W [ Y ] . In contrast , unconditional base learners ( e.g . ( 4 ) and ( 7 ) ) have to learn classifiers based on support sets , without access to task relations provided by global labels . Global labels offer significant advantages to learning the embedding model ψθ . Denevi et al . [ 3 ] proved that conditional meta-learning is advantageous over unconditional formulation by incurring a smaller excess risk , especially when the meta-distribution of tasks is organized into distant clusters 1E.g . Y = { 6 , 6 , 4 , 9 } maps to { 4 , 6 , 9 } -th rows of W . Also see Fig . 1b for an illustration . ( see [ 3 ] for further discussion ) . In practice , global labels cluster task samples for free and improve regularization by enforcing each cluster ( denoted by global label y ) to share vector W [ y ] for all task classifiers . The above analysis explains why pre-training yields significantly more robust ψθ than many ( unconditional ) meta-learning methods . Algorithm 1 : MeLa Input : meta-training set T = { Si , Qi } Ti=1 ψ0θ = argminψθ E ( S , Q ) ∈T [ L ( w ( ψθ ( S ) , ψθ ( Q ) ) ] ) Global clusters G = LearnLabeler ( ψ0θ , T ) ψ∗θ = MetaLearn ( G , T ) Return ψ∗θ Figure 2 : Effect of Pre-training on miniIMAGENET In Fig . 2 , we plot several existing results from [ 2 , 27 ] to highlight the contribution of the pretraining . For miniIMAGENET , pre-training accounted for on average 9 % improvement in ProtoNet and MatchNet . In addition , pre-training alone yields a performance mostly competitive with both ProtoNet and MatchNet , suggesting that the pre-training contributes far more towards generalization performance compared to the specific meta-learning strategies deployed . Similar trends are observed for other methods and datasets , both in the previous works and our experiments . The extensive empirical results are consistent with the theoretical advantages of conditional meta-learning .

Summary: Stemming from recent observations that standard supervised training (called "global classification") empirically works better than local meta-learners, authors propose to infer labels from a trained meta-learner in order to fine-tune the feature-extractor. leveraging the benefits of global classification without requiring absolute labels.  In more details, authors first draw a connection between global classification risk and local classification risk (Proposition 1). This proposition essentially formalises the simple idea that solving a global classification problem (i.e over the full set of classes) is harder than solving a local one (i.e with a strictly smaller set of classes).  Second, authors propose an algorithm to infer global labels for each sample in a meta-training set by leveraging the features produced by a trained local meta-learner. The method relies on alternating between clustering extracted features and pruning the centroids with low number of assigned points. During the clustering step, local constraints force samples with the same local label to share the same global assignment. This assignment is decided by a majority vote (i.e the centroid that minimises the average squared distance to all samples with same local label). Once assignments have been made, centroids are updated following standard incremental rules.     Finally, authors compare their method on popular benchmarks mini-ImageNet and Tiered-Imagenet against recent meta-learners, and perform some ablation studies on the clustering hyper-parameters and quality.