The Role of Global Labels in Few-Shot Classification and How to Infer Them

1 Introduction . A central problem in meta-learning is few-shot learning ( FSL ) , where new tasks must be learned quickly given limited amount of training data . FSL has drawn increasing attention recently due to the high cost of collecting and annotating large datasets . To tackle the challenge of model generalization in FSL , meta-learning leverages past experiences of solving related tasks by directly learning transferable knowledge over a collection of FSL tasks . A diverse range of meta-learning methods tailored for FSL have been proposed , including optimization-based [ e.g . 1 , 5 , 27 ] , metric learning [ e.g . 20 , 21 , 25 ] , and model-based methods [ e.g . 7 , 14 , 19 ] . The diversity of the existing strategies raises a natural question : do these methods share any common lessons for improving model generalization and for designing future methods ? Several papers addressed the above question . Chen et al . [ 2 ] identified that data augmentation and deeper network architecture significantly improve generalization performance across several meta-learning methods . On the other hand , Tian et al . [ 23 ] investigated a simple yet competitive approach : a linear model on top of input embeddings learned via feature pre-training . This approach ignores task structures from meta-learning and merges all tasks into a “ flat ” dataset of labeled samples . The desired input embedding is then learned by classifying all classes of the flat dataset . Extensive empirical evidences supports the efficacy of feature pre-training in FSL . Pre-training alone already outperforms various meta-learning algorithms [ 23 ] . Recently , it is used as a ubiquitous 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . pre-processing step in state-of-the-art meta-learning methods [ e.g . 18 , 26 , 27 ] . In particular , [ 27 ] reported that pre-training also significantly boosted earlier methods ( see Fig . 2 ) . Despite the significant empirical improvement , there is limited theoretical understanding of why feature pre-training provides significant performance gain for meta-learning . On the other hand , pretraining requires task merging to construct the flat dataset , which implicitly assumes access to some notion of global labels consistently adopted across tasks . However , global labels may not exist or are inaccessible , such as when each task is annotated independently with only local labels . This renders direct task merging and consequently pre-training impossible ( see Fig . 1a ) . Independent annotation captures realistic scenarios when tasks are collected organically ( e.g from different users ) , rather than generated synthetically from benchmark datasets ( e.g . miniIMAGENET ) . Possible scenarios include non-descript task labels ( e.g . tasks with numerical labels ) or even concept overlaps among labels across tasks ( e.g . sea animals vs mammals ) . In this paper , we address the issues raised above with the following contributions : • In Sec . 3 , we show that feature pre-training directly relates to meta-learning as a loss upper bound . In particular , pre-training induces a conditional meta-learning formulation [ 3 , 26 ] , which provides a principled explanation for the empirical performance gains . • In Sec . 4 , we propose Meta Label Learning ( MeLa ) , a novel framework that automatically infers some notion of latent global labels consistent with local task constraints . The inferred labels enable us to leverage pre-training to improve meta-learners ’ generalization performance , and bridges the gap between experiment settings with or without acesss to global labels . • In Sec . 5 , we demonstrate empirically the competitive performance of MeLa over a suite of benchmark datasets . We also present ablation studies to highlight its key properties . In Sec . 2 , we first review the key notions of meta-learning and FSL . The supplementary material contains proofs for the theoretical analysis , additional empirical results , and experiment setup . 2 Background . We formalize FSL in the context of meta-learning , followed by reviewing feature pre-training initially investigated in [ 23 ] . Meta-learning . FSL [ 4 ] considers a meta-training set of tasks T = { ( St , Qt ) } Tt=1 , with support set St = { ( xj , yj ) } nsj=1 and query setQt = { ( xj , yj ) } nq j=1 sampled from the same distribution . Typically , St and Qt each contains a small number of samples ns and nq respectively ( fixed for all tasks for simplicity ) . We denote by D the space of datasets of the form St or Qt . FSL aims to find the best inner learning algorithm ( referred to as base learner in the following ) Alg ( θ , · ) : D → F that maps supports sets S to predictors f = Alg ( θ , S ) , such that x 7→ f ( x ) generalizes well on the corresponding query sets . The base learner is meta-parametrized by θ ∈ Θ . Formally , the meta-learning problem for FSL is min θ∈Θ E ( S , Q ) ∈T L ( Alg ( θ , S ) , Q ) , ( 1 ) where E ( S , Q ) ∈T , 1|T | ∑ ( S , Q ) ∈T denotes the empirical distribution over the meta-training set . The task loss L : F ×D → R is the empirical risk of the learner over query sets , according to an inner loss ` : Y × Y → R , where Y is the space of labels L ( f , D ) = E ( x , y ) ∈D [ ` ( f ( x ) , y ) ] . ( 2 ) Designing effective base learners is a key focus in meta-learning literature and various strategies have been explored . We focus on a broad class of methods that we call meta-representation learning [ 1 , 6 , 10 , 15 ] , which is remarkably effective in practice and closely related to feature pre-training . Meta-representation learning infers a suitable embedding for base learners by solving min θ∈Θ E ( S , Q ) ∈T [ L ( w ( ψθ ( S ) ) , ψθ ( Q ) ] ( 3 ) where ψθ : X → Rm is a feature embedding model and ψθ ( D ) , { ( ψθ ( x ) , y ) | ( x , y ) ∈ D } the embedded dataset . In [ 1 ] , the ridge regression estimator is chosen as the base learner Alg ( θ , D ) = w ( ψθ ( D ) ) w ( ψθ ( D ) ) = argmin W E ( x , y ) ∈ψθ ( D ) ‖Wx− OneHot ( y ) ‖ 2 + λ1 ‖W‖2 , ( 4 ) where λ1 is a constant and OneHot ( y ) denotes the one-hot encoding of the label y . Among different base learner designs , ridge regression is often favored for its differentiable closed-form solution and the associated computational efficiency in optimizing ( 3 ) . Feature Pre-training . Feature pre-training has been widely used in meta-learning . It was investigated in-depth in [ 23 ] . Given the meta-training set T , a “ flat ” dataset Dglobal is constructed by merging all tasks in T : Dglobal = D ( T ) = { ( xi , yi ) } Ni=1 = ⋃ ( S , Q ) ∈T ( S ∪Q ) . ( 5 ) We then learn a embedding function ψθ on Dglobal using the standard cross-entropy loss ` ce for multi-class classification : argmin θ , W E ( x , y ) ∈Dglobal [ ` ce ( Wψθ ( x ) , y ) ] . ( 6 ) After obtaining ψθ , a novel task may be solved in the embedded space by a base learner . In [ 23 ] , the authors recommended the regularized logistic regression estimator wce ( D ) = argmin W Lce ( W , D ) + λ2 ‖W‖2 , ( 7 ) for its empirical performance . Here λ2 is a regularization constant . [ 23 ] demonstrated that feature pre-training as a standalone method already outperforms many sophisticated meta-learning strategies . It has also become a ubiquitous component in most state-of-the-art methods [ 27 , 28 , 29 ] . Furthermore , [ 27 ] demonstrated empirically that pre-training provides similar performance gain to earlier methods , making them ( mostly ) competitive with the state of the art . In Sec . 3 , we show that feature pre-training directly relates to meta-learning as a loss upper bound , and explains why pre-training contributes to improved performance . 3 Feature Pre-training as Meta-learning . In this section , we show how feature pre-training relates to meta-learning as a loss upper bound . More precisely , we show that the pre-training induces a special base learner with the corresponding meta-learning loss upper bounded by the cross-entropy loss ( Eq . ( 6 ) ) . Consequently , minimizing the cross-entropy loss also indirectly solves the induced meta-learning problem . Further , we observe that the special base learner is a conditional meta-learning formulation , which provides a principled explanation for the improved performance . Let T = { ( St , Qt ) } Tt=1 be a meta-training set . We denote the collection of query sets as Q = { Qt } Tt=1 = { ( Xt , Yt ) } Tt=1 where we writeQt = ( Xt , Yt ) as a tuple of input samplesXt = { xjt } nq j=1 and their corresponding labels Yt = { yjt } nq j=1 . For simplicity , we assume that the query sets are disjoint , namely Qt ∩ Qt′ = ∅ for any t 6= t′ . We merge all query sets into a flat dataset D ( Q ) = { ( xi , yi ) } Ni=1 = ∪Tt=1Qt , with N = nqT . Proposition 1 . With the notation and assumptions introduced above , let C be the total number of classes in D ( Q ) , and W ∈ RC×m the global classifier . Denote by W [ Y ] the sub-matrix with rows indexed by the sorted unique values1 from Y . Then , for any embedding ψθ : X → Rm E ( X , Y ) ∈Q [ Lce ( W [ Y ] , ( ψθ ( X ) , Y ) ) ] ≤ E ( x , y ) ∈D ( Q ) [ ` ce ( Wψθ ( x ) , y ) ] . ( 8 ) We outline the proof strategy here and defer the details to the supplementary material . We first observe that D ( Q ) has the same collection of samples as Q. Secondly , each matrix W [ Y ] is in fact a task classifier for the query set ( X , Y ) . Crucially , the likelihood of query samples p ( Y |X ) , estimated by task classifiers W [ Y ] , are no smaller than their likelihood estimated by the global classifier W , which forms the inequality in ( 8 ) . Combining the two observations yields Prop . 1 . The key implication of Prop . 1 is that , if global labels are available , we can use a special base learner wglobal ( D ) = wglobal ( X , Y ) = W [ Y ] , ( 9 ) for any dataset of the form D = ( X , Y ) . This base learner only depends on the intended classes ( i.e . unique values in Y ) to produce task classifiers and is independent of specific inputs from D ( illustrated in Fig . 1b ) . We note that for any task ( S , Q ) ∈ T , wglobal ( S ) = wglobal ( Q ) , since the support and query sets share the same class labels . These observations directly imply the following , Lemma 2 . For meta-training set T where tasks are annotated with global labels , we have E ( S , Q ) ∈T [ Lce ( wglobal ( S ) , ψθ ( Q ) ) ] = E ( X , Y ) ∈Q [ Lce ( W [ Y ] , ( ψθ ( X ) , Y ) ) ] . ( 10 ) The left-hand side of ( 10 ) is thus the meta-learning loss associated with base learner wglobal ( · ) . Hence , the combination of Prop . 1 and Lemma 2 implies that the empirical risk associated with this meta-learning loss is upper bounded by the standard cross-entropy loss , namely the right-hand side of ( 8 ) . The connection shows that feature pre-training also solves a meta-learning problem when we design the base learner to be wglobal ( · ) . Remark 1 . The bound in Prop . 1 is tight when the standard cross-entropy loss is 0 . Remark 1 shows that all task classifiers W [ Y ] and the embedding function ψθ are optimal , when standard cross-entropy is 0 . In practice , Remark 1 is achievable since over-parametrized neural networks could obtain zero empirical loss . We highlight two important properties for the base learner wglobal ( · ) . Firstly , wglobal is not the base learner used during meta-testing since it can not generalize to novel classes . In [ 23 ] , wglobal was simply replaced with Eq . ( 7 ) during meta-testing , while other works choose to fine-tune the pre-trained embedding model ψθ with a new base learner intended for meta-testing using episodic training . The advantage of the fine-tuning strategy is clear since it optimizes ψθ for the actual base learner used during test time . This observation is well supported by existing empirical results , with many state-of-the-art methods [ e.g . 28 , 29 ] adopting the fine-tuning strategy and surpassing standalone pre-training . Secondly , we observe that wglobal ( · ) describes a conditional meta-learning problem : the global labels Y provide additional side information to facilitating model learning . Specifically , global labels directly reveal how task samples relate to one another and wglobal could simply map global labels to task classifiers via W [ Y ] . In contrast , unconditional base learners ( e.g . ( 4 ) and ( 7 ) ) have to learn classifiers based on support sets , without access to task relations provided by global labels . Global labels offer significant advantages to learning the embedding model ψθ . Denevi et al . [ 3 ] proved that conditional meta-learning is advantageous over unconditional formulation by incurring a smaller excess risk , especially when the meta-distribution of tasks is organized into distant clusters 1E.g . Y = { 6 , 6 , 4 , 9 } maps to { 4 , 6 , 9 } -th rows of W . Also see Fig . 1b for an illustration . ( see [ 3 ] for further discussion ) . In practice , global labels cluster task samples for free and improve regularization by enforcing each cluster ( denoted by global label y ) to share vector W [ y ] for all task classifiers . The above analysis explains why pre-training yields significantly more robust ψθ than many ( unconditional ) meta-learning methods . Algorithm 1 : MeLa Input : meta-training set T = { Si , Qi } Ti=1 ψ0θ = argminψθ E ( S , Q ) ∈T [ L ( w ( ψθ ( S ) , ψθ ( Q ) ) ] ) Global clusters G = LearnLabeler ( ψ0θ , T ) ψ∗θ = MetaLearn ( G , T ) Return ψ∗θ Figure 2 : Effect of Pre-training on miniIMAGENET In Fig . 2 , we plot several existing results from [ 2 , 27 ] to highlight the contribution of the pretraining . For miniIMAGENET , pre-training accounted for on average 9 % improvement in ProtoNet and MatchNet . In addition , pre-training alone yields a performance mostly competitive with both ProtoNet and MatchNet , suggesting that the pre-training contributes far more towards generalization performance compared to the specific meta-learning strategies deployed . Similar trends are observed for other methods and datasets , both in the previous works and our experiments . The extensive empirical results are consistent with the theoretical advantages of conditional meta-learning .

Summary: Authors analyze the role of global pretraining in few-shot classification, establishing classification loss over all classes as an upper bound on the classification losses from particular tasks. Authors then consider the case where task-based labels are available but are not easily collated into a global set of labels. They introduce a task-aware clustering-based method for assigning global labels to embeddings from a task-pretrained feature extractor, allowing them to achieve near-oracle performance in settings where global labels are absent. The method improves upon task-based pretraining alone by a substantial margin.