How to Train Your MAML to Excel in Few-Shot Classification

1 INTRODUCTION . Meta-learning is a sub-field of machine learning that attempts to search for the best learning strategy as the learning experiences increases ( Thrun & Pratt , 2012 ; Lemke et al. , 2015 ) . Recent years have witnessed an abundance of new approaches on meta-learning ( Vanschoren , 2018 ; Hospedales et al. , 2020 ) , among which model-agnostic meta-learning ( MAML ) ( Finn et al. , 2017 ; Finn , 2018 ) is one of the most popular algorithms , owing to its “ model-agnostic ” nature to incorporate different model architectures and its principled formulation to be applied to different problems . Concretely , MAML aims to learn a good model initialization ( through the outer loop optimization ) , which can then be quickly adapted to novel tasks given few examples ( through the inner loop optimization ) . However , in few-shot classification ( Vinyals et al. , 2016 ; Snell et al. , 2017 ) which many meta-learning algorithms are dedicated to , MAML ’ s performance has been shown to fall far behind ( Wang et al. , 2019 ; Chen et al. , 2019 ; Triantafillou et al. , 2020 ) . In this paper , we take a closer look at MAML on few-shot classification . The standard setup involves two phases , meta-training and meta-testing , in which MAML learns the model initialization during meta-training and applies it during meta-testing . In both phases , MAML receives multiple N -way Kshot tasks . Each task is an N -class classification problem provided with K labeled support examples per class . After the ( temporary ) inner loop optimization using the support examples , the updated model from the initialization is then evaluated on the query examples of the same N classes . The loss calculated on the query examples during mete-training is used to optimize the meta-parameters ( i.e. , the model initialization ) through the outer loop . For consistency , we mainly study the scenario where meta-training and meta-testing use the same number of gradient steps in the inner loop optimization . More specifically , what MAML learns in few-shot classification is the initialization of an N -class classifier . Without loss of generality , we denote a classifier by ŷ = argmaxc∈ [ N ] w > c fφ ( x ) , where fφ is the feature extractor on an example x , and { wc } Nc=1 are the weight vectors in the linear classifier head . We use θ to represent the collection of meta-parameters { φ , w1 , · · · , wN } . Our first observation is that MAML needs a large number of gradient steps in the inner loop . For example , on MiniImageNet ( Vinyals et al. , 2016 ) and TieredImageNet ( Ren et al. , 2018a ) , MAML ’ s accuracy improves along with the increased number of gradient steps and achieves the highest around 15 ∼ 20 steps , which are much larger than the conventional usage of MAML ( Antoniou et al. , 2019 ) . We attribute this to the behavior of the model initialization learned from mutually-exclusive tasks ( Yin et al. , 2020 ) , which , without any further inner loop optimization , performs at the chance level ( i.e. , 100 N % accuracy ) on query examples , not only for the meta-testing tasks but also for the meta-training tasks . In other words , the initialized model needs many gradient steps to attain a high accuracy . Our second observation is that MAML is sensitive to the permutations of class label assignments during meta-testing . Concretely , when an N -way K-shot task arrives , MAML pairs the learned initialization of wc with the corresponding class label c ∈ [ N ] of that task . The issue resides in the “ meaning ” of c in a task . In the standard setup , each N -way task is created by drawing N classes from a bigger pool of semantic classes ( e.g. , “ dog ” , “ cat ” , “ bird ” , etc . ) , followed by a random label re-assignment into c ∈ { 1 , · · · , N } . In other words , the same set of N semantic classes can be labeled totally differently into { 1 , · · · , N } and thus be paired with { wc } Nc=1 differently . Taking a five-way task for example , there are 5 ! = 120 permutations to pair the same set of five semantic classes to the linear classifiers . In some of them , a class “ dog ” may be assigned to c = 1 ; in some others , it may be assigned to c 6= 1 . While this randomness has been shown crucial in meta-training to help MAML prevent over-fitting ( Rajendran et al. , 2020 ; Yao et al. , 2021 ; Yin et al. , 2020 ) , we find that it makes the meta-testing phase unstable . Specifically , different permutations can lead to drastically different meta-testing accuracy — on average , the best permutation for each five-way one-shot task has ∼ 15 % higher accuracy than the worst permutation , on both datasets . Building upon this observation , we investigate multiple approaches to make MAML permutationinvariant , either in the meta-testing phase alone or in both phases . We find that a simple solution — meta-training only a single vector w and using it to initialize the N linear classifiers { wc } Nc=1 — performs the best . We name this approach UNICORN-MAML , as illustrated in Figure 1 ( b ) . Concretely , at the beginning of the inner loop , UNICORN-MAML duplicates w into wc , ∀c ∈ [ N ] . After the inner loop optimization , the meta-gradients with respect to { wc } Nc=1 are aggregated to updatew in the outer loop . This design not only makes UNICORN-MAML permutation-invariant , but also ensures that no single model can solve all tasks at once without inner loop optimization , to prevent memorization over-fitting ( Yin et al. , 2020 ) . On MiniImageNet ( Vinyals et al. , 2016 ) , TieredImageNet ( Ren et al. , 2018a ) , and CUB datasets ( Wah et al. , 2011 ) , UNICORN-MAML performs on a par with or even outperforms state-of-the-art few-shot classification algorithms , while preserving the simplicity of MAML without adding any extra network modules or learning strategies . 2 MAML FOR FEW-SHOT CLASSIFICATION . 2.1 PROBLEM DEFINITION . The goal of few-shot classification is to construct a classifier using limited labeled examples . The challenge is the potential over-fitting or poor generalization problem . Following ( Vinyals et al. , 2016 ) , we define a few-shot classification problem as an N -way K-shot task , which has N classes and each class has K labeled support examples . We denote the labeled support set by S = { ( xi , yi ) } N×Ki=1 ; each ( xi , yi ) is a pair of an input and a label , where yi ∈ { 1 , 2 , · · · , N } = [ N ] . The value of K is small , e.g. , K = 1 or K = 5 . To evaluate the quality of the resulting classifier , each task is associated with a query set Q , which is composed of examples of the same N classes . The core idea of meta-learning for few-shot classification is to sample few-shot tasks T = ( S , Q ) from a set of “ base ” classes , of which we have ample examples per class . Meta-learning then learns the ability of “ how to build a classifier using limited data ” from these tasks . After this meta-training phase , we then proceed to the meta-testing phase to tackle the true few-shot tasks that are composed of examples from “ novel ” classes . By default , the “ novel ” and “ base ” classes are disjoint . It is worth noting that the total number of “ base ” ( and “ novel ” ) classes is usually larger than N ( see subsection 2.3 ) . Thus , to construct an N -way K-shot task in each phase , one usually first samples N classes from the corresponding set of classes , and randomly re-labels each sampled class by an index c ∈ [ N ] . This randomness results in the so-called mutually-exclusive tasks ( Yin et al. , 2020 ) . In this paper , we will use base ( novel ) and meta-training ( meta-testing ) classes interchangeably . 2.2 MODEL-AGNOSTIC META-LEARNING ( MAML ) . As introduced in section 1 , MAML aims to learn the initialization of an N -way classifier , such that when provided with the support set S of an N -way K-shot task , the classifier can be quickly and robustly updated to perform well on the task ( i.e. , classify the query set Q well ) . Let us denote a classifier by ŷ = hθ ( x ) = argmaxc∈ [ N ] w > c fφ ( x ) , where fφ is the feature extractor , { wc } Nc=1 are the weight vectors of the classification head , and θ = { φ , w1 , · · · , wN } collects the parameters of both . MAML evaluates hθ on S and uses the gradient to update θ into θ′ , so that hθ′ can be applied to Q . This procedure is called the inner loop optimization , which usually takes M gradient steps . θ′ ← θ for m ∈ [ M ] do ( 1 ) θ′ = θ′ − α∇θL ( S , θ′ ) Here , L ( S , θ′ ) = ∑ ( x , y ) ∈S ` ( hθ′ ( x ) , y ) is the loss computed on examples of S and α is the learning rate ( or step size ) . The cross-entropy loss is commonly used for ` . As suggested in the original MAML paper ( Finn et al. , 2017 ) and ( Antoniou et al. , 2019 ) , M is usually set to a small integer ( e.g. , ≤ 5 ) . For ease of notation , let us denote the output θ′ after M gradient steps by θ′ = InLoop ( S , θ , M ) . To learn the initialization θ , MAML leverages the few-shot tasks sampled from the base classes . Let us denote by p ( T ) the distribution of tasks from the base classes , where each task is a pair of support and query sets ( S , Q ) , MAML aims to minimize the following meta-learning objective w.r.t . θ : ∑ ( S , Q ) ∼p ( T ) L ( Q , θ′S ) = ∑ ( S , Q ) ∼p ( T ) L ( Q , InLoop ( S , θ , M ) ) . ( 2 ) Namely , MAML aims to find a shared θ among tasks , which , after inner loop updates using S , can lead to a small loss on the query set Q . ( We add the subscript S to θ′ to show that θ′S depends on S . ) To optimize Equation 2 , MAML applies stochastic gradient descent ( SGD ) but at the task level . That is , at every iteration , MAML samples a task T = ( S , Q ) and computes the meta-gradient w.r.t . θ : ∇θL ( Q , θ′S ) = ∇θL ( Q , InLoop ( S , θ , M ) ) . ( 3 ) In practice , one may sample a mini-batch of tasks and compute the mini-batch meta-gradient w.r.t . θ to optimize θ . This SGD for θ is known as the outer loop optimization for MAML . It is worth noting that calculating the gradient in Equation 3 can impose considerable computational and memory burdens because it involves a gradient through a gradient ( along the inner loop but in a backward order ) ( Finn et al. , 2017 ) . Thus in practice , it is common to apply the first-order approximation ( Finn et al. , 2017 ; Nichol et al. , 2018 ) , i.e. , ∇θL ( Q , θ′S ) ≈ ∇θ′SL ( Q , θ ′ S ) . For additional related work on meta-learning and few-shot learning , please see Appendix A .

Summary: The paper mainly investigates the effect of permutation in the class label assignment in the tasks for the MAML algorithm. First, the authors show that MAML requires a higher number of inner loop updates than what is commonly used. Then, they show that MAML is sensible to the permutation of the class labels in the tasks and experimented with diverse methods to alleviate this problem. Finally, they proposed Unicorn-MAML, a modification of the MAML algorithm that learns a single weight vector for the classifier layer, to make the model permutation invariant. They show through experiments on two different datasets (mini-ImageNet and tiered-ImageNet) that it achieves or outperforms state-of-the-art performance.