Year: 2021

Abstract:
Model-agnostic meta-learning (MAML) is arguably one of the most popular metalearning algorithms nowadays. Nevertheless, its performance on few-shot classification is far behind many recent algorithms dedicated to the problem. In this paper, we point out several key facets of how to train MAML to excel in fewshot classification. First, we find that MAML needs a large number of gradient steps in its inner loop update, which contradicts its common usage in few-shot classification. Second, we find that MAML is sensitive to the class label assignments during meta-testing. Concretely, MAML meta-trains the initialization of an N -way classifier. These N ways, during meta-testing, then have “N !” different permutations to be paired with a few-shot task of N novel classes. We find that these permutations lead to a huge variance of accuracy, making MAML unstable in few-shot classification. Third, we investigate several approaches to make MAML permutation-invariant, among which meta-training a single vector to initialize all the N weight vectors in the classification head performs the best. On benchmark datasets like MiniImageNet and TieredImageNet, our approach, which we name UNICORN-MAML, performs on a par with or even outperforms state-of-the-art few-shot classification algorithms, without sacrificing MAML’s simplicity.

heading: 1 INTRODUCTION
text: Meta-learning is a sub-field of machine learning that attempts to search for the best learning strategy as the learning experiences increases (Thrun & Pratt, 2012; Lemke et al., 2015). Recent years have witnessed an abundance of new approaches on meta-learning (Vanschoren, 2018; Hospedales et al., 2020), among which model-agnostic meta-learning (MAML) (Finn et al., 2017; Finn, 2018) is one of the most popular algorithms, owing to its “model-agnostic” nature to incorporate different model architectures and its principled formulation to be applied to different problems. Concretely, MAML aims to learn a good model initialization (through the outer loop optimization), which can then be quickly adapted to novel tasks given few examples (through the inner loop optimization).
However, in few-shot classification (Vinyals et al., 2016; Snell et al., 2017) which many meta-learning algorithms are dedicated to, MAML’s performance has been shown to fall far behind (Wang et al., 2019; Chen et al., 2019; Triantafillou et al., 2020).
In this paper, we take a closer look at MAML on few-shot classification. The standard setup involves two phases, meta-training and meta-testing, in which MAML learns the model initialization during meta-training and applies it during meta-testing. In both phases, MAML receives multiple N -way Kshot tasks. Each task is an N -class classification problem provided with K labeled support examples per class. After the (temporary) inner loop optimization using the support examples, the updated model from the initialization is then evaluated on the query examples of the same N classes. The loss calculated on the query examples during mete-training is used to optimize the meta-parameters (i.e., the model initialization) through the outer loop. For consistency, we mainly study the scenario where meta-training and meta-testing use the same number of gradient steps in the inner loop optimization.
More specifically, what MAML learns in few-shot classification is the initialization of an N -class classifier. Without loss of generality, we denote a classifier by ŷ = argmaxc∈[N ]w > c fφ(x), where fφ is the feature extractor on an example x, and {wc}Nc=1 are the weight vectors in the linear classifier head. We use θ to represent the collection of meta-parameters {φ,w1, · · · ,wN}.
Our first observation is that MAML needs a large number of gradient steps in the inner loop. For example, on MiniImageNet (Vinyals et al., 2016) and TieredImageNet (Ren et al., 2018a), MAML’s accuracy improves along with the increased number of gradient steps and achieves the highest around 15 ∼ 20 steps, which are much larger than the conventional usage of MAML (Antoniou et al., 2019). We attribute this to the behavior of the model initialization learned from mutually-exclusive tasks (Yin et al., 2020), which, without any further inner loop optimization, performs at the chance level (i.e., 100 N % accuracy) on query examples, not only for the meta-testing tasks but also for the meta-training tasks. In other words, the initialized model needs many gradient steps to attain a high accuracy.
Our second observation is that MAML is sensitive to the permutations of class label assignments during meta-testing. Concretely, when an N -way K-shot task arrives, MAML pairs the learned initialization of wc with the corresponding class label c ∈ [N ] of that task. The issue resides in the “meaning” of c in a task. In the standard setup, each N -way task is created by drawing N classes from a bigger pool of semantic classes (e.g., “dog”, “cat”, “bird”, etc.), followed by a random label re-assignment into c ∈ {1, · · · , N}. In other words, the same set of N semantic classes can be labeled totally differently into {1, · · · , N} and thus be paired with {wc}Nc=1 differently. Taking a five-way task for example, there are 5! = 120 permutations to pair the same set of five semantic classes to the linear classifiers. In some of them, a class “dog” may be assigned to c = 1; in some others, it may be assigned to c 6= 1. While this randomness has been shown crucial in meta-training to help MAML prevent over-fitting (Rajendran et al., 2020; Yao et al., 2021; Yin et al., 2020), we find that it makes the meta-testing phase unstable. Specifically, different permutations can lead to drastically different meta-testing accuracy — on average, the best permutation for each five-way one-shot task has ∼ 15% higher accuracy than the worst permutation, on both datasets. Building upon this observation, we investigate multiple approaches to make MAML permutationinvariant, either in the meta-testing phase alone or in both phases. We find that a simple solution — meta-training only a single vector w and using it to initialize the N linear classifiers {wc}Nc=1 — performs the best. We name this approach UNICORN-MAML, as illustrated in Figure 1 (b). Concretely, at the beginning of the inner loop, UNICORN-MAML duplicates w into wc,∀c ∈ [N ]. After the inner loop optimization, the meta-gradients with respect to {wc}Nc=1 are aggregated to updatew in the outer loop. This design not only makes UNICORN-MAML permutation-invariant, but also ensures that no single model can solve all tasks at once without inner loop optimization, to prevent memorization over-fitting (Yin et al., 2020). On MiniImageNet (Vinyals et al., 2016), TieredImageNet (Ren et al., 2018a), and CUB datasets (Wah et al., 2011), UNICORN-MAML performs on a par with or even outperforms state-of-the-art few-shot classification algorithms, while preserving the simplicity of MAML without adding any extra network modules or learning strategies.

heading: 2 MAML FOR FEW-SHOT CLASSIFICATION

heading: 2.1 PROBLEM DEFINITION
text: The goal of few-shot classification is to construct a classifier using limited labeled examples. The challenge is the potential over-fitting or poor generalization problem. Following (Vinyals et al., 2016), we define a few-shot classification problem as an N -way K-shot task, which has N classes and each class has K labeled support examples. We denote the labeled support set by S = {(xi, yi)}N×Ki=1 ; each (xi, yi) is a pair of an input and a label, where yi ∈ {1, 2, · · · , N} = [N ]. The value of K is small, e.g., K = 1 or K = 5. To evaluate the quality of the resulting classifier, each task is associated with a query set Q, which is composed of examples of the same N classes. The core idea of meta-learning for few-shot classification is to sample few-shot tasks T = (S,Q) from a set of “base” classes, of which we have ample examples per class. Meta-learning then learns the ability of “how to build a classifier using limited data” from these tasks. After this meta-training phase, we then proceed to the meta-testing phase to tackle the true few-shot tasks that are composed of examples from “novel” classes. By default, the “novel” and “base” classes are disjoint. It is worth noting that the total number of “base” (and “novel”) classes is usually larger than N (see subsection 2.3). Thus, to construct an N -way K-shot task in each phase, one usually first samples N classes from the corresponding set of classes, and randomly re-labels each sampled class by an index c ∈ [N ]. This randomness results in the so-called mutually-exclusive tasks (Yin et al., 2020). In this paper, we will use base (novel) and meta-training (meta-testing) classes interchangeably.

heading: 2.2 MODEL-AGNOSTIC META-LEARNING (MAML)
text: As introduced in section 1, MAML aims to learn the initialization of an N -way classifier, such that when provided with the support set S of an N -way K-shot task, the classifier can be quickly and robustly updated to perform well on the task (i.e., classify the query set Q well). Let us denote a classifier by ŷ = hθ(x) = argmaxc∈[N ]w > c fφ(x), where fφ is the feature extractor, {wc}Nc=1 are the weight vectors of the classification head, and θ = {φ,w1, · · · ,wN} collects the parameters of both. MAML evaluates hθ on S and uses the gradient to update θ into θ′, so that hθ′ can be applied to Q. This procedure is called the inner loop optimization, which usually takes M gradient steps.
θ′ ← θ for m ∈ [M ] do (1)
θ′ = θ′ − α∇θL(S,θ′) Here, L(S,θ′) = ∑ (x,y)∈S `(hθ′(x), y) is the loss computed on examples of S and α is the learning rate (or step size). The cross-entropy loss is commonly used for `. As suggested in the original MAML paper (Finn et al., 2017) and (Antoniou et al., 2019), M is usually set to a small integer (e.g., ≤ 5). For ease of notation, let us denote the output θ′ after M gradient steps by θ′ = InLoop(S,θ,M). To learn the initialization θ, MAML leverages the few-shot tasks sampled from the base classes. Let us denote by p(T ) the distribution of tasks from the base classes, where each task is a pair of support and query sets (S,Q), MAML aims to minimize the following meta-learning objective w.r.t. θ:∑
(S,Q)∼p(T )
L(Q,θ′S) = ∑
(S,Q)∼p(T )
L(Q, InLoop(S,θ,M)). (2)
Namely, MAML aims to find a shared θ among tasks, which, after inner loop updates using S, can lead to a small loss on the query set Q. (We add the subscript S to θ′ to show that θ′S depends on S .) To optimize Equation 2, MAML applies stochastic gradient descent (SGD) but at the task level. That is, at every iteration, MAML samples a task T = (S,Q) and computes the meta-gradient w.r.t. θ: ∇θL(Q,θ′S) = ∇θL(Q, InLoop(S,θ,M)). (3) In practice, one may sample a mini-batch of tasks and compute the mini-batch meta-gradient w.r.t. θ to optimize θ. This SGD for θ is known as the outer loop optimization for MAML. It is worth noting that calculating the gradient in Equation 3 can impose considerable computational and memory burdens because it involves a gradient through a gradient (along the inner loop but in a backward order) (Finn et al., 2017). Thus in practice, it is common to apply the first-order approximation (Finn et al., 2017; Nichol et al., 2018), i.e.,∇θL(Q,θ′S) ≈ ∇θ′SL(Q,θ ′ S).
For additional related work on meta-learning and few-shot learning, please see Appendix A.

heading: 2.3 EXPERIMENTAL SETUP
text: As our paper is heavily driven by empirical observations, we first introduce the three main datasets we experiment on, the neural network architectures we use, and the implementation details.
Dataset. We work on MiniImageNet (Vinyals et al., 2016), TieredImageNet (Ren et al., 2018a), and CUB datasets (Wah et al., 2011). MiniImageNet contains 100 semantic classes; each has 600 images. Following (Ravi & Larochelle, 2017), the 100 classes are split into meta-training/validation/testing sets with 64/16/20 (non-overlapped) classes, respectively. That is, there are 64 base classes and 20 novel classes; the other 16 classes are used for hyper-parameter tuning. TieredImageNet (Ren et al., 2018a) has 608 semantic classes, which are split into the three sets with 351/97/160 classes, respectively. On average, each class has ∼ 1, 300 images. CUB (Wah et al., 2011) has 200 classes, which are split into the three sets with 200/50/50 classes following (Ye et al., 2020a). All images are resized to 84× 84, following (Lee et al., 2019; Ye et al., 2020a). Training and evaluation. During meta-training, meta-validation, and meta-testing, we sample N - way K-shot tasks from the corresponding classes and images. We follow the literature (Snell et al., 2017; Vinyals et al., 2016) to study the five-way one-shot and five-way five-shot tasks. As mentioned in subsection 2.1, every time we sample five distinct classes, we randomly assign each of them an index c ∈ [N ]. During meta-testing, we follow the evaluation protocol in (Zhang et al., 2020; Rusu et al., 2019; Ye et al., 2020a) to sample 10, 000 tasks. In each task, the query set contains 15 images per class. We report the mean accuracy (in %) and the 95% confidence interval.
Model architecture. We follow (Lee et al., 2019) to use a ResNet-12 (He et al., 2016) architecture for fφ (cf. subsection 2.2), which has wider widths and Dropblock modules (Ghiasi et al., 2018). We note that many recent few-shot learning algorithms use this backbone. We also follow the original MAML (Finn et al., 2017) to use a 4-layer convolutional network (ConvNet) (Vinyals et al., 2016).
Implementation details. Throughout the paper, for simplicity and consistency, we use
• the first-order approximation for calculating the meta-gradient in the outer loop; • the same number of gradient steps in the inner loop during meta-training and meta-testing; • the weights pre-trained on the entire meta-training set to initialize φ, following the recent prac-
tice (Ye et al., 2020a; Rusu et al., 2019; Qiao et al., 2018). We note that in meta-training we still optimize this pre-trained φ in the “outer” loop to search for a better initialization.
MAML has several hyper-parameters and we select them on the meta-validation set. Specifically, for the outer loop, we learn with at most 10, 000 tasks: we group every 100 tasks into an epoch. We apply SGD with momentum 0.9 and weight decay 0.0005. We start with an outer loop learning rate 0.002 for ConvNet and 0.001 for ResNet-12, which are decayed by 0.5 and 0.1 after every 20 epochs for ConvNet and ResNet-12, respectively. For the inner loop, we have to set the number of gradient step M and the learning rate α (cf. Equation 1). We provide more details in the next section.

heading: 3 MAML NEEDS A LARGE NUMBER OF INNER LOOP GRADIENT STEPS
text: We find that for MAML’s inner loop, the number of gradient updates M (cf. Equation 1) is usually searched in a small range close to 1, e.g., M ∈ [1, 5] (Antoniou et al., 2019). At first glance, this makes sense according to the motivation of MAML (Finn et al., 2017) — with a small number of gradient steps, the resulting model will have a good generalization performance.
In our experiment, we however observe that it is crucial to explore a larger M 1. Specifically, we consider M ∈ [1, 20] along with α ∈ [10−4, 100]. We plot the meta-testing accuracy of five-way one-shot tasks on the three datasets in Figure 22, using both ResNet and ConvNet backbones. We find that MAML achieves higher and much more stable accuracy (w.r.t. the learning rate) when M is larger, e.g., larger than 10. Specifically, for MiniImageNet with ResNet, the highest accuracy 64.42% is obtained with M = 15, higher than 62.90% with M = 5; for TieredImageNet with ResNet, the highest accuracy 65.72% is obtained with M = 15, higher than 59.08% with M = 5. As will be seen in section 6, these results with a larger M are comparable with many existing algorithms.
1We reiterate that for simplicity and consistency we apply the same M in meta-training and meta-testing. 2We tune hyper-parameters on the meta-validation set and find that the accuracy there reflects the meta-testing accuracy well. We show the meta-testing accuracy here simply for a direct comparison to results in the literature.
0 1 5 10 15 20 25 30 Inner Iterations
20 30 40 50 60 70 80 90 Ac cc ur ac y (% )
Meta-Training Meta-Testing
MiniImageNet, R
0 1 5 10 15 20 25 30 Inner Iterations
20 30 40 50 60 70 80 Ac cc ur ac y (%
)
Meta-Training Meta-Testing
TieredImageNet, R
0 1 5 10 15 20 25 30 Inner Iterations
20 30 40 50 60 70 80 90 Ac cc ur ac y (%
)
Meta-Training Meta-Testing
CUB, R
0 1 5 10 15 20 25 30 Inner Iterations
20 30 40 50 60 70 Ac cc ur ac y (% )
Meta-Training Meta-Testing
MiniImageNet, C
0 1 5 10 15 20 25 30 Inner Iterations
20 25 30 35 40 45 50 55 60 Ac cc ur ac y (% )
Meta-Training Meta-Testing
TieredImageNet, C
0 1 5 10 15 20 25 30 Inner Iterations
20 30 40 50 60 70 Ac cc ur ac y (% )
Meta-Training Meta-Testing
CUB, C
Figure 3: We plot the change of the five-way one-shot classification accuracy (on the query set), averaged over 10,000 tasks sampled from either the meta-training (red) or meta-testing classes (blue), along with the process of inner loop updates, using the best model initialization learned by MAML. C: ConvNet; R: ResNet.
To analyze why MAML needs a large M , we plot the change of classification accuracy along with the inner loop updates in Figure 3. Specifically, we first perform meta-training using theM value selected by meta-validation, for each pair of dataset and backbone. We then analyze the learned initialization on few-shot tasks sampled from the meta-training and meta-testing classes, by performing 0 ∼ 30 inner loop updates using the support set and reporting the accuracy on the query set. We conduct the same experiments for five-way five-shot tasks in Appendix C.
We have two observations. First, in general, the more inner loop updates we perform for a few-shot task, the higher the accuracy is, no matter if it is a meta-training or meta-testing task. This trend aligns with the few-shot regression study in (Finn et al., 2017). Second, before any inner loop update, the learned initialization θ has a ∼ 20% accuracy on average, i.e., the accuracy by random classification. Interestingly, this is the case not only for meta-testing tasks but also for meta-training tasks, even though the learned initialization does contain the classification head {wc}Nc=1. This explains why a larger number of inner loop gradient steps are needed, since the learned initialized model has to be updated from performing random predictions to achieving a much higher classification accuracy.
We attribute the second observation to the random class label assignments in creating few-shot tasks (cf. subsection 2.1 and subsection 2.3), which make the created tasks mutually-exclusive (Yin et al., 2020) — i.e., a single model cannot solve them all at once before inner loop optimization. Concretely, for a few-shot task of a specific set of N semantic classes (e.g., {“dog”, “cat”, · · · ,“bird”}), such a randomness can turn it into different tasks from MAML’s perspective. For instance, the class “dog” may be assigned to c = 1 and paired with w1 at the current task, but to c = 2 and paired with w2 when it is sampled again. For a five-way task, the same set of five semantic classes can be assigned to {1, · · · , 5} in 120 (i.e., 5!) different permutations. As a result, if we directly apply the learned initialization of MAML without inner loop updates, the accuracy on few-shot tasks of the same set of semantic classes but in different permutations could cancels each other out. (Please see subsection B.1 for details.) Besides, since the randomness occurs also in meta-training, {wc}Nc=1 will be discouraged to learn specific knowledge of any semantic classes (Rajendran et al., 2020; Yao et al., 2021; Yin et al., 2020), hence producing an accuracy at the chance level even on meta-training tasks.
Algorithm 1: Evaluation of the effect of class label permutations on meta-testing tasks. Given the learned initialization θ by MAML for t ∈ {1, · · · , 2000} do
Sample a meta-testing task T = (S,Q) and initialize an accuracy vector at ∈ R120 for p ∈ {1, · · · , 120} do
Shuffle labels with a specific permutation π : [N ] 7→ [N ]: (S,Q) becomes (Sπ,Qπ) Update θ to get θ′ = InLoop(Sπ,θ,M), evaluate θ′ onQπ , and record at[p]
end Sort values in the accuracy vector at in the descending order and get a(sort)t
end Average over 2000 tasks by 1
2000 ∑ t a (sort) t
Related work. Several following-up works of MAML, e.g., (Hsu et al., 2019), use different numbers of inner loop steps in meta-training (e.g., M = 1 ∼ 5) and meta-testing (e.g., M = 20 ∼ 50 ). We make M equal in the two phases for consistency and provide a detailed analysis of why MAML needs a large M value. On large-scale tasks beyond few-shot learning, Shin et al. (2021) also finds the necessity of a larger number of inner loop steps, but from a different perspective than ours.

heading: 4 MAML IS SENSITIVE TO THE LABEL PERMUTATIONS IN META-TESTING
text: The randomness in class label assignments raises an interesting question: do different permutations result in different meta-testing accuracy after inner loop updates? More specifically, if {wc}Nc=1 are paired with the N classes differently, will the updated model after the inner loop perform differently?
To answer this question, we conduct a detailed experiment: Algorithm 1 summarizes the procedure. We focus on five-way one/five-shot tasks on MiniImageNet and TieredImageNet, using the ResNet backbone. For each task type and dataset combination, we first meta-train the model initialization using MAML, and then evaluate the learned initialized on 2, 000 meta-testing tasks. For each task, there are 120 permutations; each permutation, after the inner loop, would likely lead to a different model and query set accuracy. We sort the 120 accuracy for each task, and take average over 2, 000 tasks for each position in the sorted list. This results in 120 averaged accuracy, each for a specific position in the sorted list. Specifically, the highest accuracy corresponds to the case that each task cherry picks its best permutation according to the query set accuracy after inner loop optimization.
We show the histogram of the 120 average meta-testing accuracy in Figure 4. There exists a huge variance. Specifically, the best permutation can be 15%/8% higher than the worst in one/five-shot tasks. The best permutation is also much higher than vanilla MAML’s results (from section 3), which are 64.42%/83.44%/65.72%/84.37%, corresponding to the four sub-figures from left to right in Figure 4. What’s more, the best permutation can easily achieve state-of-the-art accuracy (see section 6).
Of course, so far we find the best permutation via cherry-picking — by looking at the meta-testing accuracy — so it is like an upper bound. However, if we can find the best permutation without looking at the query sets’ labels or make MAML permutation-invariant, we may practically improve MAML.
Related work. We note that we investigate a different effect caused by the permutations of class label assignments from (Rajendran et al., 2020; Yao et al., 2021; Yin et al., 2020). There, the permutations lead to mutually-exclusive meta-training tasks, which help prevent MAML from over-fitting. Here, we look at the meta-testing tasks, and permutations result in a huge variance of meta-testing accuracy.
Table 2: Taking ensemble over updated of different permutations. (The confidence interval is omitted due to space limit.)
Mini Vanilla Full Rotated Tiered Vanilla Full Rotated
1-Shot 64.42 65.50 65.37 1-Shot 65.72 66.68 66.63 5-Shot 83.44 84.43 84.40 5-Shot 84.37 84.83 84.81
Table 3: We average the top-layer classifiers and expand it to N -way during meta-testing.
MiniImageNet TieredImageNet
1-Shot 64.40 ± 0.21 66.24 ± 0.24 5-Shot 84.24 ± 0.13 84.52 ± 0.16

heading: 5 MAKING MAML PERMUTATION-INVARIANT DURING META-TESTING
text: We study approaches that can make MAML permutation-invariant during meta-testing. That is, we take the same learned initialization θ as in section 4 without changing the meta-training phase.
We first investigate searching for the best permutation for each task. As we cannot access query sets’ labels, we use the support sets’ data as a proxy. We choose the best permutation according to which permutation, either before or after inner loop updates (less practical), leads to the highest accuracy or smallest loss on the support set. Table 1 summarizes the results: none of them leads to consistent gains. We hypothesize two reasons. First, due to the mutually-exclusive tasks in metatraining, the learned θ by MAML would produce chance-level predictions before updates. Second, after updates, the support set accuracy quickly goes to 100% and is thus not informative.
In stead of choosing one from many, we further explore taking ensemble (Breiman, 1996; Zhou, 2012; Dietterich, 2000) over the predictions made by updated models of different permutations. We note that this makes MAML permutation-invariant but inevitably needs more computations. To make the ensemble process clear, we permute the weight vectors in {wc}Nc=1 rather than the class label assignments in a task: the two methods are equivalent but the latter is easier for aggregating the predictions. We study two variants: (a) full permutations (i.e., 120 of them in five-way tasks), which is intractable for larger N ; (b) rotated permutations, which is to rotate the index of {wc}Nc=13, leading to N permutations. Table 2 shows the results — ensemble consistently improves MAML. Even with the rotated version that has much fewer permutations than the full version, the gains are comparable.
We emphasize that our focus here is to make MAML permutation-invariant in meta-testing, not to explore and compare all potential ways of performing ensemble on MAML.
We further explore an efficient approach to make MAML permutation-invariant, which is to manipulate the learned initialization of {wc}Cc=1. Concretely, MAML is sensitive to the permutations in class assignments becausewc 6= wc′ for c 6= c′. One method to overcome this is to makewc = wc′ during meta-testing. Here, we investigate re-initializing each wc by their average in meta-testing: wc ← 1N ∑N c=1wc. By doing so, no matter which permutation we perform, the model after inner loop optimization will be the same and lead to the same query set accuracy. Table 3 summarizes the results; this approach improves vanilla MAML (see Table 2) in three of the four cases.
At first glance, this approach may not make sense since the resulting {wc}Cc=1, before updates, simply make random predictions. However, please note that even the original {wc}Cc=1 have an averaged chance accuracy (cf. Figure 3). In Appendix D, we provide an explanation of this approach by drawing an analogy to dropout (Srivastava et al., 2014). In short, in meta-training, we receive a task with an arbitrary permutation, which can be seen as drawing a permutation at random for the task. In meta-testing, we then take expectation over permutations, which essentially lead to the averaged wc.
3That is, we consider re-assignwc tow(c+γ mod N)+1, where γ ∈ [N ].

heading: 6 UNICORN-MAML: LEARNING A SINGLE WEIGHT VECTOR
text: The experimental results in section 5 are promising: by making MAML permutation-invariant in meta-testing, we can potentially improve vanilla MAML. While ensemble inevitably increases the computation burdens, the method by manipulating the learned initialization of {wc}Nc=1 keeps the same run time as vanilla MAML. In this section, we investigate the latter approach further. We ask:
If we directly learn a single weight vectorw to initialize {wc}Nc=1 in meta-training, making the inner loop optimization in meta-training and meta-testing consistent and both permutation-invariant, can we further improve the accuracy?
Concretely, we redefine the learnable meta-parameters θ of MAML, which becomes θ = {φ,w}, where φ is for the feature extractor. We name this method UNICORN-MAML, as we meta-train only a single weight vector w in the classification head. The inner loop optimization and outer loop optimization of UNICORN-MAML very much follow MAML, with some slight changes.
• Inner loop optimization: At the beginning, w is duplicated into {wc = w}Nc=1. That is, we use w to initialize every wc ∈ {wc = w}Nc=1. These {wc = w}Nc=1 then undergo the same inner loop optimization process as vanilla MAML (cf. Equation 1). • Outer loop optimization: Let us denote the updated model by θ′ = {φ′,w′1, · · · ,w′N}. To perform the outer loop optimization forw in meta-training, we collect the gradients derived from the query set Q. Let us denote by∇wcL(Q,θ′) the gradient w.r.t. the initialedwc (cf. subsection 2.2). Since wc is duplicated from w, we obtain the gradient w.r.t. w by ∑ c∈[N ]∇wcL(Q,θ′).
Table 4 summarizes the results of UNICORN-MAML, together with the results by MAML and existing few-shot learning algorithms. UNICORN-MAML consistently improves MAML. Specifically, on MiniImageNet, UNICORN-MAML has a 0.7% gain on one-shot tasks and 0.8% gain on five-shot tasks. On TieredImageNet, UNICORN-MAML has significant improvements (3.5% gain on one-shot tasks and 1.6% gain on five-shot tasks). Importantly, UNICORN-MAML performs on a par with many state-of-the-art methods on the one-shot setting, and notably outperforms them in the five-shot setting. Specifically, compared to ProtoMAML and MetaOptNet, which are both permutation-invariant (see the related work paragraph at the end of this section), UNICORN-MAML notably outperforms them.
Other results. We evaluate UNICORN-MAML on CUB (Wah et al., 2011) and use the ConvNet backbone on MiniImageNet in the appendix. UNICORN-MAML achieves promising improvements.
Why does UNICORN-MAML work? While for UNICORN-MAML, the mutually exclusive tasks (Yin et al., 2020) make no difference in meta-training, the design of UNICORN-MAML encourages itself to learn from the support set — otherwise, the query set accuracy will be at the chance level.
Embedding adaptation is needed. We analyze UNICORN-MAML in terms of its inner loop updates during meta-testing, similar to Figure 3. This time, we also investigate updating or freezing the
feature extractor φ. Figure 5 shows the results on five-way one- and five-shot tasks on both datasets. UNICORN-MAML’s accuracy again begins with 20% but rapidly increases along with the inner loop updates. In three out of four cases, adapting the feature extractor φ is necessary for claiming a higher accuracy, even if the backbone has been well pre-trained, which aligns with the recent claim by Arnold & Sha (2021): “Embedding adaptation is still needed for few-shot learning.”
Experiments on larger shots and transferability. MAML or similar algorithms that involve a bi-level optimization problem (i.e., inner loop and outer loop optimization) are often considered more complicated and computationally expensive than algorithms without bi-level optimization, such as ProtoNet (Snell et al., 2017) or SimpleShot (Wang et al., 2019). Nevertheless, the inner loop optimization does strengthen MAML’s adaptability during meta-testing, especially (a) when the meta-testing tasks are substantially different from the meta-training tasks (e.g., meta-training using MiniImageNet but meta-testing on CUB) or (b) when the number of shots increases (e.g., from 1 ∼ 5 to 10 ∼ 50). In Table 5 and Table 6, we conduct further experiments to justify these aspects. Related work. We note that some variants of MAML are permutation-invariant, even though they are not designed for the purpose. For example, LEO (Rusu et al., 2019) computes class prototypes (i.e., averaged features per class) to represent each class and use them to produce task-specific initialization. However, it introduces additional sub-networks. MetaOptNet (Lee et al., 2019) performs inner loop optimization only on {wc}Nc=1 (till convergence), making it a convex problem which is not sensitive to the initialization and hence the permutations. This method, however, has a high computational burden and needs careful hyper-parameter tuning for the additionally introduced regularizers. Proto-MAML (Triantafillou et al., 2020) initializes the linear classifiers {wc}Nc=1 with the prototypes, which could be permutation-invariant but cannot achieve accuracy as high as our UNICORN-MAML.

heading: 7 DISCUSSION AND CONCLUSION
text: There have been an abundance of “novel” algorithms proposed for few-shot classification (Hospedales et al., 2020; Wang et al., 2020). In these papers, MAML (Finn et al., 2017) is frequently considered as a baseline, but shows inferior results. This raises our interests. Is it because MAML is not suitable for few-shot classification, or is it because MAML has not been applied appropriately to the problem?
We thus conduct a series of analyses on MAML for few-shot classification, including hyper-parameter tuning and the sensitivity to the permutations of class label assignments in few-shot tasks. We find that by using a large number of inner loop gradient steps (in both meta-training and meta-testing), MAML can achieve comparable results to many existing algorithms. By further making MAML permutation-invariant to the class label assignments, we present UNICORN-MAML, which arrives at the state-of-the-art accuracy on five-shot tasks, without the need to add extra sub-networks. We hope that UNICORN-MAML could serve as a strong baseline for future work in few-shot classification.

heading: A BACKGROUNDS ON META-LEARNING AND FEW-SHOT LEARNING
text: Training a model under data budgets is important in machine learning, computer vision, and many other application fields, since the costs of collecting data and labeling them are by no means negligible. This is especially the case for deep learning models in visual recognition (He et al., 2016; Dosovitskiy et al., 2021; Simonyan & Zisserman, 2015; Szegedy et al., 2015; Krizhevsky et al., 2012; Huang et al., 2017), which usually need thousands of, millions of, or even more images to train (Russakovsky et al., 2015; Deng et al., 2009; Guo et al., 2016; Thomee et al., 2016; Mahajan et al., 2018; Joulin et al., 2016) in a conventional supervised manner. Different from training a model to predict at the instance level, meta-learning attempts to learn the inductive bias across training tasks (Baxter, 2000; Vilalta & Drissi, 2002). More specifically, meta-learning aims to train a “meta-model” to summarize the common characteristics of tasks and generalize them to those novel but related tasks (Maurer, 2009; Maurer et al., 2016; Denevi et al., 2018). Meta-learning has been applied in various fields, including few-shot learning (Ravi & Larochelle, 2017; Snell et al., 2017; Wang & Hebert, 2016; Ye et al., 2020a;b; Sung et al., 2018; Zhang et al., 2018a; Wang et al., 2018b), optimization (Andrychowicz et al., 2016; Wichrowska et al., 2017; Li & Malik, 2017; Bello et al., 2017), reinforcement and imitation learning (Stadie et al., 2018; Frans et al., 2018; Wang et al., 2017a; Duan et al., 2016; 2017; Yu et al., 2018), unsupervised learning (Garg & Kalai, 2018; Metz et al., 2019; Edwards & Storkey, 2017; Reed et al., 2018), continual learning (Riemer et al., 2019; Kaiser et al., 2017; Al-Shedivat et al., 2018), imbalance learning (Wang et al., 2017c; Ren et al., 2018b), transfer and multi-task learning (Motiian et al., 2017; Balaji et al., 2018; Ying et al., 2018; Zhang et al., 2018b; Li et al., 2019; 2018), active learning (Ravi & Larochelle, 2018; Sharma et al., 2018; Bachman et al., 2017; Pang et al., 2018), data compression (Wang et al., 2018a), architecture search (Elsken et al., 2019), recommendation systems (Vartak et al., 2017), data augmentation (Ratner et al., 2017), teaching (Fan et al., 2018), hyper-parameter tuning (Franceschi et al., 2017; Probst et al., 2019), etc.
In few-shot learning (FSL), meta-learning is applied to learn the ability of “how to build a classifier using limited data” that can be generalized across tasks. Such an inductive bias is first learned over few-shot tasks composed of “base” classes, and then evaluated on tasks composed of “novel” classes. For example, few-shot classification can be implemented in a non-parametric way with soft nearest neighbor (Vinyals et al., 2016) or nearest center classifiers (Snell et al., 2017), so that the feature extractor is learned and acts at the task level. The learned features pull similar instances together and push dissimilar ones far away, such that a test instance can be classified even with a few labeled training examples (Koch et al., 2015). Considering the complexity of a hypothesis class, the model training configurations (i.e., hyper-parameters) also serve as a type of inductive biases. Andrychowicz et al. (2016); Ravi & Larochelle (2017) meta-learn the optimization strategy for each task, including the learning rate and update directions. Other kinds of inductive biases are also explored. Hariharan & Girshick (2017); Wang et al. (2018b) learn a data generation prior to augment examples given few images; Dai et al. (2017) extract logical derivations from related tasks; Wang et al. (2017b); Shyam et al. (2017) learn the prior to attend images.
Model-agnostic meta-learning (MAML) (Finn et al., 2017) proposes another inductive bias, i.e., the model initialization. After the model initialization that is shared among tasks has been meta-trained, the classifier of a new few-shot task can be fine-tuned with several steps of gradient descent from that initial point. The universality of this MAML-type updates is proved in (Finn & Levine, 2018). MAML has been applied in various scenarios, such as uncertainty estimation (Finn et al., 2018), robotics control (Yu et al., 2018; Clavera et al., 2019), neural translation (Gu et al., 2018), language generation (Huang et al., 2018), etc.
Despite the success, there are still problems with MAML. For example, Nichol et al. (2018) handle the computational burden by presenting a family of approaches using first-order approximations; Rajeswaran et al. (2019) propose to leverage implicit differentiation, making the calculation of the meta-gradients much efficient and accurate. Antoniou et al. (2019) provide a bunch of tricks to train and stabilize the MAML framework. Bernacchia (2021) points out that negative rates of gradient updates help in some scenarios. Rajendran et al. (2020); Yao et al. (2021); Yin et al. (2020) argue that the learned initialization by MAML may be at high risk of (a) memorization over-fitting, where it solves meta-training tasks without the need of inner loop optimization, or (b) learner over-fitting, where it over-fits to the meta-training tasks and fails to generalize to the meta-testing tasks. They thus propose to improve MAML by imposing a regularizer or performing data augmentation.
Since MAML applies a uniform initialization to all the tasks (i.e., the same set of {wc}Nc=1 and φ), recent methods explore ways to better incorporate task characteristics. Lee et al. (2019); Bertinetto et al. (2019) optimize the linear classifiers {wc}Nc=1 (not the feature fφ) till convergence in the inner loop; Triantafillou et al. (2020) initialize the linear classifiers using class prototypes (i.e., aggregated features per class) so they are task-aware even before the inner loop optimization. Another direction is to enable task-specific initialization for the entire model (Requeima et al., 2019; Vuorio et al., 2019; Yao et al., 2019; Rusu et al., 2019), which often needs additional sub-networks.
Our work is complementary to the above improvements of MAML: we find an inherent permutation issue of MAML in meta-testing and conduct a detailed analysis. We then build upon it to improve MAML. We note that some of the above methods can be permutation-invariant even though they are not designed for the purpose. For example, LEO (Rusu et al., 2019) computes class prototypes (i.e., averaged features per class) to represent each semantic class4. However, it introduces additional sub-networks. MetaOptNet (Lee et al., 2019) performs inner loop optimization only on {wc}Nc=1 (till convergence), making it a convex problem which is not sensitive to the initialization and hence the permutations. This method, however, has a high computational burden and needs careful hyperparameter tuning for the additionally introduced regularizers. Proto-MAML (Triantafillou et al., 2020) initializes the linear classifiers {wc}Nc=1 with the prototypes, which could be permutation-invariant but cannot achieve accuracy as high as our UNICORN-MAML.

heading: B PERMUTATIONS OF CLASS LABEL ASSIGNMENTS
text: We provide more analyses and discussions on the permutation issue in the class label assignment. As illustrated in Figure 1 (a), few-shot tasks of the same set of N semantic classes (e.g., “unicorn”, “bee”, etc.) can be associated with different label assignments (i.e., c ∈ [N ]) and are paired with the learned initialization {wc}Nc=1 of MAML differently. For five-way tasks, there are 120 permutations. In section 4, we study how the permutations affect the meta-testing accuracy (after the inner loop optimization of φ and {wc}Nc=1) and see a high variance among the permutations. We note that, the inner loop optimization updates not only the linear classifiers {wc}Nc=1, but also φ of the feature extractor. Different permutations therefore can lead to different feature extractors.
Here, we further sample a five-way one-shot meta-testing task, and study the change of accuracy along with the inner loop updates (using a MAML trained with a fixed number of inner loop updates). Specifically, we plot both the support set and query set accuracy for each permutation. As shown in Figure F, there exists a high variance of query set accuracy among permutations after the inner loop optimization. This is, however, not the case for the support set. (The reason that only three curves appear for the support set is because there are only five examples, and all the permutations reach 100% support set accuracy within five inner loop steps.) Interestingly, for all the permutations, their initialized accuracy (i.e., before inner loop optimization) is all 20%. After an investigation, we find that the meta-learned {wc}Nc=1 (initialization) is dominated by one of them; i.e., all the support or query examples are classified into one class. While this may not always be the case for other few-shot tasks or if we re-train MAML, for the task we sampled, it explains why we obtain 20% for all permutations. We note that, even with an initial accuracy of 20%, the learned initialization can be updated to attain high classification accuracy.
4We note that while Requeima et al. (2019); Vuorio et al. (2019); Yao et al. (2019) also enable task-specific initialization with additional sub-networks for task embedding, their methods cannot resolve the permutation issue. This is because they take an average of the feature embeddings over N classes to represent a task.
0 1 5 10 15 20 25 30 Inner Iterations
20
30
40
50
60
70
80
90
100
Su pp
or t
Se t
Ac cc
ur ac
y (%
)
0 1 5 10 15 20 25 30 Inner Iterations
20
30
40
50
60
70
80
90
100
Q ue
ry S
et A
cc cu
ra cy
( %
)
Figure F: The support (left) and query (right) set accuracy on a randomly sampled five-way one-shot meta-testing task from MiniImageNet. We plot the accuracy of each permutation (totally 120) along with the process of inner loop optimization (the same permutation is colored the same in the left and right images).
We further compare the change of support and query set accuracy along with the inner loop optimization in Figure F. We find that, while both accuracy increases, since the support set accuracy converges quickly and has a smaller variance among permutations, it is difficult to use its information to determine which permutation leads to the highest query set accuracy. This makes sense since the support set is few-shot: its accuracy thus cannot robustly reflect the query set accuracy. This explains why the methods studied in Table 1 cannot determine the best permutation for the query set.
B.1 MATHEMATICAL EXPLANATION
We provide a simple mathematical explanation for why, on average, the query set accuracy is at the chance level if we directly apply the learned initialized model. Suppose we have a five-way task with five semantic classes {“dog”, “cat”, “bird”, “car”, “person”}. Without loss of generality, let us assume that the query set has only five examples, one from each class. Let us also assume that the best permutation — i.e., the best assignment of {w1, · · · ,w5} to these classes — gives a 100% query set accuracy using the initialized model. Since there are in total 120 possible permutations, there will be 10 of them with 60% accuracy (i.e., by switching two-class indices), 20 of them with 40% accuracy (i.e., by shuffling the indices of three classes such that they do not take their original indices), 45 of them with 20% accuracy, and 54 of them with 0% accuracy. Taking an average over these permutations gives a 20% accuracy. In other words, even if one of the permutations performs well, on average the accuracy will be close to random.

heading: C ADDITIONAL EXPERIMENTAL RESULTS
text: Similar to Figure 2, we plot the meta-testing accuracy of five-way five-shot tasks on the three datasets over ResNet and ConvNet backbones in Figure 2. We get a similar trend with Figure 2 where MAML achieves higher and much more stable accuracy (w.r.t. the learning rate) when M is larger than 15. Specifically, for MiniImageNet with ResNet, the highest accuracy 83.44% is obtained with M = 20, higher than 80.81% with M = 5.
We also plot the change of five-way five-shot classification accuracy (on the query set), averaged over 10,000 tasks sampled from either the meta-training (red) or meta-testing classes (blue), along with the process of inner loop updates, using the best model initialization learned by MAML in Figure H for each pair of dataset and backbone. There indicates a similar phenomenon as Figure 3, where MAML needs a large M .
We further evaluate UNICORN-MAML on CUB dataset with ResNet-12 backbone and MiniImageNet with the four-layer ConvNet backbone. The results are listed in Table G and Table H, respectively. By comparing UNICORN-MAML with others, we find the carefully tuned MAML shows promising results and UNICORN-MAML outperforms the existing methods.
(a) MiniImageNet
83.44
1E-3 5E-3 1E-2 5E-2 1E-1
1
5
10
15
20 74
76
78
80
82
84
MiniImageNet, ResNet(b) TieredImageNet
84.37
1E-3 5E-3 1E-2 5E-2 1E-1
1
5
10
15
20 76
78
80
82
84
TieredImageNet, ResNet(c) CUB
90.35
1E-3 5E-3 1E-2 5E-2 1E-1
1
5
10
15
20 82
84
86
88
92
90
CUB, ResNet
(d) MIniImageNet - ConvNet
71.72
1E-3 5E-3 1E-2 5E-2 1E-1
1
5
10
15
20 62
64
66
68
70
72
MiniImageNet, ConvNet(e) TieredImageNet - ConvNet
71.49
1E-3 5E-3 1E-2 5E-2 1E-1
1
5
10
15
20 62
64
66
68
72
70
TieredImageNet, ConvNet(f) CUB - ConvNet
74.60
1E-3 5E-3 1E-2 5E-2 1E-1
1
5
10
15
20 64
66
68
70
72
74
CUB, ConvNet
Figure G: Heat maps of MAML’s five-way five-shot accuracy on MiniImageNet, TieredImageNet, and CUB w.r.t. the inner loop learning rate α (x-axis) and the number of inner loop updates M (y-axis). For each map, we set accuracy below a threshold to a fixed value for clarity; we denote the best accuracy by a black box.
0 1 5 10 15 20 25 30 Inner Iterations
20 30 40 50 60 70 80 90
100
Ac cc
ur ac
y (%
)
Meta-Training Meta-Testing
MiniImageNet, ResNet
0 1 5 10 15 20 25 30 Inner Iterations
20 30 40 50 60 70 80 90
100
Ac cc
ur ac
y (%
)
Meta-Training Meta-Testing
TieredImageNet, ResNet
0 1 5 10 15 20 25 30 Inner Iterations
20 30 40 50 60 70 80 90
100
Ac cc
ur ac
y (%
)
Meta-Training Meta-Testing
CUB, ResNet
0 1 5 10 15 20 25 30 Inner Iterations
20 30 40 50 60 70 80 Ac cc ur ac y (%
)
Meta-Training Meta-Testing
MiniImageNet, ConvNet
0 1 5 10 15 20 25 30 Inner Iterations
20 30 40 50 60 70 80 Ac cc ur ac y (% )
Meta-Training Meta-Testing
TieredImageNet, ConvNet
0 1 5 10 15 20 25 30 Inner Iterations
20 30 40 50 60 70 80 Ac cc ur ac y (%
)
Meta-Training Meta-Testing
CUB, ConvNet
Figure H: We plot the change of the five-way five-shot classification accuracy (on the query set), averaged over 10,000 tasks sampled from either the meta-training (red) or meta-testing classes (blue), along with the process of inner loop updates, using the best model initialization learned by MAML.

heading: D ADDITIONAL EXPLANATIONS OF OUR STUDIED METHODS
text: We provide some more explanations on the ensemble and forced permutation invariance methods introduced in section 5. For the ensemble method, give a few-shot task, we can permute {wc}Nc=1 to pair them differently with such a task. We can then perform different inner loop optimization to obtain a set of five-way classifiers that we can perform ensemble upon. In the main text, we average the posterior probabilities of these five-way classifiers to make the final predictions.
Since the permutation affects the meta-training phase as well, we can interpret the meta-training phase as follows. Ever time we sample a few-shot task T = (S,Q), we also sample a permutation π : [N ] 7→ [N ] to re-label the classes. (We note that, this is implicitly done when few-shot tasks are
Table G: 5-Way 1/5-Shot classification accuracy and 95% confidence interval on CUB, evaluated over 10,000 tasks with a ResNet-12 backbone. ‡: methods with a ResNet-18 backbone. †: We train MAML with 5 inner loop steps in both meta-training and meta-testing. ?: we carefully select the number of inner loop steps for MAML, based on the meta-validation set.
ResNet-12 1-Shot 5-Shot
MatchNet (Vinyals et al., 2016) 66.09 ± 0.92 82.50 ± 0.58 ProtoNet (Snell et al., 2017) 71.87 ± 0.85 85.08 ± 0.57 DeepEMD (Zhang et al., 2020) 75.65 ± 0.83 88.69 ± 0.50 Baseline++ (Chen et al., 2019)‡ 67.02 ± 0.90 83.58 ± 0.54
AFHN† (Li et al., 2020) 70.53 ± 1.01 83.95 ± 0.63 Neg-Cosine (Liu et al., 2020)‡ 72.66 ± 0.85 89.40 ± 0.43 Align (Afrasiyabi et al., 2020)‡ 74.22 ± 1.09 88.65 ± 0.55
MAML (5-Step†) 76.53 ± 0.20 88.34 ± 0.16 MAML (Our reimplementation?) 77.67 ± 0.20 90.35 ± 0.16
UNICORN-MAML 78.07 ± 0.20 91.67 ± 0.16
Table H: 5-Way 1-Shot and 5-Shot classification accuracy and 95% confidence interval on MiniImageNet over 10,000 tasks with a four-layer ConvNet backbone. †: We train MAML with 5 inner loop steps in both meta-training and meta-testing. ?: we carefully select the number of inner loop steps for MAML, based on the meta-validation set.
ConvNet 1-Shot 5-Shot
MAML (Finn et al., 2017) 48.70 ± 1.84 63.11 ± 0.92 MAML++ (Antoniou et al., 2019) 52.15 ± 0.26 68.32 ± 0.44
Reptile (Nichol et al., 2018) 49.97 ± 0.32 65.99 ± 0.58 FEAT (Ye et al., 2020a) 55.15 ± 0.20 71.61 ± 0.16
KTN (visual) (Peng et al.) 54.61 ± 0.80 71.21 ± 0.66 PARN (Wu et al., 2019) 55.22 ± 0.84 71.55 ± 0.66 MAML-MMCF (Yao et al., 2021) 50.35 ± 1.82 64.91 ± 0.96 MAML (5-Step†) 53.15 ± 0.20 67.01 ± 0.16 MAML (Our reimplementation?) 54.89 ± 0.20 71.72 ± 0.16 UNICORN-MAML 55.70 ± 0.20 72.68 ± 0.16
sampled.) We then take Tπ = (Sπ,Qπ) to optimize θ in the inner loop. That is, in meta-training, the objective function in Equation 2 can indeed be re-written as∑
(S,Q)∼p(T ),π∼p(π)
L(Qπ,θ′) = ∑
(S,Q)∼p(T ),π∼p(π)
L(Qπ, InLoop(Sπ,θ,M)), (D)
where p(π) is a uniform distribution over all possible permutations. Equation D can be equivalently re-written as ∑
(S,Q)∼p(T ),π∼p(π)
L(Q,θ′π) = ∑
(S,Q)∼p(T ),π∼p(π)
L(Q, InLoop(S,θπ,M)), (E)
where θπ means that the initialization of the linear classifiers {wc}Nc=1 are permuted; θ′π is the corresponding updated model. This additional sampling process of π is reminiscent of dropout (Srivastava et al., 2014), which randomly masks out a neural network’s neurons or edges to prevent an over-parameterized neural network from over-fitting. During testing, dropout takes expectation over the masks. We also investigate a similar idea during meta-testing, by taking expectation (i.e., average) on the learned initiation of the linear classifiers over different permutations. This results in a new initialization during the meta-testing phase: wc ← 1N ∑N c=1wc.

Review summaries:
This paper analyses the well-known and well-studied MAML algorithm and raises two key observations. First one is requiring a high number of inner-loop updates, and the second one is the variation in the meta-test accuracy when permuting the indices of the classes. Then it proposes to simply meta-train a single vector and duplicate it as initialization for the classifier head to make MAML permutation-invariant and improve its generalization performance on meta-test tasks. 

This paper is an interesting "re"search on a classical meta-learning method MAML. It shows that both the inner update iterations and task label assignment have a clear influence on the performance of MAML. It raises some questions and uses a set of in-depth experiments to draw conclusions. 

The authors study the MAML algorithm and propose two ways to improve the performance. First, they study the number of inner gradient steps. Second, they look at the permutation of labels when learning a new task which ideally should not make any difference. However, it seems that during testing this could lead to different test performances on a target task. Based on this observation, they propose a novel and very interesting solution to share the weights of the classification layer (which they call UNICORN MAML).

The paper mainly investigates the effect of permutation in the class label assignment in the tasks for the MAML algorithm. First, the authors show that MAML requires a higher number of inner loop updates than what is commonly used. Then, they show that MAML is sensible to the permutation of the class labels in the tasks and experimented with diverse methods to alleviate this problem. Finally, they proposed Unicorn-MAML, a modification of the MAML algorithm that learns a single weight vector for the classifier layer, to make the model permutation invariant. They show through experiments on two different datasets (mini-ImageNet and tiered-ImageNet) that it achieves or outperforms state-of-the-art performance.