How to Train Your MAML to Excel in Few-Shot Classification
Authors: Han-Jia Ye
Wei-Lun Chao
Venue: ICLR 2022 Poster (2022)
Keywords: meta-learning
few-shot learning
classification
MAML

Abstract: Model-agnostic meta-learning (MAML) is arguably one of the most popular meta-learning algorithms nowadays.
Nevertheless, its performance on few-shot classification is far behind many recent algorithms dedicated to the problem. In this paper, we point out several key facets of how to train MAML to excel in few-shot classification. First, we find that MAML needs a large number of gradient steps in its inner loop update, which contradicts its common usage in few-shot classification. Second, we find that MAML is sensitive to the class label assignments during meta-testing. Concretely, MAML meta-trains the initialization of an $N$-way classifier. These $N$ ways, during meta-testing, then have "$N!$" different permutations to be paired with a few-shot task of $N$ novel classes. We find that these permutations lead to a huge variance of accuracy, making MAML unstable in few-shot classification. Third, we investigate several approaches to make MAML permutation-invariant, among which meta-training a single vector to initialize all the $N$ weight vectors in the classification head performs the best. On benchmark datasets like MiniImageNet and TieredImageNet, our approach, which we name UNICORN-MAML, performs on a par with or even outperforms many recent few-shot classification algorithms, without sacrificing MAML's simplicity.

Published as a conference paper at ICLR 2022
HOW TO TRAIN YOUR MAML TO EXCEL IN FEW-SHOT CLASSIFICATION
Han-Jia Ye
Wei-Lun Chao The Ohio State University
State Key Laboratory for Novel Software Technology, Nanjing University
ABSTRACT
Model-agnostic meta-learning (MAML) is arguably one of the most popular meta- learning algorithms nowadays. Nevertheless, its performance on few-shot classifi- cation is far behind many recent algorithms dedicated to the problem. In this paper, we point out several key facets of how to train MAML to excel in few-shot classifi- cation. First, find that MAML needs large number of gradient steps in its inner loop update, which contradicts its common usage in few-shot classification. Second, we find that MAML sensitive the class label assignments during meta-testing. Concretely, MAML meta-trains the initialization of an N-way classifier. These N ways, during meta-testing, then have " "N!" different permutations be paired with a few-shot task of N novel classes. We find that these permutations lead to a huge variance of accuracy, making MAML unstable in few-shot classification. Third, we investigate several approaches to make MAMI permutation-invarian which meta-training a single vector to initialize all the N weight vectors the classification head performs the best. On benchmark datasets like MinilmageNet and TieredImageNet. our approach, which we name UNICORN- MMMM, performs par with or even outperforms many recent few-shot classification algorithms, without sacrificing MAML's simplicity.
among
INTRODUCTION
Meta-learning is sub-field of machine learning which attempts search for the best learning strategy as the learning experiences increases (Thrun & Prati [012 Lemke et al. (2015). Recent years have
witnessed an abundance of new approaches on meta-learning (Vanschoren) 2018 Hospedales et al.)
2020), among which model-agnostic meta-learning (MAML) (Finn et al. [017]] Finn 2018)) is one of the most popular algorithms, owing its "model-agnostic" nature to incorporate different model architectures and its principled formulation to be applied to different problems. Concretely, MAML aims to learn a good model initialization (through the outer loop optimization), which can then be quickly adapted to novel tasks given few examples (through the inner loop optimization).
However, in few-shot classification (Vinyals et al,, 2016; Snell al. 2017) which many meta-learning algorithms are dedicated to, MAML's performance has been shown to fall far behind (Wang et al. 2019 Chen et al. 2019: Triantafillou et al. 2020).
In this paper, we take a closer look at MAMI on few-shot classification. The standard setup involves two phases, meta-training and meta-testing, in which MAMI. learns the model initialization during meta-training and applies it during meta-testing. In both phases, MAML receives multiple NNwwyy K- shot tasks. Each task is an -cllass classification problem provided with K labeled support examples per class. After the (temporary) inner loop optimization using the support examples, the updated model from the initialization is then evaluated on the query examples of the same N classes. The loss calculated on the query examples during mete- mete-training is used to optimize the meta- pettapprsseterss (i.e., the model initialization) through the outer loop. For consistency, we mainly study the scenario where meta-training and mela-testing use the same number of gradient steps the inner loop optimization.
More specifically, what MAMI. learns for few-shot classification is the initialization of an N-class classifier. Without loss of generality, we denote classifier by ÿ arg max _ (xx), where N fo is the feature extractor on an example æ; [ are the weight vectors in the linear classification head. We use 0 represent the collection of meta-parameters (Φ, un," ".. wx)

---

Published as a conference paper at ICLR 2022
Vanilla MAMLL
A few sewtshat
Another fow-shot task
Feature extractor
Class: 1 ("unicom"")
Class: ("bee")
Class: N ("bee")
Class: ("unicarn")
w
Permutation
(a)
Unicom MAMIL (meta parameters 61
Initalication the inner-loop optimization
few-shot task (support set]
Festure extractor
Linear clastifler
Feoture extractor
Class: ("unicorn")
W1
W
f
W
Class.N ("bee")
WN
(b)
Figure I: The problem permutations in label assignments, and the illustration of UNICORN-MAML.
(a) A vanilla MAML learns the initialization of Φb and (wa]e= (i.e,, the N weight vectors). Each of (we } paired with the corresponding class label c [M] of few-shot task. A few-shot lask, however, may consist of the same set of semantic classes but in different permutations of class label assignments, leading to larger variance in meta-testing accuracy. (b) In contrast, our UNICORN-MAML, besides learning Φ. learns only a single weight vector w and uses it to initialize all the N weight vectors [w-]c=1 at the beginning of the inner loop. That is, UNICORN- MAML directly forces the learned model initialization to be cnttuutttionnnivaiinnnt
Our first observation is that MAML needs a large number of gradient steps in the inner loop. For example, on MindmmageNet (Vinyals et al. 2016) and TigredmmageNeet (Ren et al.. 2018a), MAML's accuracy improves along with the increased number of gradient steps and achieves the highest around 15 ~ 20 steps, which are much larger than the conventional usage of MAML (Antoniou et al. 2019). We attribute this to the behavior of the model initialization learned from mutually mutually-exclusig excllusie tasks (Yin) et al. 2020), which, without any further inner loop optimization, performs at the chance level (Lee, 10011 accuracy) on query examples, not only for the meta-testingg tasks but also for the meta-training tasks. In other words, the initialized model needs many gradient steps to attain a high accuracy.
Our second observation is that MAML is sensitive to the permutations of class label assignments during meta-testing. Concretely, when an N-way K-shot task arrives, MAML pairs the learned initialization of we with the corresponding class label e e [NV of that task. The issue resides in the "meaning" of in a task. In the standard sctup, each N-way task is created by drawing N classes from a bigger pool of semantic classes (e.g.. "dog". "cat", "bird", etc.), followed by a random label re-assignment into E (1..... N}. In other words, the same set of N semantic classes can be labeled totally differently into (1.... .N3 and thus be paired with [ur differently. Taking a five-way task for example, there are 5! = 120 permutations to pair the same set of five semantic classes to the linear classification head. some of them, a class "dog" may be assigned to c 1; in some others, it may be assigned to + 1. While this randomness has been shown crucial in meta- meta-training to help MAML prevent ever-fitting (Rajendran et al.) 2020 Yao et al.) 2027 Yin et al.) 2020, we find that it makes the mela-testing phase unstable. Specifically, different permutations can lead to drastically different meta-testing accuracy - on average, the best permutation for each five-way one-shot task has- 15% higher accuracy than the worst permutation, on both datasets.
Building upon this observation, we investigate multiple approaches to make MAML permutation- invariant, either in the meta-testing phase alone or in both phases. We find that simple solution
meta-truining only a single vector w and using it to initialize the N linear classifiers {w_) c=1
X
performs the best. We name this approach UNICORN MAMM, as illustrated in Figure (b). Concretely, at the beginning of the inner loop, MIICORN-MAML duplicates w into w., Vc E []]
After the inner loop optimization, the meta-gradient with respect to (wa) are aggregated to update w in the outer loop (during meta-training). This design not only makes UNICORN-MAML permutation-invann invariinnt,, but also ensures that no single model can solve all tasks at once without inner loop optimization, to prevent memorization over-fitting (Yin et al. 2020). On MinilmageNet (Vinyals) et al. 2016), TeeredImageeet (Ren et al.) 2018a), and CUB datasets (Wah et al., 2011). UNICORN- MAML performs a par with or even outperforms many recent few-shot classification algorithms, while preserving the simplicity of MAML without adding any extra network modules or learning strategies. Our code is available https: // http///tttttutgtt//// com Han- Jia UNT CORN-MAMT.
2

---

Published as a conference paper at ICLR 2022
2
MAML FOR FEW-SHOT CLASSIFICATION
2.1 PROBLEM DEFINITION
The goal of few-shot classification is to construct classifier using limited labeled examples. The challenge is the potential over-fitting or poor generalization problem. Following (Vinyals et al. 2016). define few-shot classification problem an N-way K-shot task, which has N classes and each class has K labeled support examples. We denote the labeled support set by S ( 31 ) N
each (æ:; yi) is a pair of an input (e.g., . image) and a class label, where y/ E {1, 2, ... , [=] K The value of K is small, e.g., K each task is associated with a query set 2, which is composed of examples of the same N classes.
[ K 5. To evaluate the quality of the resulting classifier,
The core idea of meta-learning for few-shot classification is to sample few-shot tasks T = (8, Q) from a set of "base" classes, of which we have ample examples per class. Meta-learning then learns the ability of "how to build a classifier using limited data" " from these tasks. After this meta-training phase, we then proceed to the meta-testing phase to tackle the true few-shot tasks that are composed of examples from "novel" classes. By default, the "novel" and "base" classes are disjoint. It is worth noting that the total number of "base" (and "novel") classes is usually larger than N (see subseectoo 2.3). Thus, to construct an N-way K-shot task in each phase, one usually first samples N classes from the corresponding set of classes, and randomly re-labels each sampled class by an index c E [Av. This randomness results in the so-called mutually-excllssin tasks (Yin et al. 2020). In this paper, we will use base (novel) and meta-training (meta-testing) classes interchangeably.
2.2 MODEI META-LEARNING (MAMI)
As introduced in incction I, MAML aims to learn the initialization of an Naw classifier, such that when provided with the support set S of an N-way K-shot task, the classifier can be quickly and robustly updated to perform well on the task (i.e., classify the query set Q well). Let us denote a classifier by ò he (z) = arg max w. (xx), where fø is the feature extractor, {w.) are the weight vectors of the classification head, and 0 {Φ. w1. wNr collects the parameters of both. MAML evaluates he on S and uses the gradient to update O into 0', so that he can be applied to Q. This procedure called the inner loop optimization, which usually takes M gradient steps.
0' + 0
for m E [M] do
0' = 0 V (')
Here, 0(),0') = Liaries ((hor (ax), v) is the loss computed on examples of S and a is the learning rate (or step size). The cross-entropy loss is commonly used for f. As suggested in the original MAML paper (Finn et al. (2017) and (Antoniou et al. 2019), M is usually set to a small integer (e.g.. $ 5). For ease of notation, let us denote the output O' after M gradient steps by 0' InLoop(S, M).
To learn the initialization 0, MAML leverages the few-shot tasks sampled from the base classes. Let us denote by PTT) the distribution of tasks from the base classes, where each task is pair of support and query sets (S, Q). MAML aims to minimize the following meta-training objective w.r.t. 0:
E ((2,6') (s.2)~p(T)
[(2, InLop(SS M)).
(S. (T.
Namely, MAML aims to find a shared 0 among tasks, which, after inner loop updates using S, can lead to small loss on the query set Q. (We add the subscript s to O' to show that O's depends on S.) To optimize Equation MAML applies stochastic gradient descent (SGD) but at the task level. That is, at every iteration, MAML samples a task T = (S, Q) and computes the meta meta-gradient w.r.t. 0: VoL(Q, 0's) = VoC(Q, InLoop(S, 0, M)).
(3)
In practice, one may sample mini-batch of tasks and compute the mini- mini-batch meta- gratiit w.r.t to optimize 0. This SGD for is known as the outer loop optimization for MAML. It is worth noting that calculating the gradient in Equation can impose considerable computational and memory burdens because it involves a gradient through a gradient (along the inner loop but in a backward order) [Finn et al. 2017). Thus in practice, it is common to apply the first-order approximation (Finn et al. 2017 Nichol et al. 2018), i.e., VoL(Q, 0's) ~ V Vog ((O, 0'g).
For additional related work on meta-learning and few-shot learning, please see Appendix A
3

---

Published as a conference paper at ICLR 2022
2.3 EXPERIMENTAL SETUP
As our paper is heavily driven by empirical observations, we first introduce the three main datasets we experiment on, the neural network architectures we use, and the implementation details.
Dataset. We work on MinelmageNet (Vinyals et al. 2016). Iageed (Ren et al. (0188) Sa and CUB datasets (Wah al., (201T) MinilmageNett contains 100 semantic classes; each has 600 images. Following (Ravi & Larochelle 2017), the 100 classes are split into meta-traine traaiingg/aaiiaationon sets with 64/16/20 overlaapa)pped) classes, respectively. That is, there are 64 base classes and 20 novel classes; the other 16 classes are used for syyper-paramete tuning. TieredImageNet (Ren) (et al., 2018a) has 608 semantic classes, which are split into the three sets with 3 60 classes, respectively. On average, each class has 1,300 images. CUB (Wah et al. 2011) has 200 classes, which are split into the three sets with 200/50/50 classes following (Ye et al. 2020a All images are resized to 84 x 84, following (Lee et al. 2019: Ye et al., 2020a).
Training and evaluation. During meta-training, meta-validation, and meta-testing, we sample N- way KSshot tasks from the corresponding classes and images. We follow the literature (Snell et al. 2017 Vinyals et al. 2016) to study the five-way one-shot and five-way five-shot tasks. As mentioned in subsecctio 2.1 every time we sample five distinct classes, randomly assign each of them an
index e E N During meta-testing, we follow the evaluation protocol in (Zhang et al. 2020 Rusu al.
2019 Ye et 2020a) to sample 10, 000 tasks. In each task, the query set contains 15 images
per class. We report the mcan accuracy (in %) and the 95% confidence interval.
(He et al. 2016) architecture for fo subsectioo 2.2). which has wider widths and Dropblock modules (Ghiasi et al. 2018). We note that many recent few-shot learning algorithms use this backbone. We also follow the original MAML (Finn et al.) 2017) to use 4-layer convolutional network (ConvNet) (Vinyals et al. 2016).
Model architecture. We follow (Lee et al. 2019) to use a ResNet- 12
Implementation details. Throughout the paper, for simplicity and consistency, we use
the first-order approximation for calculating the meta vreta-gradient in the outer loop;
the same number of gradient steps in the inner loop during meta-training and meta-testing;
the weights pre-trrined on the entire meta-training set to initialize Φ. following the recent prac-
tice (Ye et al. 20200a] Rusu al., 2019 Qiao et al. 2018). We note that in meta-training we still optimize this pre-trained & in the "outer" loop to search for better initialization for MAML.
and we select them on the meta-validation set. Specifically, for the outer loop, we learn with at most 10, 000 tasks: we group every 100 tasks into an epoch. We apply SGD with momentum 0.9 and weight decay 0.0005. We start with an outer loop learning rate 0.002 for ConvNet and 0.001 for ResNet-12, which are decayed by 0.5 and 0.1 after every 20 epochs for ConvNet and ResNet- 12, respectively. For the inner loop, we have to set the number of gradient steps M and the learning rate a (cf. [Equation I). We provide more details in the next section.
MAML has several
MAML NEEDS A LARGE NUMBER OF INNER LOOP GRADIENT STEPS
We find that for MAML's inner loop, the number of gradient updates M (cf. Equation I is usually
selected within a small range close to 1, e.g. M E [1, 5) (Antoniou et al. 2019). At first glance, this makes sense according to the motivation of MAML (Finn et al. 2017) with a small number of gradient steps, the resulting model will have a good generalization performance.
In our experiment, we however observe that it crucial to explore a larger MI Specifically, we consider M E [1, 20) along with E [10- 10% We plot the meta-testing accuracy of five-way one-shot tasks on the three datasets in Figure 2F] using both ResNet and ConvNet backbones. We find that MAML achieves higher and much more stable accuracy (w.rt. the learning rate) when M is larger, e.g., larger than 10. Specifically, for MinilmageNet with ResNet, the highest accuracy 64.42% is obtained with M = 15, higher than 62.90% with M = 5; for TieredImageNet with ResNet, the highest accuracy 65. 72% is obtained with M = 15, higher than 59.08% with M = 5. As will be seen in section 6. these results with larger M are already close to the ofate-of-the-art performance.
I We reiterate that for simplicity and consistency we apply the same M in meta-training and meta-testing. We tune hyper-parameters on the meta-validation set and find that the accuracy there reflects the meta-testing accuracy well. We show the meta-testing accuracy here simply for a direct comparison results in the literature.
4

---

Published as a conference paper at ICLR 2022
10
10
CLR
61
61
15
77AT
20
5E-3
MinilmageNet,, ResNet
Iaared ResNet
CUB, ResNet
5
53
10
48
10
82
SUM
51
5E-2
1E-1
MinilmageNet, ConvNet
TieredImageNes ConvNet
CUB, ConvNet
Figure 2: Heat maps of MAML's five-way one-shot accuracy on MinilmageNet. TieredImageNet, and CUB w.r.t the inner loop learning (X-axis) and the number of inner loop updates M (y-axis). For each heat map. we set accuracy below a threshold to fixed value for clarity; we denote best accuracy by black box.
Accourag 1o) 2.2 22833882
Piddy Transigg
Peta-raainn
Mode Testing
Mata Tosteg
Meta-Trrie
Tell-Tering
Korations
Rerations
MinilmageNet, R VeeredImageNet, R
CUB,
MinilmageNet, C TieredImageNet,
CUB, C
Figure 3: We plot the change of the five-way one-shot classification accuracy (on the query set), averaged over 10,000 tasks sampled from either the meta veta-training (red) meta-testing classes (bluc), along with the process of inner loop updates, using the best model initialization learned by MAML. C: ConvNet; R: ResNet.
To analyze why MAML needs large M, plot the change of classification accuracy along with the inner loop updates in Figure 3] Specifically, we first perform meta- veeta-trinnng using the M value selected by vntaa-vaiiattin, for each pair of dataset and backbone. We then analyze the learned model initialization on few-shot tasks sampled from the meta-training and meta-testing classes, by performing 30 inner loop updates using the support set and reporting the accuracy on the query set. We conduct the same experiments for five-way five-shot tasks in Appendix D.
We have two observations. First, general, the more inner loop updates we perform for few-shot task, the higher the accuracy is, no matter if it is meta-training meta-testing task. This trend aligns with the few-shot regression study in (Finn et al.) 2017). Second, before any inner loop update, the learned initialization has 20% accuracy average, i.e., the accuracy by random classification. Interestingly, this is the case not only for meta- meta-testing testing tasks but also for meta- training tasks, even though the learned initialization does contain the classification head (w.) ) This explains why larger number of inner loop gradient steps is needed: the learned initialized model has to be updated from performing random predictions achieving a much higher classification accuracy.
We attribute the second observation to the random class label assignments in creating few-shot tasks (cf. subsection 2. and subsection 2.3), which make the created tasks mutually-exclusse (Yin et al.) 2020) - i.e., single model cannot solve them all at once before inner loop optimization. Concretely, for feo-shot task of a specific set of N semantic classes (e.g., ["dog". "cat". "bird")), such randomness can turn it into different tasks from MAML's perspective. For instance, the class "dog" may assigned to e 1 and paired with un at the current task, but when it is sampled again. For five-way task, the same set of five semantic classes can be assigned to {1...... 5} via 120 (i.e., 5!) different permutations. As a result, we directly apply the learned initialization of MAML without inner loop updates, the accuracy on few-shot tasks of the same set of semantic classes (but in different permutations) can cancels each other out. (Please see subsection C. for details.) Besides, since the randomness occurs also in meta-training, each w. will be discouraged to learn specific knowledge towards any semantic class (Rajendran et al. 2020 Yao et al. 2021 Yin et al. 2020), hence producing accuracy at the chance level even meta-training tasks.
and paired with uz
5

---

Published as a conference paper at ICLR 2022
30
30
30-
20
20
20
36
11
(a) MiullageNet, I-Shot
(b) MintlmagNNer, 5-Shot
(c) TeredImageeet I-Shot
(d) Net S-Shot
Figure 4: The histogram of the 120 meta-testing accuracy (averaged over 2, 000 tasks), each corresponds to a
specific position in the sorted list of each task's 's accuracy among 120 permutations. The x-axis corresponds to accuracy (range); the y-axis corresponds to counts. The backbone is ResNet-12.
Algorithm 1:
Evaluation of the effect of class label permutations on meta-testing tasks.
Given the learned initialization by MAML.
for t E (1..... 2000} do
Sample meta-testing task T (S, Q) and initialize an accuracy vector JR" for p E {1, -1.....,, 120} do
Shuffle class lahels with a specifie permutation क : [N] + [N]; ie., (S, Q) becomes (Sa, 2+) Update 0 to get 0' = InLoop(S., 0, M), evaluate 0' on 2x, and record the accuracy into as end
Sort values in the accuracy vector a, in the descending order and get a
end
Average 2000 tasks by Σ, a
Related work. Several following-up works of MAML, e.g. (Hsu et al. 2019). use different numbers of inner loop steps in meta-training (e-g., M make M equal in the two phases for consistency, and provide detailed analysis of why MAML needs large M value. On large-scale tasks beyond few-shot learning, (Shin al. (2021) also found the necessity of a larger number of inner loop steps, but from different perspective than ours.
5) and meta-testing (e.g,, M 20 50). We
MAML IS SENSITIVE TO THE LABEL PERMUTATIONS IN META-TESTING
The randomness in class label assignments raises an interesting question: do different permutations result in different meta-testing accuracy after inner loop updates? ? More specifically, if { [we)mm N are paired with the N classes differently, will the updated model after the inner loop perform differently?
To answer this question, we conduct a detailed experiment: Algorithm T summarizes the procedure. We focus five-way one/five-shot tasks MiniImageNet and Iagredm using the ResNet backbone. For each task type and dataset combination, we first meta-train the model initialization using MAML, and then evaluate the learned initialized on 2, 000 meta-testing tasks. For each task, there are 120 permutations; each permutation, after the inner loop, would likely lead to a different model and query set accuracy. We sort the 120 accuracy for each task, and take average for each position in the sorted list over 2, 000 tasks. This results in 120 averaged accuracy, each for specific position in the sorted list. Specifically, the highest accuracy corresponds to the case that each task cherry-picks its best permutation according to the query set accuracy after inner loop optimization.
We show the histogram of the 120 average meta-testing accuracy in Figure 4. There exists a huge vari- ance. Specifically, the best permutation can be 15%/8% higher than the worst in one/five-shot tasks. The best permutation is also much higher than vanilla MAML's results (from /ection 3). which are 64.42% /66/8834444/6.... 44%%/5..22%%/ 72%7744.77449 %. corresponding to the four sub-figures from left to right in Figure 4| What's 's more, the best permutation can easily achieve state-of-the-arr accuracy (see /section 6).
Of course, far we find the best permutation via accuracy - so it is like an upper bound. However, we can find the best permutation without looking at the query sets' labels make MAML intimuutationnivvriinn we may practically improve MAML.
by looking the meta-testing
Related work. We note that we investigate the effect of the permutations of class label assignments differently from (Rajendran et al. 2020 Yao et al.) 2021 Yin et al. (020).. There, the permutations lead to mutually-excluse
meta- trainin tasks, which help prevent MAML from over-fitting. Here,
we look at the meta-testing tasks, and permutations result in huge variance of meta-testing accuracy.

---

Published as a conference paper at ICLR 2022
Table I: The meta-testing accuracy over 2,000 tasks given different permutation selection strategies.
MinlmageNeet
Tagreel
Select the permutation by
1-Shot
5-Shot
1-Shot
5-Shot
None
64.42 1 0.20
83.44 1 0.13
65.72 = 0.20
84.37
0.16
Initial Support Acc
64.42 1 0.20
83,95 1 0.13
65.06 0.20
84.32 0.16
Initial Support Loss
64.42 1 0.20
83,91 + 0.13
65.42 = 0.20
84.23 0.16
Updated Support Acc
64,42 = 0.20
83.95
= 0.13
65.01 = 0.20
84.37 + 0.16
Updated Support Loss
64.67 = 0.20
84.05 = 0.13
65.43 0.20
84.22 + 0.16
Table 2:
Ensemble over updated models of different permuta- tions. (The confidence interval is omitted due to space limit.)
Table 3: We average the top-layer classifiers and expand it to N-way during meta-testing.
Mini
Vanilla
Full Rotated
Tiered
Vanilla
Full
Rotated
MindImageNet
TieredImageNet
1-Shot 64.42
65.50
65.37
I-Shot
65.72
66.63 84.83 84.81
66.68
1-Shot
64.40 1 0.21
66.24 1: 0.24
5-Shot 83.44
84.43
84.40
5-Shot
84.37
5-Shot
84.24 + 0.13
84.52 0.16
5 MAKING MAML PERMUTATIIN--IVARIANTT DURING META-TESTING
We study approaches that can make MAMI. permutation-invatiee during meta-testing. That is, we take the same learned initialization O as in section without changing the meta-training phase.
We first investigate searching for the best permutation for each task. As we cannot access query sets' labels, we use the support sets' data to which permutation, either before or after inner loop updates (less practical), leads to the highest accuracy or smallest loss on the support set. Table summarizes the results: none of them leads to consistent gains. We hypothesize two reasons. First, due to mutually-exclusse the learned by MAML would produce chance-level predictions before updates (see Appendix C). Second, after updates, the support set accuracy quickly goes to 100% and is thus not informative.
proxy. We choose the best permutation according
tasks in meta-training,
In stead of choosing one from many, we further explore taking ensemble (Breiman 1996 Zhou 2012 |Detterich (000) over the predictions made by updated models of different permutations. We note that this makes MAML make the ensemble process clear, we permute the weight vectors in {wa]a² rather than the class label assignments in a task: the two methods are equivalent but the former is easier for aggregating predictions. We study two variants: (a) full permutations (i.e., 120 of them in five- way tasks). which is intractable for larger N; (b) rotated permutations, which rotates the index c in { [w I leading to N permutations. Table 2) shows the results ensemble consistently improves MAML. Even with the rotated version that has much fewer permutations than the full version, the gains are comparable.
but inevitably needs more computations. To
We emphasize that our focus here is make MAML cerrmutattion-ivvriinn in meta meta-testing, not explore and compare all potential ways of performing ensemble on MAML.
We further explore efficient approach make MAML permutation--nvarrint,, which is manipu- late the learned initialization of (w,) [ c=1- Concretely, MAMI. is sensitive to the permutations in class assignments because w. A w for c * c' One method overcome this make w. we' during meta-testing. Here, we investigate re-initializing each w. by their average in meta-testing: W. + N Σm= W... By doing so, no matter which permutation we perform, the model after inner loop optimization will be the same and lead to the same query set accuracy. Table 3 summarizes the results; this approach improves vanilla MAML (see Table 2) in three of the four cases.
At first glance, this approach may not make sense since the resulting [w.}_ before inner loop updates, are identical and simply make random predictions. However, plccllpppeeeeeeee note that even the original [wr) 1C have an averaged chance accuracy (cf. Figure 3). In Apppnddix E we provide an explanation of this approach by drawing an analogy to dropout (Srivastava et al.] 2014). In short, in meta-training, we receive a task with an arbitrary permutation, which can be seen as drawing a permutation random for the task. In meta-testing, then take expectation over permutations, which essentially lead to the averaged we
Thht is, we consider re-assign w. We(4 mod N941, where ? E [N].

---

Published as a conference paper at ICLR 2022
Table 5-Way 1-Shot and 5-Shot classification accuracy and 95% confidence interval on MinilmageNet and TieredImageNet (over 10,000 tasks), using ResNet-12 12 as the backbone. i: MAML with 5 inner loop steps in meta- veetattrining/testingg *: we carefully select the number of inner loop steps, based on the meta-validatie set.
Datasct +
MinilmageNet
TieredImageNett
Sctups
1-Shot
5-Shot
1-Shot
5-Shot
ProtoNet (Snell et al. 2017
62.39 + 0.20
80.53 1 0.20
68.23 1 0.23
84.03 1 0.16
ProtoMAML Tnaatahllou et al 2020
64.12 = 0.20
81.24
0.20
68.46 = 0.23
84.67 = 0.16
MetaOptNet (Lec ct al. 2019)
62.64 = 0.35
78.63 = 0.68
65.99 = 0.72
81.56 = 0.53
MTL+E3BM (Sun ot 2019
63.80 0.40
80.10 0.30
71.20 0.40
85.30 = 0.30
RFS-Distill (Tian cl al. 2020
64.82 = 0.60
82.14 1:14:0.433 0.43
69.74 + 0.72
84.41 = 0.55
DeepEMDD (Zhang et al.
2020
65,91 + 0.82
82.41 + 0.56
71.52 + 0.69
86.03 + 0.49
MATE-MetaQpt (Chen al.
(020
62.08 + 0.64
78.64
71.16 0.87
86.03 + 0.58
DSN-MR (Simon et al.
2020
64.60 = 0.72
79.51
0.50
67.39
0.82
82.85 = 0.56
FEAT Yee al. 2020a
66.78 = 0.20
$2.05
70.80
0.23
84.79 = 0.16
MAML (5-Step') MAML (our reimplementation")
62,90 = 0.20
80.81 + 0.14
59.08 = 0.20
80.04 - 0.16
64,42 = 0.20
83,44 0.14
65.72 = 0.20
84.37 = 0.16
UNICORN-MAMI.
65.17 = 0.20
84.30 + 0.14
69.24 0.20
86,06 = 0.16
6 UNICORN-MAML LEARNING A SINGLE WEIGHT VECTOR
The experimental results in section 5 are promising: by making MAML permutation-innaaiaa in meta-testing, we can potentially improve vanilla MAML. While ensemble inevitably increases the computational burdens, the method by manipulating the learned initialization of {w | keeps the same run time as vanilla MAML. In this section, we investigate the latter approach further. We ask:
If we directly learn a single weight vector w to initialize (we) Je= in meta-training, making the inner loop optimization in meta- meta-training and meta-testing consistent and both permutation invariinn, can we further improve the accuracy?
Concretely, we redefine the Icarnable meta centa-prrmmterss θ of MAML, which become 0 = { w) where & is for the feature extractor. We name this method UNICORN-MAMEL only a single weight vector w in the classification head. The inner loop optimization and outer loop optimization of UNICORN- MAML very much follow MAML, with some slight changes.
we meta-train
Inner loop optimization: At the beginning, w is duplicated into (w, = w That is, we w initialize every w., Vc E [N]. These {w, w m then undergo the same inner loop optimization process as vanilla MAML (cf. [Equation T).
Outer loop optimization: Let denote the updated model by 0' ( w'y). To perform the outer loop optimization for w in mnta-trriiinn,, we collect the gradients derived from the query set Q. Let denote by V V., [w, 0¹) the gradient w.r.t. the initial w. (cf. /ubsection 2.2
Since w. is duplicated from w. we obtain the gradient w.r.t. w by ΣCELN) V C(Q, 0").
Table 4 summarizes the results of MIICORN-MAMEL MAML, and many existing few-shot learning algorithms. MACCORN--AME consistently improves MAML: on MinilmageNet, UNICORN- -MAML has a 0.7% gain on one-shot tasks and 0.8% gain on five-shot tasks; on /iereedmageNet,, UNICORN- MAML has significant improvements (a 3.5% gain on one-shot tasks and 1.6% gain on five-shot tasks). More importantly, UNICORN-MAME performs on par with many recent algorithms on one-shot tasks, and achieves the highest accuracy on five-shot tasks. Specifically, compared to ProtoMAML and MetaOptNet, which are both permutation permutation-invariiat invariant variants of MAML (see the related work paragraph at the end of this section), MNNCORN-MAML notably outperforms them.
Other results. We evaluate ANICORN-MAME backbone on MinilmageNett in the appendix. UNICORN- MULCORN-MAML achieves promising improvements.
CUB (aah et al. 2011) and use the ConvNet
Why does UNICORN-MAME work? The design of UNICORN- MAML ensures that, without using the support set to update the model, the model simply performs at the chance level on the query set. In other words, its formulation inherently helps prevent memorization over-fitting (Yin et al.) 2020).
Embedding adaptation is needed. We analyze UNICORN- UNICORN-MAML in terms of its inner loop updates during meta-testing, similar Figure This time, we also investigate updating or freezing the
8

---

Published as a conference paper at ICLR 2022
ACCCUL
Melo test Accourg
MAN 1022 Accourany
Freeze
Freece
Freeze &
Freece UMACORNNMA
UMCORN MAVL
UMCORN MAVL
UMICORN MAVL
Inner Iterations
Inner Inneritersttons
Inner Itarations
Inner Iterations
(d) Tireedimageeer S-Shot Figure 5: The change of meta-testing accuracy (over 10,000 tasks) along with the process of inner loop updates based on UNICORN-MN UNILORN-MAML. We investigate updating/freezing the feature extractor Φb
(n) MivllmageNet, I-Shot
MinllaggNee, S-Shot
(c) TieredImageNet, I-Shot
Table 5: 5- Way 10/20/30/50- Sootoo classification accuracy on MinilaageNett over 10,000 tasks with a ResNet backbone. (The confidence interval is omitted due space limit.)
Table 6: B-Way 1/5-Shot classification accu- racy on CUB based on the best learned model with ResNet backbone from MinilmageNet.
Multi-Shot
10-shot 30-shottt---hOOO---0
MinilmageNeet CUB
1-Shot5-Shot
SiopleShot Wang al. 2019
84.89
86.91
87.53
88.08
Bascling++ Chen al.
2019)
50.37
73.30
ProtoNet Snell 2017
82.83
84.61
85.07
85.57
ProtoNet (Sell et 2017
50.01
72.02
FEAT
Ye al. 2020a
85.15
87.09
87.82
87.83
Neg-Cosine Liu et al.
2020
47.74 69.30
MAML (our implementation)
88.08
90.23
91.06
92.14
MAML (our implementation)
51.25 73.86
UNICORN- UNICORN-MAML
88.38
90.96
91.96
92.86
UNICORN MAML
51.80 75.67
feature extractor 6. Figure 5) shows the results on five-way one- and five-shot tasks on both datasets. ENICORN-MAMIL is accuracy again begins with 20% but rapidly increases along with the inner loop updates. In three out of four cases, adapting the feature extractor @ is necessary for claiming higher accuracy, even if the backbone has been well pre-trained, which aligns with the recent claim by Arnold & Sha (2021): "Embedding adaptation is still needed for few-shot learning."
Experiments on larger shots and transferability. MAML or similar algorithms that involve a bi-level optimization problem (i.e., inner loop and outer loop optimization) are often considered more complicated and computationally expensive than algorithms without bi-level optimization, such as ProtoNet (Snell et al.] 2017) or SimpleShot (Wang et al. 2019). Nevertheless, the inner loop optimization does strengthen MAML's adaptability during meta-testing, especially (a) when the meta-testing tasks are substantially different from the meta-training tasks (e.g., meta-training using MinilmageNet but meta-testing to 10 ~ 50). In Table 3] and Table 6, we conduct further experiments to justify these aspects.
CUB) (b) when the number of shots increases (e.g., from 1 ~ 5
Related work. We note that some variants of MAML are permutation-invariann even though they are not designed for the purpose. For example, LEO (Rusu et al. 2019) computes class prototypes (i.e, averaged features per class) encode each class and uses them to produce task-specific initialization. However, it introduces additional sub-networks. MetaOptNet (Lee et al. 2019) performs inner loop optimization only on ( the initialization and hence the permutations. This method, however, has a high computational burden and needs careful hyper-parameter tuning for the additionally introduced regularizers. Proto-MAML (Triantafillou et al. (0200) initializes the linear classifiers [weJ with the prototypes, which could be permutation- invariant but cannot achieve accuracy as high as our UNICORN- MAML.
(w.). ) c= (till convergence), making it convex problem which is not sensitive
DISCUSSION AND CONCLUSION
There have been an abundance of "novel" algorithms proposed for few-shot classification (Hospedales) al. 2020, Wang et al.) 2020)). In these papers, MAML (Finn al., 2017) frequently considered as baseline, but shows inferior results. This raises our interests. Is it because MAML is not suitable for few-shot classification, or is it because MAMI. has not been applied appropriately to the problem?
We thus conduct series of analyses on MAML for few-shot classification, including hyper-parameter tuning and the sensitivity to the permutations of class label assignments in few-shot tasks. We find that by using large number of inner loop gradient steps (in both meta-training and meta-testing), MAML can achieve comparable results to many existing algorithms. By further making MAML permutation- invariant to the class label assignments, existing algorithms on five-shot tasks, without the need to add extra sub-networks. We hope that UNICORN-NN MAMM.. could serve strong baseline for future work in shw-shot classification.
present UNICORN- MAML. which outperforms many
9

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
This research is supported by National Key R&D Program of China (2020AAA010940)), CCF-Baidu Open Fund (2021PP15002000) NSFC (61773198, 61921006, 62006112, 62176117), Collaborative Innovation Center of Novel Software Technology and Industrialization, NSF of Jiangsu Province (BK20200313), NSF IIS-2107077, NSF DAC-2118240, NSF OAC-2112606, and the OSU GI Development funds. We are thankful for the generous support of computational resources by Ohio Supercomputer Center and AWS Cloud Credits for Research. We thank Sébastien M.R. Arnold (USC) for helpful discussions.
REPRODUCIBILITY STATEMENT
The details of datasets, model architectures, hyper-parameters, and evaluation metrics are described in subsection 2.3 |] Appendix BJ Our code is available at / thub. comb//co/////mmmommmooo//o// Han-Jia a/ UNICORN-MER MAML including the initialization weights pre-trained on the meta meta-training set and the checkpoints.
REFERENCES
Arman Afrasiyabi, Jean-François Lalonde, and Christian Gagné. Associative alignment few-shot image classification, In ECCV, pp. 18-35, 2020. 20
Maruan Al-Shedivat, Trapit Bansal, Yuri Burda, Ilya Sutskever, Igor Mordatch, and Picter Abheel. Continuous adaptation via meta-leamiig nonstationary and competitive environments. ICLR, 2018. T6
Marcin Andrychowicz, Misha Denil, Sergio Gomez Colmenarejo, Matthew W. Hoffman, David Pfau, Tom Schaul, and Nando de Freitas. Learning learn by gradient descent by gradient descent. In NIPS, pp. 3981-3989, 2016.
I6
Antreas Antoniou, Harrison Edwards, and Amos J. Storkey. How train your MAML. In ICLR, 2019.
Sébastien M. R. Amold and Fei Sha. Embedding adaptation is still needed for few-shot learning. abs/2104.07255, 2021.
CoRR,
Philip Bachman, Alessandro Sordoni, and Adam Trischler. Learning algorithms for active learning. In ICML, pp. 300-310, 2017. [6
Yogesh Balaji, Swami Sankaramarayanan, and Rama Chellappa. Metareg: Towards domain generalization using meta-regularrization. In NeurlPS, pp. 1006-1016, 2018. 16
Jonathan Baxter. A model of inductive bias learning. Journal of Artificial Intelligence Research, 12:149-198, 2000. [6
Irwan Bello, Barret Zoph. Vijay Vasudevan, Quoe Le. Neural optimizer search with reinforcement learning. In JCML, pp. 459-468, 2017. I6
Alberto Bernacchia. Meta-learning with negative learning rates. In ICLR, 2021.
Luca Bertinetto, João F. Henriques, Philip H. S. Torr, and Andrea Vedaldi. Mcta-lcaming with differentiable closed-form solvers. In ICLR, 2019. 17
Leo Breiman. Bagging predictors. Machine learning, 24(2):123-140, 1996.
Wei-Yu Chen, Yen-C Men-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Bin Huang. A closer look at few-shot classification. ICLR, 2019.
20
Xiaohan Chen, Zhangyang Wang, Siyu Tang, and Krikamol Muandet. MATE: plugging in model awareness task embedding for meta learning. NeurIPS, 2020. g
Ignasi Clavera, Anusha Nagabandi, Simin I Liu, Ronald S Fearing, Picter Abbeel, Sergey Levine, and Chelsea Finn. Learning adapt dynamic, real-world environments through meta-reiifforemment learning. In ICLR, 2019. 77
Wang-Zhou Dai, Stephen Muggleton, Jing Wen, Alireza Tamaddoni-Nezhade and Zhi-Hua Zhou. Logical vision:
One-shot meta-interpreeiie leamnnng from real images. In IIP., pp. 46-62, 2017. 16
10

---

Published as a conference paper at ICLR 2022
Giulia Denevi, Carlo Ciliberto, Dimitris Stamos, and Massimiliano Pontil. Learning learn around A common mean. In NeurlPS, pp.
2018. 16
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. CVPR. 2009.
16
Guncet Singh Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto, A baseline for few-shot image classification. ICLR, 2020.
[7
Thomas G Dietterich. Ensemble methods in machine learning. In International workshop multiple classifier systems, 2000.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICI.R, 2021. T6
Yan Duan, John Schulman, Xi Chen, Peter Bartlett, lyy Sutskever, and Picter Abbecl. RI?: Fast reinforcement learning via slow reinforcement learning. CoRR, abs/161 I. ,02779, 2016. [6]
Yan Duan, Marcin Andrychowicz, Bradly Stadie, OpenAl Jonathan Ho, Jonas Schneider, Ilya Sutskever, Picter Abbeel, and Wojeiech Zaremba. Onc-shot imitation learning. In NIPS, pp. 1087-1098, 2017. T6
Harrison Edwards and Amos Storkey. Towards neural statistician. In ICLR, 2017. T6
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture scarch: A survey. Journal of Machine Learning Research, 10:55:1-55::1, 2019. T6
Yang Fan, Fei Tian, Tao Qin, Xiang- Li, and Tic- Yan Liu. Learning teach. In ICLR, 2018. T6
Nanyi Fei, Zhiwu Lu, Tao Xiang, and Songfang Huang, MELR: meta-lcarning via modeling cpisode-level relationships for few-shot learning. In ICLR, 2021. 16
Chelsea Finn. Learning to Learn with Gradients. PhD thesis, UC Berkeley, 2018.
Chelsea Finn and Sergey Levine. Meta-learning and universality: Deep representations and gradient descent can approximate any learning algorithm. In ICLR, 2018. 16
Chelsea Finn, Pieter Abbecl, and Sergey Levine. aggostcc meta-lcarning for fast adaptation of deep T6 20
networks. In ICML, pp. 1126-110 1135, 2017.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning. In NeurlPS, pp. 9537-9548, 2018. 16
Luca Franceschi, Michele Donini, Paolo Frasconi, and Massimiliano Pontil. A bridge between hyperparameter optimization and larning-to-learn. CoRR, abs/1 2017. [6
Kevin Frans, Jonathan Ho, Xi Chen, Picter Abbeel, and John Schulman. Meta learning shared hierarchies. In ICLR, 2018. [6]
Vikas Garg and Adam Kalai. Supervising unsupervised leamiing. In NeurlPS, pp. 4996-5006, 2018. 16
Golnaz Ghiasi, Tsung-Yi Yi Lin, and Quoe V. Le. Dropblock: A regularization method for convolutional networks. In NewrlPS, pp. 10750-10760, 2018. 4
Jiatao Gu, Yong Wang, Yun Chen, Victor K. Li, and Kyunghyun Cho. Meta-learning for low-resource neural machine translation. EMNLP, pp. 3622-3631, 2018. T7
Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He, and Jianfeng Gao. Ms-celeb- dataset and benchmark large-scale face recognition. ECCV. pp. 87-102, 2016. [6]
Bharath Hariharan and Ross Girshick. Low-shot visual recognition by shrinking and hallucinating features. In ICCV, pp. 3037-3046, 2017. [6
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pp- 770-778, 2016.
Timothy M. Hospedales, Antreas Antoniou, Paul Micaclli, and Amos Storkey. Meta-leaming in neural networks: A survey. CoRR, abs/2004.05439, 2020. П
Kyle Hsu, Sergey Levine, and Chelsea Finn. Unsupervised learning via meta-leaming. In ICLR, 2019. 0
11

---

Published as a conference paper at ICLR 2022
Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR, pp. 2261-2269, 2017. [6
Po-Sen Huang. Chenglong Wang, Rishabh Singh, Wen tau Yih, and Xiaodong He. Natural language structured query generation via meta-lcarning. In ACT. pp. 732-738, 2018. [7
Armand Joulin, Laurens Van Der Maaten, Allan Jabri, and Nicolas Vasilache. Learning visual features from large weakly supervised data. In ECCV, pp. 67-84, 2016. [6
tukasz Kaiser, Ofir Nachum, Aurko Roy, and Samy Bengio. Learning remember rare events. In ICLR, 2017. [6
Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. Siamese neural networks for one-shot image recognition. In ICML Deep Learning Workshop, volume 2015. T6
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural
networks. In NIPS, pp. 1106-1114, 2012. T6
Kwonjoon Lee, Subhransu Maji. Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In CVPR, pp. 10657-10665, 2019.
17
Christiane Lemke, Marcin Budka, and Bogdan Gabrys, Metalearning: survey of trends and technologies. Artificial intelligence review, 44(1):117- 130. 2015.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Learning to generalize: Meta-Icarning for domain generalization. In AAAI, pp. 3490-34904 1990-3497, 2018. I6
Kai Li, Yulun Zhang, Kumpeng Li, and Yun Fu. Adversarial feature hallucination networks for few-shot Icarning.
In CVPR, pp. 13470-13479, 2020. 20
Ke Li and Jitendra Malik. Learning to optimize. In ICLR, 2017. 16
Yiying Li, Yongxin Yang, Wei Zhou, and Timothy M Hospedales. Feature-critic networks for heterogeneous domain generalization. In ICML, pp. 3915-3924, 2019. [6
Bin Liu, Yue Cao, Yutong Lin, Qi Li, Zheng Zhang, Mingsheng Long, Han Hu. Negative margin matters:
Understanding margin in few-shot classification. In ECCV, pp. 438 455, 2020. 9 20
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. In ECCV, 196, 2018.
16
Andreas Maurer. Transfer bounds for lincar feature learning. Machine Learning, 75(3):327-350, 2009, T6
Andreas Maurer, Massimiliano Pontil, and Bernardino Romera-Paredes.
Paredes. The benefit of multitask representation
learning. Journal of Machine Learning Research, 17:81:1-81:32, 2016. [6]
Luke Metz, Niru Maheswaranathan, Brian Cheung, and Jascha Sohl-Dickstein. Meta-learingg update rules for unsupervised representation learning. In /CLR, 2019. [6
Sacid Motiian, Quinn Jones, Seyed Mehdi Tranmanesh, and Gianfranco Doretto. Few-shot adversarial domain adaptation. In NIPS, pp- 6673-6683, 2017. 16
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. abs/1883.00299) 2018.
CoRR,
20
Kunkun Pang. Mingzhi Dong, Yang Wu, and Timothy Hospedales. Meta-learning transferable active learning policies by deep reinforcement learning. CoRR, abs/ 806.04795 2018. [6
Zhimao Peng. Zechao Li, Junge Zhang, Yan Li, Guo-Jun Qi, and Jinhui Tang. Few-shot image recognition with knowledge transfer. ICCV, pp. 441 449. 20
Philipp Probst, Anne-Laure Boulesteix, and Bernd Bischl. Tunability: Importance of hyperparameters of
machine learning algorithms. Journal of Machine Learning Research,
2019. [6
Siyuan Qiao, Chenxi Liu, Wei Shen, and Alan L Yuille. Few-shot image recognition by predicting parameters from activations. CVPR, pp. 7229-7238, 2018. 4
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of maml. In ICI.R, 2020. 17
12

---

Published as a conference paper at ICLR 2022
Janarthanan Rajendran, Alex Irpan, and Eric Jang. Meta-learning requires mela-auggmentation. In NeurIPS. 2020. 2 3
Aravind Rajeswaran, Chelsea Finn, Sham M. Kakade, and Sergey Levine. Meta-leaming with implicit gradients. In NeurlPS, pp. 113-124, 2019. 17
Alexander J Ratner, Henry Ehrenberg, Zeshan Hussain, Jared Dunnmon, and Christopher Ré. Learning to compose domain-specific transformations for data augmentation. In NIPS, pp. 3236-3246, 2017.
T6
Sachin Ravi and Hugo Larochelle. Optimization model for few-shot learning. In ICLR, 2017.
16
Sachin Ravi and Hugo Larochelle. Meta-learning for batch mode active learning. In ICLR Workshop. 2018. 16
Scott Reed, Yutian Chen, Thomas Paine, Aäron van den Oord, M. Ali Eslami, Damilo Jimenez Rezende, Oriol Vinyals, and Nando Freitas. Few-shot autoregressive density estimation: Towards learning to learn distributions. In ICLR, 2018. I6
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenenbaum, Hugo Larochelle, and Richard Zemel. Meta-learning for semi-supervised few-shot classification. In ICLR, 2018a.
Mengye Ren, Wenyuan Zeng, Bin Yang. and Raquel Urtasun. Learning to reweight examples for robust deep learning. In ICML, pp. 4331 1-4400 2018b. 16
James Requcima, Jonathan Gordon, John Bronskill, Sebastian Nowozin, and Richard E. Turner. Fast and flexible multi-task classification using conditional neural adaptive processes. In NeurlPS, pp. 7957-7968, 2019. 17
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. In ICLR, 2019. [6
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh. Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei-Fei Li. Imagenet large scale visual recognition challenge. IJCV, 115(3):211-25555 2015. 16
Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. In ICLR, 2019,
Sahil Sharma, Ashutosh Jha, Parikshit Hegde, and Balaraman Ravindran. Learning to multi-task by active sampling. In ICLR, 2018. [6
Jaewoong Shin, Hae Beom Lee, Boqing Gong, Sung Hwang. Large-scale meta-learning with continual trajectory shifting. In ICML, pp. 9603-9613, 2021.
Pranav Shyam, Shubham Gupta, Ambedkar Dukkipati. Attentive recurrent comparators. ICML, pp. 3173-3181, 2017.
16
Christian Simon, Piotr Koniusz, Richard Nock, and Mchrtash Harandi. Adaptive subspaces for few-shot lcarning. In CVPR, pp. 4135-4144, 2020.
Karen Simonyan and Andrew 7isserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015. 16
Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. In NIPS, pp. 4080-4000 4000- 2017.
20
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):
1959,1958, 2014. 7 21
Bradly Stadie, Ge Yang, Rein Houthooft, Peter Chen, Yan Duan, Yuhuai Wu, Pieter Abbeel, and Tlya Sutskever. The importance of sampling in meta-reinforeement learning. In NeurlPS, pp. 9300-9310, 2018. 16
Qianru Sun, Yaoyao Liu, Zhaozheng Chen, Tat-Seng Chua, and Bernt Schiele. Meta-transfer learning through hard tasks. CoRR, abs/1910.03648,, 2019.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip H. S. Torr, and Timothy M. Hospedales. Learning compare: Relation network for few-shot learning. CVPR, pp. 1199-1208, 2018. 16
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan. Vincent Vanhoucke, and Andrew Rahinovich. Going deeper with convolutions. In CVPR, pp. 1-9, 2015. T6
13

---

Published as a conference paper at ICLR 2022
Bart Thomee, David Shamma. Gerald Friedland, Benjamin Elizalde, Karl Ni. Douglas Poland, Damian Borth, and Li-Jia Li. Yfcc100m: The new data multimedia research. Communications of ACM, 59(2):64-73, 2016. [6
Sebastian Thrun and Lorien Pratt. Learning to learn. Springer Science Business Media, 2012.
Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B. Tenenbaum, and Phillip Isola. Rethinking fow-shot image classification; good embedding you need? In ECCV, pp. 266-282, 2020.
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin. Pascal Lamblin, Kelvin Xu, Ross Goroshin. Carles Gelada,
Kevin Swersky, Antorne Manzagol, and Hugo Larochelle. Meta-datasct: dataset of datasets for learning to Icarn from few examples. In ICLR, 2020.
8 17
Hung-Yo Yu Tseng, Hsin- Hsin-Ying Lee, Jia-Bin Huang, and Ming- Hsuan Yang. Cross-domain few-shot classification via learned feature-wise transformation. In ICLR, 2020. I6
Joaquin Vanschoren. Meta-learning: A survey. CoRR, abs/ 1810000333548 2018.
Manasi Vartak, Arvind Thiagarajan, Conrado Miranda, Jeshua Bratman, and Hugo Larochelle. A meta-learning perspective on cold-start recommendations for items. In NIS, pp. 6907 6917, 2017. [6
Ricardo Vilalta and Youssef Drissi. A perspective view and survey of meta-learning. Artificial Intelligence Review, 18(2):77-55, 2002. 16
Orial Vinyals, Charles Blundell, Tim Lillicrap, Koray Kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. In NIPS, pp. 3630-3638, 2016.
3
4
16
20
Risto Vuorio, Shao-Hup Sun, Hexiang Hu, and Joseph J. L Lim. Multimodal model-agnostic meta- lettalla via
task-aware modulation. In NeurlPS. pp. 1- 2019. 17
C. Wah, S. Branson, P. Welinder, P. Perona. and S. Belongie. The Caltech- Birds-200-2011 Dataset.
Technical Report WNNS-TR-2011000 California Institute of Technology. 201
4
Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning reinforcement learn. In CogSci, 2017a. T6
Peng Wang, Lingqiao Liu, Chunhua Shen, Zi Huang, Anton van den Hengel, and Heng Tao Shen. Multiattention network for one shot learning. In CVPR, pp. 6212-6220, 201 [6
Tongzhou Wang, Jun Yan Zhu, Antonio Torralba, and Alexci A. Efros. abs/1811.10999, 8a. 16
Dataset distillation.
CoRR.
Yan Wang, Wei-Lun Chao, Kilian Weinberger, and Laurens der Maaten. Simpleshot: Revisiting nearest-
neighbor classification for few-shot learning. CuRR, 0tos//911.0442... 2019. 1
Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni. Generalizing from few examples: A survey
on few-shot learning. ACM Computing Surveys (CSUR), 53(3);1-34, 2020. p
Yu-Xio Yu-Xiong Wang and Martial Hebert. Learning learn: Model regression networks for easy small sample learning. In ECCV, pp. 616-63666 634, 2016. [6
Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. Learning model tail. NIPS, pp. 7032-7042.
2017c. [6
Yu-Xiong Wang. Ross Girshick, Martial Hebert, Bharath Hariharan. Low-shot learning from imaginary data. In CVPR, pp. 7278-7286, 2018b. [6
Olga Wichrowska, Niru Maheswaranathan, Matthew Hoffman, Sergio Gomez Colmenarejo, Misha Denil, Nando de Freitas, and Jascha Sohl-Dickstein. Learned optimizers that scale and generalize. In ICML, pp-
3751-3760, 2017. 16
Ziyang Wu, Yuwei Li, Lihua Guo, and Kui Jia. PARN: position-awe aware relation networks for few-shot learning. In ICCV. pp. 6658 06686666,6666 2019. 20
Huaxiu Yao, Ying Wei, Junzhou Huang, and Zhenhui Li. Hierarchically structured meta-learning. In ICML, pp. 7045-7054, 2019.
Huaxiu Yao, Long-Kat Kai Huang, Linjun Zhang, Ying Wei, Li Tian, James Zou, Junzhou Huang, et al. Improving generalization in meta-leariin via task augmentation. In ICMI. pp. 11887-11897, 2021.
14

---

Published as a conference paper at ICLR 2022
Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, and Fei Sha. Few-shot learning via embedding adaptation with set-to-set functions, In CVPR. pp. 2805-8814, 2020a.
20
Han-Jia Ye, Xiang- Sheng, and De-Chuan Zhan. Few-shot learning with adaptively initialized task
optimizer: a practical meta-learning approach. Machine Learning, 109(3):643 664, 2020b. [6 17
Han-Jia Ye, Lu Ming, De-Chuan Zhan, and Wei-Lum Chao. Few-shot learning with strong teacher. CoRR, abs/2107.00199 2021. [6
Mingzhang Yin. George Tucker, Mingyuan Zhou, Sergey Levine, and Chelsea Finn. Meta-learning without
memorization. ICLR, 2020. 2B
17
Wei Ying, Yu Zhang, Junzhou Huang, and Qiang Yang. Transfer learning via learning transfer. In ICML, pp.
5072 5072-5081, 2018. I6
Tianhe Yu, Chelsea Finn, Sudeep Dasari, Annie Xie, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot imitation from observing humans via aunain-adaptive meta-learning. In Robotics: Science and Systems, 2018. T6
Zhang, Yujun Cai, Guosheng Lin, and Chunhua Shen. Deepemd: Few-shot image classification with differentiable earth mover's distance and structured classifiers. In CVPR, pp. 12200-12210, 2020.
Ruixiang Zhang, Tong Che, Zoubin Ghahramani, Yoshua Bengio, and Yangqiu Song. Metagan: An adversarial approach to few-shot learning. In NeurIPS, pp. 2371-2380, 2380, 2018a. T6
Yu Zhang, Ying Wei, and Qiang Yang. Learning to multitask. In NeurlPS, pp. 5776-5787, 2018b. I6
Zhi- Hua Zhou. Ensemble methods: foundations and algorithms. Chapman and Hal//CRC, 2012.
15

---

Published as a conference paper at ICLR 2022
Appendix
We provide contents that we omit in the main text.
Appendix A
Additional background and related work.
Appendix B
details of experimental setups (cf. |section 2p.
Appendix
details and analyses of permutations of class label assignments (cf. section 4).
Appendix D
Additional experimental results.
Appendix E
Additional explanation of our studied methods.
A BACKGROUNDS ON META-LEARNING AND FEW-SHOT LEARNING
Training a model under data budgets is important in machine learning, computer vision, and many other application fields, since the costs of collecting data and labeling them are by means negligible.
This is especially the case for deep learning models in visual recognition (He et al.
2016
Dosovitskiy Huang et al.
al.
2021
[Simonyan & Zisserman 2015 Szegedy al. 2015 Krizhevsky al.
2012
2017).
which usually need thousands of, millions of, or even more images to train Russakovsky et al.
2015 Deng et al.) 2009) Guo et al.) 2016 Thomee et al. 2016 Mahajan al.
2018 Joulin et al. 2018]Joulin
2016 in conventional supervised manner. Different from training a model to predict at the instance level, meta-learning attempts to learn the inductive bias across training tasks (Baxter 2000: Vilalta & Drissi 2002). More specifically, meta-learning aims to train "meta-model" to summarize the common characteristics of tasks and generalize them to those novel but related tasks (Maurer) 2009)]
Maurer et al. 2016; Denevi et al. 20188] Ye et al., (021). Meta- Metanlearning has been applied in various
fields, including few-shot learning Ravi & Larochelle| 2017: Snell et al.) 2017: Wang & Hebert
2016 Ye et al. 2020a Sunng et al., 2018|Zhang
al. [0]] 80088al Wang al.
[0188]] seng et al.
2020
Fei et al. [002). optimization (Andrychowicz et 2016] Wichrowska et al.
[217 Li & Malik
2017
Bello al. [0077) reinforcement and imitation learning (Stadie et al. [2018] Frans 2018
Wang
ct al. 201 7a Duan et al., 2016 2017 Yu et al. 2018). unsupervised learning (Garg & Kalai
2018
Metz et al. [2019] Edwards & Storkey 2017; 7 Reed et al. 2018), continual learning (Riemer et al.
2019 Kaiser et al.
2017 Sh-Sdviiv et al. 20118). imbalance learning (Wang et al. [017c] Ren et al.
2018b), transfer and multi-task learning (Motian al. 2017 Balaji et al.] [0188] Ying al. [0188]
Zhhagg et al. 2018b] Li et al. 2019; 2018), 8) active learning (Ravi & Larochelle] 2018 Sharma et al. 2018 Bachman et al. 2017] Pang et al. 2018), data compression (Wang et al. 2018a), architecture
search (Elsken et al. 2019)) recommendation systems (Vartak et al. (017)) data augmentation (Ratner
ctt al. 2017). teaching (Fan et al. 2018), hyper-parameter tuning (Franceschi et al. 2017 Probst et al. etc.
2019
In few-shot learning (FSL), meta-learning is applied to learn the ability of "how to build a classifier using limited data" that can be generalized across tasks. Such inductive bias is first learned over few-shot tasks composed of "base" classes, and then evaluated on tasks composed of "novel" classes. For example, few-shot classification can be implemented in non-parametric way with soft nearest neighbor (Vinyals et al.] 2016) or nearest center classifiers (Snell et al., (017), so that the feature extractor is learned and acts the task level. The learned features pull similar instances together and push dissimilar ones far away, such that a test instance can be classified even with a few labeled training examples (Koch al. 2015). Considering the complexity of hypothesis class, the model training configurations (i.e.,
also serve as a type of inductive biases. (Andrychowicz al. (2166); Ravi & Larochelle (2017) meta-learn the optimization strategy for each task, including the learning rate and update directions. Other kinds of inductive biases are also explored. Hariharan Girshick (2017); Wang et al. (20T8h) learn data generation prior to augment examples given few images; Dai et al. (2017) extract logical derivations from related tasks; Wang et al. (2017b); Shyam ct al. (017)) learn the prior to attend images.
Model-agnostic meta-learning (MAML) (Finn et al.) 2017) proposes another inductive bias, i.e., the model initialization. After the model initialization that is shared among tasks has been mntaatrainedd, the classifier of new few-shot task can be fine-tuned with several steps of gradient descent from that initial point. The universality of this MAML-type updates is proved in (Finn & Levine 2018). MAML has been applied in various scenarios, such as uncertainty estimation (Finn et al. 2018
16

---

Published as a conference paper at ICLR 2022
robotics control (Yu et al. 2018 Clavera et al. 2019), neural translation (Gu et al. 2018), language generation (Huang et al. 2018). etc.
Axampie,Nicholl Nichol al. (20 8 handle the computational burden by presenting a family of approaches using first-order approximations. Rajeswaran al. (2019) propose leverage implicit differentiation, making the calculation of the meta-gradients much efficient and accurate. Anttoniuu et al. (019) provide a bunch of tricks to train and stabilize the MAML framework. [ernacchia (2021) points
Despite the success, there still problems with MAML. For example,
that negative rates of gradient
updates help in some scenarios. RRajendran et al. (2020); Yao el al. 2) Yin et al. (2020) argue that the learned initialization by MAML may be at high risk of (a) memorization over wett-nitinng where it solves meta- meta-training tasks without the need of inner loop optimization, or (b) learner over-fitting, where over-fits to the meta-training tasks and fails to generalize to the meta-testing tasks. They thus propose to improve MAML by imposing regularizer or performing data augmentation.
Since MAML applies a uniform initialization all the tasks (i.e., the same set of { and Φ). recent methods explore ways to better incorporate task characteristics, [ee et al. ( Bertinetto ct al. ) Raghu et al. (2020) optimize the linear classifiers {we)] 12 (not the feature fø) till convergence in the inner loop: Triantafillou et al. (2020): Ye et al. (2020b) initialize the linear classifiers using class prototypes (i.e., aggregated features per class) so they are task-aware even before the inner loop optimization. Another direction is to enable task-specific initialization for the
entire model (Requeima et al. 2019, Vuorio et al. 2019: Yao et al.) 2019 Rusu et al. 2019), which
often needs additional sub-networks.
Our work is complementary to the above improvements of MAML: issue of MAML in meta-festing and conduct a detailed analysis. We then build upon improve MAML. We note that some of the above methods can be permuration-invarian even though they are not designed for the purpose. For example. LEO (Rusu et 2019) computes class prototypes (i.e., averaged features per class) to represent each semantic class" However, it introduces additional sub-networks. MetaOptNet (Lee al., 2019) performs inner loop optimization only on { (till convergence), making it convex problem which is not sensitive to the initialization and hence the permutations. This method, however, has a high computational burden and needs careful hyper- parameter tuning for the additionally introduced regularizers. Proto-MAML (Triantafillou et al. (0200) initializes the linear classifiers [w.] N c=1 with the prototypes, which could be permutation-invan inviaairnnnnniiinnnnnnnaa but We note that one fundamental difference
we find an inherent permutation
J===
cannot achieve accuracy high between our work and (Dhillon et al.] 2020). LEO (Rusu et al. 2019), and Proto-MAML (Triantafillou et al. 2020 is the motivation: they aim to provide the initialization of the classifier head with better semantic meanings, while our goal is to resolve the sensitivity of MAML to the permutations.
UNICORN-MAME.
B DETAILS OF EXPERIMENTAL SETUPS
We follow the pre- training procedure in Ye et al. 2020a). During pre-training append the feature extractor backbone with fully-comnected layer for classification, and train it to classify all classes in the base class set (e.g., 64 classes in the MinilmageNet) with the cross-entropy loss. In this stage, we apply standard ImageNet image augmentations (e.g., random crop and random flip). The best pre-trained feature extractor (i.e.. epoch) is selected based on the one-shot classification accuracy on the meta-validation set. Specifically, we sample one-shot tasks from the meta- validatiiot set, and apply the nearest neighbor classifier on top of the extracted features to evaluate the quality of the backbone. Finally, the best pre-trained backbone is used to initialize the feature extractor for MAML.
C PERMUTATIONS OF CLASS LABEL ASSIGNMENTS
We provide more analyses and discussions on the permutation issue in the class label assignment. As illustrated in Figure (a), few-shot tasks of the same set of N semantic classes (e.g., "unicorn", "bee", etc.) can be associated with different label assignments (i.e., e E [N) and are paired with the learned initialization J.y of MAMI. differently. For five-way tasks, there are 120 permutations.
WWe note that while Requeima et al. 2019) Yuorio et al. 2019 Yao et al. (019) also enable task- specific initialization with additional sub-networks for task embedding, their methods cannot resolve the permutation issue. This is because they take an average of the feature emheddings over N' classes represent task.
17

---

Published as a conference paper at ICLR 2022
Support Set Acccuracy (%)
Query Set Acccuracy (%)
100
90
90-
80
40
30
20
10
15
15
30
Inner Iterations
Inner Iterations
Figure F: The support (left) and query (right) set accuracy on a randomly sampled five-way one-shot meta-testing task from MinilmageNet. We plot the accuracy of each permutation (totally 120) along with the process of inner loop optimization (the same permutation colored the same in the left and right images).
In section 4] we study how the permutations affect the meta-testing accuracy (after the inner loop optimization of Φ and {w=)==) and see a high variance among the permutations. We note that, the inner loop optimization updates not only the linear classifiers {wa) but also Φ of the feature extractor. Different permutations therefore can lead to different feature extractors.
Here, we further sample five-way one-shot meta-testing task, and study the change of accuracy along with the inner loop updates (using a MAML trained with fixed number of inner loop updates). Specifically, plot both the support set and query set accuracy for each permutation. As shown in Figure F] there exists a high variance of query set accuracy among permutations after the inner loop optimization. This is, however, not the curves appear for the support set is because there are only five examples, and all the permutations reach 100% support set accuracy within five inner loop steps.) Interestingly, for all the permutations, their initialized accuracy (i.e, before inner loop optimization) all 20%. After an investigation, we find that the meta-learned { JN (initialization) is dominated by of them; i.e., all the support or query examples are classified into one class. While this may not always be the case for other few-shot tasks or if we re-train MAML, for the task sampled, it explains why obtain 20% for all permutations. We note that, even with an initial accuracy of 20%, the learned initialization can be updated attain high classification accuracy.
for the support set. (The reason that only three
We further compare the change of support and query set accuracy along with the inner loop opti- mization in Figure F We find that, while both accuracy increases, since the support set accuracy converges quickly and has a smaller variance among permutations, is difficult to use its information to determine which permutation leads to the highest query set accuracy. This makes sense since the support set is few-shot: its accuracy thus cannot robustly reflect the query set accuracy. This explains why the methods studied Table cannot determine the best permutation for the query set.
C. MATHEEMATICAL EXPLANATION
We provide a simple mathematical explanation for why, on average, the query set accuracy is at the chance level if we directly upply the learned initialized model. Suppose we have five-way task with five semantic classes {"dog", " "cat". "bird", "car", "person").. Without loss of generality, let us assume that the query set has only five examples, one from each class. Let us also assume that the gives 100%
best permutation - i.e., the best assignment of (wn, ws) } to these classes query set accuracy using the initialized model. Since there are total 120 possible permutations, there will be 10 of them with 60% accuracy (i.e., by switching two-class indices), 20 of them with 40% accuracy (i.e., by shuffling the indices of three classes such that they do not take their original indices), 45 of them with 20% accuracy, and 54 of them with 0% accuracy. Taking average these permutations gives a 20% accuracy. other words, even one of the permutations performs well, on average the accuracy will be close random.
18

---

Published as a conference paper at ICLR 2022
84
92
84
82
90
76
76
$3.44
74
20
20
90.35
82
11.3 5L-3
16:2
5E-2
1E-1
1E.3 SE- 12:2
5k.1
1611
IR-3
SE.3
1E-2
5R-2
IE-1
MinelmageNet, ResNet
aagee, ResNet
CUB, ResNet
72
72
74
70
70
72
68
68
10
10
68
15
15
66
20
74.99
IF-3 5E-3 IE-2 SE-2
1F-1
1F-3 5E-3
IF-2
5E-2
1E-1
1B-3 5E-3
1E:2
SE-2
1E-1
MinilmageNet, ConvNet
Tagred ConvNet
CUB, ConvNet
Figure G: Heat maps of MAML's five-way five-shot accuracy MinilmageNet, TieredImageNet, and CUB w.rt. the inner loop learning rate a (X-axis) and number of inner loop updates M (y-axis). For each map, we set accuracy below a threshold to fixed value for clarity; we denote the best accuracy by a black box.
Acccuracy (%)
100
100
Acccuracy (%)
100
90
90
80
Acccuracy
70
70
60
60
Meta-Training
50
Meta-Training
50
Meta-Training Meta-Testing
40
40
40
Meta-Testing
30
Meta-Testing
30
20
20
15 20 25 30
10 15 20 25 30 Inner Iterations
1
5
10 15 20 25 30 Inner Iterations
Inner Iterations
MinilmageNet, ResNet
TieredImageNet, ResNet
CUB, ResNet
Acceuracy (%)
Acccuracy aaayyyy [%)
80
70
70
70
60
60
60
50
50
Acccurae
50
40
Meta- eetanira Meta-Testing
40
Meta-Training
40
Meta-Training Meta-Testing
30
30
Meta Testing
30
20
5 10 15 20 25 30 Inner Iterations
20
25 30
10 15 20 25 30 Inner Iterations
Inner Iterations
MinilmageNet, ConvNet
TieredImageNet, ConvNet
CUB, ConvNet
Figure H: We plot the change of the five-way five-shot classification accuracy (on the query set), averaged over 10,000 tasks sampled from either the meta vreta-taaning (red) or meta-testing classes (blue), along with the process of inner loop updates, using the best model initialization learned by MAML.
D ADDITIONAL EXPERIMENTAL. RESULTS
Similar to Figure 2, we plot the meta-testing accuracy five-way five-shot tasks on the three datasets over ResNet and ConvNet backbones in Figure We get similar trend with Figure 2 where MAML achieves higher and much more stable accuracy (w.nt. the learning rate) when is larger than 15. Specifically, for MinilmageNet with ResNet, the highest accuracy 83.44% is obtained with M 20, higher than 80.81% 1% with M 5.
We also plot the change of five-way five-shot classification accuracy (on the query set), averaged over 10,000 tasks sampled from either the meta meta-training (red) or meta-testing classes (blue), along with the process of inner loop updates, using the best model initialization learned by MAMI. in Figure Hor
19

---

Published as a conference paper at ICLR 2022
Table G: 5-Way 1/5-Shot classification accuracy and 95% confidence interval on CUB, evaluated over 10,000 tasks with a ResNet-12 12 backbone. f: methods with 18 backbone. f: We train MAML with 5 inner loop steps in both meta-training and meta-testing. *: we carefully select the number of inner loop steps for MAML, based on the meta-validation set.
ResNet-12
I-Shot
5-Shot
MatchNet
Vinyaal et al.
2016
66.09 = 0.92
82.50 + 0.58
ProtoNet
Snell et 2017
71.87 0.85
85.08 0.57
DeepEMD
Zhang et al.
2002
75.65 0.83
88.69 0.50
Baseline++
Chen et al.
2019)
67.02 = 0.90
33.582.55
AFHNt
Li et al.
2020)
70.53 1.01
83.95 0.63
Neg-Cosine
(Liu et al.
2020
72.66 0.85
Align (Afrasiyabi et al.
2020
74.22 1.09
88.65 + 0.55
MAMI. (5-Step')
76.53 0.20
88.34 0.16
MAML (our reimplementation")
77.67 0.20
90.35 + 0.16
UNICORN-MANE MAML
78.07 = 0.20
91.67 0.16
Table H: Way 1-Shot and 5-Shot classification accuracy and 95% confidence interval on MinilmageNet over 10,000 tasks with four-layer ConvNet backbone. f: We train MAML with 5 inner loop steps in both meta-training and meta-lesting. *: we carefully select the number inner loop steps for MAML, based on the meta-validation set.
ConvNet
1-Shot
5-Shot
MAML
Finn et al. 2017)
48.70 + 1.84
63.11 + 0.92
MAML++ (Atonioou et al. 2019
52.15 0.26
68.32 + 0.44
Reptile (Nichol et al. 2018)
65.99 + 0.58
FEAT (Ye et al. 2020a)
55.15
71.61 + 0.16
KTN (visual) (Peng et al.
+.80
71.21 + 0.66
PARN [u et al. 2019
71.55 0.66
MAML-MMCF (Yao et al. 2021
50.35 = 1.82
64.91 - 0.96
MAML (5-Step')
53.15 0.20
67.01 + 0.16
MAML (our reimplementation")
54.89 + 0.20
71.72 + 0.16
UNICORN-MAML.
55.70 0.20
72.68 + 0.16
each pair of dataset and backbone. There indicates a similar phenomenon as Figure where MAML needs large M.
We further evaluate UNICORN-ME -MAML on CUB dataset with ResNet- 12 backbone and on MinilmageNett with the four-layer ConvNet backbone. The results are listed in [abll G and Ta ble H respectively. By comparing UNICORN-M -AML.. with others, we find the carefully tuned MAMI. shows promising results and UNICORN-MAMIL outperforms the existing methods.
E ADDITIONAL EXPLANATIONS OF OUR STUDIED METHODS
We provide some more explanations on the ensemble and forced permutation-invariae methods introduced in disectiion For the ensemble method, given a few-shot task, we can permute (w,) to pair them differently with such task. We can then perform different inner loop optimizations to obtain a set of five-way classifiers that we can perform ensemble upon. the main text, we average the posterior probabilities of these five-wer way classifiers to make the final predictions.
Since the permutation affects the meta-tras tetta-traiiing phase as well, we can interpret the meta-training phase as follows. Ever time sample few-shot task T = (S, Q), also sample a permutation π [N] + [N] to re-label the classes. (We note that, this is implicitly done when few-shot tasks are sampled.) We then take T, (S., 2x)) to optimize 0 in the inner loop. That is, in meta-training, the
20

---

Published as a conference paper at ICLR 2022
objective function in Equation 2
can indeed be re- re-written as
) p(pp)) C(O,. 0')
E
E
0, M)),
(D)
(S. (S.0))p((Tarrp
where p(T) is uniform distribution over all possible permutations. re-written as
Equation can be equivalently
E (S. (...)=-(T)xxppppa ((Q,6"))
E
((Q, InLoop(S, M)),
(E)
(8.
where 0x means that the initialization of the linear classifiers {we]e= J. are permuted; 0- is the corresponding updated model. This additional sampling process of π is reminiscent of dropout (Sri) vastava et al., 2014), which randomly masks out neural network's s neurons or edges to prevent an over- whe-pparameterized neural network from over-fitting During testing, dropout takes expectation over the masks. We also investigate similar idea during meta-tes meta-testing. by taking expectation (i.e., average) on the learned initiation of the linear classifiers over different permutations. This results in new initialization during the meta-testing phase: w. N Eee wee
21


Reviews:
[{"review_id": "_K21-0XW_hP", "reviewer": "Reviewer_zwrd", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n\n- The paper is well written and easy to follow. The main problem at hand, the random label assignment during evaluation, is introduced through different experimentations, showing why and where is the problem in MAML.\n- A good explanation and novel investigation of this problem. The results show that MAML can perform on-par with and even outperforms other state-of-the-art meta-learning methods.\n- The experiments are detailed and multiple methods are tested to try to alleviate the permutation label assignment problem for MAML, before introducing their proposed modification to the algorithm.\n- The experimental settings and hyperparameters are all mentioned and documented to help reproducibility.\n- Even though the experiments focuses on the MAML algorithm, the authors compare their results with related works, but also discuss how other methods deal with the random label assignment.\n\n### Weaknesses\n\n- I am quite surprised by the performance they achieve with their vanilla MAML, even without considering the increased inner steps. I never saw performance as high in other papers using MAML with a ResNet12. Maybe the difference come from the pretrained weights ? I didn't find any information on how the backbone was pretrained though and I think this information would help for reproducibility.", "summary_of_the_paper": "The paper mainly investigates the effect of permutation in the class label assignment in the tasks for the MAML algorithm. First, the authors show that MAML requires a higher number of inner loop updates than what is commonly used. Then, they show that MAML is sensible to the permutation of the class labels in the tasks and experimented with diverse methods to alleviate this problem. Finally, they proposed Unicorn-MAML, a modification of the MAML algorithm that learns a single weight vector for the classifier layer, to make the model permutation invariant. They show through experiments on two different datasets (mini-ImageNet and tiered-ImageNet) that it achieves or outperforms state-of-the-art performance.", "summary_of_the_review": "I found the paper clear and easy to read. All the experiments are detailed as well as the thought process behind them. I would appreciate more information on the full training process to improve reproducibility.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "XhESmg3MoAA", "reviewer": "Reviewer_eimD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is an interesting study which is mainly inspired by the empirical results. However, I have some concerns that needs to be addressed: \n\n \n \n\n- In equation (1), the gradient is taken with respect to the $\\theta^{‘}$ as we are in the inner-loop? \n\n \n \n\n- One main concern of this study is that when MAML pairs the learned classifier head with different permutations of a task, there could be an inconsistency in the performance. However, I believe this limitation does not exist in works like ProtoNet, as we have no $\\omega$ there for pairing, and the prototype of each class does not depend on the class index but support set samples of that class. One major question is that, since ProtoNet has implicitly addressed this concern before, what is the *importance* of this study? \n\n \n \n\n- Considering good pairing between meta-test task and classifier head is not a new idea in few-shot learning. For example, in Dhillon et. al. [1], they implicitly consider this by initializing the classifier’s head using the normalized logits for each class, inspired by cosine distance. However, this work uses the simple idea of repeating a meta-learned vector as the classifier's weights. I think you need to add this work and similar works to related work, and emphasize on the differences. \n\n \n \n\n- My *major concern* is about your motivation for this study. Let’s have a quick look at the possibility of the permutation you discussed as motivation and illustrated in Figure 1.  \n\nConsidering the meta-test set of miniImageNet, there are 20 classes on this set. Then for 5-way classification as we need to sample 5 classes to construct a task (or episode), the number of permutations of 20 classes taken 5 at a time is 20!/(20-5)! = 1860480. Among these, 5!=120 could be the task that has the same classes. So, you are planning for an event with a chance of 120/1860480=0.000064 which is really negligible. This gets even worse for other datasets like tieredImageNet and CUB as they have larger numbers of classes in meta-test, or even for real scenarios with larger unseen classes. \n\nHowever, looking at your results, I think your solution is more general  and may be also considered as differences in the class attributes. So, you need to change your motivating example and make it more general. Although your solution answers, I think you have approached it with the wrong motivation. \n\n \n \n\n- The procedure to produce the results in figure 3 is not clear to me. Do you use meta-validation performance to select M? Or do you just test different values of M for training and testing? Please elaborate on this. \n\n \n \n\n- In section 5, when you propose simple solutions to make MAML permutation-invariant during meta-testing, what is the point of searching for the best permutation for a meta-training task? How is it going to affect the training of more powerful features and improving generalization? \n\n \n \n\n- Based on the results in Table 4, UNICORN-MAML (which has a lower number of parameters w.r.t MAML) improves its performance. However, I am concerned when authors mention that they have compared with state-of-the-art, because they have not included recent works in the field, like [2], which shows produced results fall behind current state-of-the-art. I think it is more reasonable to mention that UNICORN-MAML can be considered as a strong baseline for future works, and I am ok with this. \n\n \n \n\n- In section 6, I didn’t find the answer for the question “Why does UNICORN-MAML work?” informative enough. You may give more detail which backed up with results, or some facts from previous works, or simply omit this. \n\n \n \n\n- The explanation of Figure 5 is not relevant to the main purpose of the paper.  As one of key observations in your work, you have mentioned that MAML needs lots of inner-loop updates in Figure 3. However, when providing similar results (as you mentioned in the main text) for UNICORN-MAML, you conclude that it aligns with previous work that “we need to also adapt the feature extractor”? How do you relate this finding to your intention in this work?  \n\nIt needs to be related to the main context. Please elaborate on this. \n\n \n \n\n- Table 6 does not provide a fair comparison with current algorithms for cross-domain few-shot classification. As mentioned in [3], in the same scenario (miniImageNet $\\rightarrow$ CUB), a metric-based algorithm like RelationNet, achieves 57.77% in 1-shot. Using FT proposed in [3], it can also achieve up to 59.94%. However, you have reported 51.80% as SOTA result in this case? \n\n \n \n\nReferences: \n\n[1] A Baseline for Few-Shot Image Classification, ICLR 2020. \n\n[2] MELR: Meta-Learning via Modeling Episode-Level Relationships for Few-shot Learning, ICLR 2021. \n\n[3] Cross-Domain Few-shot Classification via Learned Features-wise Transformation, ICLR 2020. \n\n \n ", "summary_of_the_paper": "This paper analyses the well-known and well-studied MAML algorithm and raises two key observations. First one is requiring a high number of inner-loop updates, and the second one is the variation in the meta-test accuracy when permuting the indices of the classes. Then it proposes to simply meta-train a single vector and duplicate it as initialization for the classifier head to make MAML permutation-invariant and improve its generalization performance on meta-test tasks. ", "summary_of_the_review": "This work proposes a simple yet interesting algorithm which meta-trains just a single vector and duplicates it as initialization for the all classification head weights for various classes in MAML. It improves the performance of the MAML, however I have some serious concerns regarding the importance, the validity of the motivation and the fairness of the results for this study. \n\n \n ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WqG39GPMrLm", "reviewer": "Reviewer_Qepb", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Advantages\n\n1. The paper is interesting for me. I believe it is worth questioning the key settings in the benchmark or classical baseline methods. This encourages people to rethink why the model works or not in specific scenarios. Also, the idea of UNICORN-MAML is simple and easy to implement.\n\n2. This paper offers a detailed exploration for two factors (in MAML based meta-learning benchmark) that are ignored by previous studies: one is the number of inner-updates in each training/testing task and the other is the task label assignment in each task. It is interesting to see the statistics in Fig4 that permutations make such a big difference to specific test tasks.\n\nDisadvantages\n\n1. From the results in Table 2,5,6, it is pity that there is little concrete performance boost by using the new MAML (UNICORN-MAML) compared to MAML (ours) --- where I assume ours means \"our careful implementation of MAML\".\n\n2. Main conclusions or results are from arbitrarily designed experiments on existing benchmarks of few-shot learning---small tasks of object classification. It is hard to guarantee that the conclusion maintains in meta-learning larger-scale tasks or in another new benchmark. \n\n3. MAML was implemented originally not only on few-shot classification tasks but also on reinforcement learning tasks. While it is clear that this new UNICORN-MAML may not be straightforwardly used for RL, as it studies the specific problems in the image classification settings.", "summary_of_the_paper": "This paper is an interesting \"re\"search on a classical meta-learning method MAML. It shows that both the inner update iterations and task label assignment have a clear influence on the performance of MAML. It raises some questions and uses a set of in-depth experiments to draw conclusions. ", "summary_of_the_review": "I think it is interesting to research and rethink the possible issues in the existing baselines or benchmarks. I have a few concerns about the significance of this current version of the paper because of the poor performance or the arbitrary design of experiments. I would like to see the comments of other reviewers and the discussion with the authors to make a final decision. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LHZETngsEqO", "reviewer": "Reviewer_mgm6", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nI really liked the way that the authors study the MAML algorithm. They performed very systematic and reasonable experiments. The second part of the paper that looks at the permutation of the labels is interesting. Ideally, this should not have happened since the task is the same and labels should not impact the performance. For example, for metric-based approaches such as ProtoNets[2], this never happens. Also, the proposed approach (UNICORN-MAML) just defines a weight vector that is shared for all classes such that permutation has no impact on the updates. It is a very interesting way to solve the problem.\n\n\n\n\n\n\n\n\n[1] Hsu K, Levine S, Finn C. Unsupervised Learning via Meta-Learning. In Int'l Conf. on Learning Representations 2018.\n\n[2] Snell J, Swersky K, Zemel R. Prototypical networks for few-shot learning. In Proc. of the 31st Int'l Conf. on Neural Information Processing Systems 2017.\n", "summary_of_the_paper": "The authors study the MAML algorithm and propose two ways to improve the performance. First, they study the number of inner gradient steps. Second, they look at the permutation of labels when learning a new task which ideally should not make any difference. However, it seems that during testing this could lead to different test performances on a target task. Based on this observation, they propose a novel and very interesting solution to share the weights of the classification layer (which they call UNICORN MAML).", "summary_of_the_review": "Finally, I think this is a thorough study of a very useful algorithm and can improve many papers that are based on this approach. It is a good improvement on MAML that is based on a very detailed systematic study. As a result, I vote for acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Three of four reviewers rated this paper as an 8. 
These positive reviewers felt that this paper provided a lot of value through extensive experimentation with MAML in the few-shot setting. It was felt that the detailed analysis of the inner and outer loop of MAML provided a lot of understanding to the reader regarding the behaviour of MAML. The fourth reviewer giving a score of 3 remains concerned about high variance on some experiments. However the strength of ratings from the other reviewers make the AC more than comfortable giving an accept recommendation for this work.
Decision: Accept (Poster)