A Lazy Approach to Long-Horizon Gradient-Based Meta-Learning
Authors: Muhammad Abdullah Jamal
Liqiang Wang
Boqing Gong
Venue:  (2021)
Keywords: meta-learning
learning to learn

Abstract: Gradient-based meta-learning relates task-specific models to a meta-model by gradients. By this design, an algorithm first optimizes the task-specific models by an inner loop and then backpropagates meta-gradients through the loop to update the meta-model. The number of inner-loop optimization steps has to be small (e.g., one step) to avoid high-order derivatives, big memory footprints, and the risk of vanishing or exploding meta-gradients. We propose an intuitive teacher-student scheme to enable the gradient-based meta-learning algorithms to explore long horizons by the inner loop. The key idea is to employ a student network to explore the search space of task-specific models adequately (e.g., by more than ten steps), and a teacher then takes a ``leap'' toward the regions probed by the student. The teacher not only arrives at a high-quality model but also defines a lightweight computation graph for meta-gradients. Our approach is generic, as we verify its effectiveness with four meta-learning algorithms over three tasks: few-shot learning, long-tailed classification, and meta-attack.

Under review as a conference paper at ICLR 2021
A LAZY APPROACH TO LONG-HORIZON GRADIENT- BASED META-LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Gradient-based meta-learning relates task-specific models to a meta-model by gradients. By this design, an algorithm first optimizes the task-specific models by inner loop and then backpropagates meta-ggradiett, through the loop to update the meta-model. The number of inner-loop optimization steps has to be small (e.g., one step) to avoid high-order derivatives, big memory footprints, and the risk of vanishing or exploding meta-gradients. We propose an intuitive teacher- student scheme to enable the gradient-basee meta-learning algorithms to explore long horizons by the inner loop. The key idea is to employ a student network to adequately explore the search space of task-specific models (c.g.. by more than ten steps), and teacher then takes "leap" toward the regions probed by the student. The teacher not only arrives at high-quality model but also defines lightweight computation graph for meta-gradients. Our approach is generic; it performs well when applied to four meta-learning algorithms over three tasks: few-shot learning, long-taid long-tailed classification, and meta-attase meta-attack.
INTRODUCTION
Humans can quickly learn the skills needed for new tasks by drawing from fund of prior knowledge and experience. To grant machine learners this level of intelligence, meta-learning studies how to leverage past learning experiences more efficiently learn for new task (Vilalta & Drissi 2002). A hallmark experiment design provides meta-learner a variety of few-shot learning tasks (meta- traitatrainin) and then desires it to solve previously unseen and yet related few-shot learning tasks (meta-test). This design enforces "learning to learn" because the few-shot training examples are insufficient for learner achieve high accuracy any task in isolation.
Recent meta- leatnlea methods hinge on deep neural networks. Some work learns a recurrent neural network as update rule to model (Ravi & Larochelle] 2016 (Andyychowicz al., 2016). Another line of methods transfers an attention scheme across tasks (Mishra et al. 2017] Vinyals et al. (016a). Cradient-bases meta-learning gains momenta recently following the seminal work (Finn et 2017). It is model-agnosee meta-learning (MAML), learning global model initialization from which meta-learner can quickly derive task-specific models by using few training examples.
In its core, MAML is a bilevel optimization problem (Colson et al. 2007). The upper level searches for the best global initialization, and the lower level optimizes individual models, which all share the common initialization, for particular tasks sampled from task distribution. This problem is hard to solve. Finn al. (2017) instead propose "greedy" algorithm, which comprises two loops. The inner loop samples tasks and updates the task-specific models by A steps using the tasks' training examples. The k-step updates write differentiable computation graph. The outer loop updates the common initialization by backpropagating meta- meta-gradient gradients through the computation graph. This method is "greedy" in that the number inner steps often small (c.g., k = 1). The outer loop takes actions before the inner loop sufficiently explores its search space.
This "greedy" algorithm due to practical constraints that backpropagating meta meta-gradients through the inner loop incurs high-order derivatives, big memory footprints, and the risk of vanishing or exploding gradients. For the same reason, some related work also turns to greedy strategies, such as meta-attack (Du al. 2019) and learning reweigh examples (Ren al. 2018b).

---

Under review as a conference paper at ICLR 2021
To this end, it is natural to pose at least two questions. Would less greedy gradient-based meta- learner (say, >> 10 inner-loop updates) achieve better performance? ? How to make it less greedy?
To answer these questions, provide some preliminary results by introducing lookahead opti- mizer (Zhang et al. 2019) into the inner loop. It is intuitive to describe it as a teacher-student scheme. We use student neural network explore the search space for given task adequately (by large number i of updates), and a teacher network then takes a "leap" toward the regions visited by the student. As result, the teacher network not only arrives at high-performing model but also defines a very lightweight computation graph for the outer loop. In contrast to the traditionally "greedy" meta-learning framework used in MAML (Finn al., 2017), meta-attact (Du et al. 2019), ) learning to reweigh examples (Ren et al.] 201 etc., our approach has a "lazy" teacher. It sends student to optimize for a task up to many steps and moves only once after that.
Rajeswaran et al. (2199) proposed a less "greedy" MAML, with which this work shares a similar goal, but our approach improves the gradient-based meta-learning framework rather than a par- ticular algorithm. Hence, we evaluate it on different methods and tasks, including MAML and Reptile (Nichol et al. 2018) for few-shot learning, a two-component weighting algorithm (Jamal et al. 2020) for long-tailed classification, and meta-attack (Du et al. 2019). Extensive results provide exploration in the inner loop improves
an affirmative answer to the first question above: meaa-leamnrr's 's performance. We expect approach, along with the compelling experimental results, can facilitate future work to address the second question above.
2 "GREEDy" GRADIENT-BASHE META-LEARNING
We first review gradient-basee based meta- learnig from the perspective of "search space carving".
Notations, Let Pr denote task distribution. For each task drawn from the distribution T ~ Pr. we have a training set Drr and a validation set Poats both in the form of { 31). (x22, .2)..... where I' and Um are respectively an input and a label. We learn a predictive model for the task by minimizing an empirical loss CTD,, (e) (e-g., cross-entropy) over the training set while using the validation set to choose hyper-parameters (e.g.. early stopping), where e collects all trainable parameters the model. Similarly, we denote by L% (Ø) the loss calculated over the validation set.
Meta-learning as "space carving". Instead of focusing an isolated task, meta-learning takes a global view and introduces a meta- model, parameterized by 6, that can improve the learning efficiency for all individual tasks drawn from the task distribution Pr. The underlying idea is to derive a fask-specific model from not only the training set Der but also the meta-model 0, i.c., M(0, Drr). We refer to M(0 Drr) the "carved" search space for the task-specific model é. where the "carving" function is realized as an attention module in (Vinyals et al. 2016a Mishra al., (017), as a conditional neural process in (Garnelo et al., 2018; Gordon et al. 2020)), gradient-based update rule in (Finn et al., 20177: Park & Oliva 2019] Li et al. 2017 Nichol et al. 2018). and as a regularized optimization problem in (Rajeswaran et al. 2019 hoou al. 2019)).
An optimal mota- 6* is supposed to yield the best task-specific models in expectation,
0" arg min ET P CD (o" (0) subject o" (8) + arg min sEMi0,Do)
(o.. (1)
One can estimate the optimal meta-model A from some tasks and then use to "carve" the search space, M(" Drr), for novel tasks' models.
Gradient-basee meta-learning. One of the notable meta-learning methods is MAML Finn et al. 2017), which uses a pradienttbasedd update rule to "carve" the search space for task task-specific model,
MMAMIL (0, D..)) = (do - 0) U (0, 10 + - vV (6) -
j = 1,2, .., ,k] ,k)
(2)
where the mota-model 0 becomes an initialization to the task-specific model do. the other candidate models .I.... &x are obtained by gradient descent, and > is a learning rate. Substituting it into equation ( dR E MMMMML (0, Dtr) is naturally solution to the lower-level optimization problem, and MAML solves the upper-level optimization problem by gradient descent,
0 -- 0 - -BBTTPy DoaInT T V Vo (0).
(3)
2

---

Under review as a conference paper at ICLR 2021
MAML
Implicit MAML
"Lazy" MAML
0
03
23
02
spec-fc parameter
Computation
Task-specific gradients V D. (6)
Meta gradients
V. E, C paa
(0)
To compute the meta-gradients Vo E. CT. (o.(0), MAML FFnn al. 2017) differen- tiates through the inner updates, the implicit MAML (Rajeswaran et al. 2019) approximates local curvatures, while we differentiate through the "lazy" teacher's one-step "leap". The exploratory student may make many steps of inner updates before the teacher's "leap".
Figure I:
where 3 is learning rate, and dh (f) indicates the dependency on the meta-model 0. The gradient must backpropagate through the chain of updates in eq. (2). which has to be short (e.g.. k 1) to avoid big memory footprints, high-order derivatives, and the risk of vanishing or exploding gradients.
We say MAML is "greedy" that descends meta-gradients for the meta-model 0 before it runs adequate updates to the task-specific model db. As an increasing number of works adopt the gradient- based "search space carving" for task specisc models (Li et al. 2017 Rajeswaran et al. 2019 Park & Oliva 2019 [leennehhag al.) 2019 Yin et al. 2019), they also bear greedy algorithms, Relaxing the greedy strategy may benefit not one, but a variety of, meta-learning methods and tasks.
3 A "LAZY" APPROACH TO GRADIENT-BASED META-LEARNING
In this section, we describe a "lazy" meta-learning approach, which is readily applicable to different gradient-based meta-learning algorithms. We first describe the general approach an improvement to MAML and then customize it for few-shot learning, long-tailed classification, and meta-attack.
3.1 GENERAL APPROACH
Given a meta-model 6, we "carve" the search space for specifcci models M(0, Dtr) by teacher-studen scheme. The key idea is to let student explore the search space adequately using the training set task-specific model without worrying the length of the update chain because a teacher will examine the explored regions by the student, followed by one-step "leap". Hence, one can update the meta-model by backpropagating gradients through the teacher's "leap", not the student's update chain (ignoring that the chain starts from the meta- vocta.mode). Figure iillutrrats the main idea.
An exploratory student acts exactly the same as the gradient- based updates in MAML except that explores the feasible space by large number of steps (k 10), resulting in & + checkpoints of task-specific model á E .MMAML (0,, (, Drr) = { j = 0, ... k}. is clear from Section 2 that we cannot backpropagate the meta- eeta-gradients through the long chain of checkpoints, Φu. Pk, made by the exploratory student.
lazy teacher sits the initialization inn 0 till the student stops. It then takes "leap" towards the region explored by the student. The teacher essentially defines another "carved search space" for the task-speeific model é
MiAzy (t)) := 70 + (1 Rx-
(4)
where E [0, 1). The region Rx.b. is convex hull of the last checkpoints the student visited:
R = 0% -4- + 016-++20k- 6+2 + agdk.
(5)
where the coefficients {a} nen-negative and their sum equals 1, i.e., 04-641 The last b checkpoints presumably cover high-quality task aask-specific model ø by a better chance than the first few checkpoints. We shall experiment with b = 3 and b 1.
as 1.
3

---

Under review as a conference paper at ICLR 2021
Any task-specific model Φ in this "lazy" space MLAZY (9, Da-) is determined by the hyper- srperparameters y and ag-b+1;" ... aj over which we conduct a grid search to minimize the validation loss C (Ø). This is similar in spirit to meta-SGD (Li et al. 2017), which uses the validation data to search for the hyper-parameter of learning rates.
Denote by 50 + (1 - (1 the task-specific model as a result of the grid search. Notably, it is only one hop away from the meta-model 0, making easy to compute meta-gradients. Concretely, the meta- gradient descent for the meta model / becomes < 0 -P P..ITT
V VoLT... pat (50 + - ())).
which is apparently more manageable than the gradients in eq. (3) when h > 1.
Algorithm presents our "lazy" approach in detail. In the outer while-loop, we sample a batch of tasks {Ti) (Line 3, or L.3) and use them to make a gradient update to the meta meta-model f (L13). All task-specific mod- els {oro) are initialized to the current meta- model (L6). For each task Tr the student first runs gradient descent with respect to the task-specific model Φ; up to k steps (L8), and the teacher then takes "leap" from the initial meta-model 0 according to the checkpoints visited by the student (L10-11).
Algorithm "I.azy" Meta-Learning
Require: A distribution over tasks Pr
Require: Learning rates 17, B
Ensure: The meta model 0
I: Randomly initialize the meta-model 0
2: while not done do
3: Sample a batch of tasks {T; ~ Pr}
for all [T,} do
5:
Sample data Der and Doal for T; Φ.,0 0
7:
for = 1,2, 2,. , do
//student
8:
Φi + 01 1/ C (0t,-1) end for
Remarks. Our "lazy" teacher is motivated by the lookahead optimizer (Zhang et al. 2019). They have some key differences as follows due to the meta-learning setup. We initialize mul- tiple task-specific models by the meta-model. Moreover, we dynamically choose the "leap" rate ? by validation set. Finally, the val- idation data allows us to take advantage of not one checkpoint, but region around the checkpoints visited by the student.
10:
Grid-search MLAzY Der) such that CT is minimized at 9;0 + (1 -9:); //teacher
11:
6,(0) + 20 + (1 9) //teacher
12:
end for
13:
0 <- 0 N E, C (b)(
14:
end while
The teacher's role is similar the skip connection in ResNet (He et al. 2016). They bridge two otherwise distant points such that gradients can effectively propagate between them.
We share the same goal, to make MAML less "greedy", as the recently proposed implicit gradients (iMAML) (Rajeswaran et al., 2019). iMAMI. changes the lower-level problem in eq. (T) to an fz- regularized problem, which lends analytical expression for the meta-gradient. But it is expensive to compute and has to be approximated by a conjugate gradient algorithm. The Eg regularization also falls short in capturing structural relations between meta-model and task-specific models.
3.2 FEW-SHOT LEARNING, -AIGGT CLASSIFICATION AND
Since the "lazy" teacher does not change the innermost loop of gradient-based meta-lcarning - it instead "Icaps" over the chain of updates to the task-specifie model Φ. we can apply it to different algorithms. We evaluate it on few-shot learning, long-tailed classification, and meta-attack, in which meta-learning based methods have led to state-of-the-as art results.
Few-shot learning in this paper concerns an N-way- Khot classification problem. To customize Algorithm for this problem, we randomly select N classes for each task T, and then draw from each class K examples with labels, K of which are assigned to the training set DDr and one is to the validation set D Besides, we choose the hyper-paramenn classification accuracy on the validation set, instead of the loss in L10, Algorithml !
7/8 by using the task-specific model's
There is an interesting "trap" few-shot learning, identified as over-fitting by memorization (Yin) et al. 2019). The tasks } drawn from distribution PT are supposed to be i.i.d., but they could be correlated in the following scenario. Suppose there exists a global order of all classes. If we maintain this order among the N classes in each task, the meta- meta- could over-fit the tasks seen during meta- mett-ttainnng by memorizing the functions that solve these tasks, and it would fail to generalize to new tasks. Hence, it is important randomly shuffle the N classes every time we sample them for
4

---

Under review as a conference paper at ICLR 2021
Algorithm 2 "Lazy" Two- Componentnt Weighting for Long- Tailed Recognition
Require: A training set Der whose class frequency is long-tailed, balanced validation set Deal
Require: Class-wise weights {wy) estimated by using Cui et al. (2019)
Require:
Learning rates v), 7, pre-training steps 4, fine-tuning steps l2
1: Train a recognition network, parameterized by 0, for I steps by a standard cross-entropy loss
2: for = t1 1....
do
3: Sample a mini-batch B from the training set Dr
4: Set <- 0, E B, and denote by c c== {G, i E B}
5: Compute LB c) = [ Eien(" (uy + ()) //C, is cross-entropy over the i-th input
6: Update ((c) < 0 V (O,
// The "lazy" teacher, which depends on
7:
Initialize student model by setting do ((c)
8:
for j = ....... do
9:
Update the student model by gradient descent Φ5 +- Φj V end for
10:
11: Grid search for r the teacher's "leap", -((c) + k) yields high accuracy vall
Update t c -V..D... (70(c) (1 7)(x)
13:
Compute LB(0, e) (cf. Line 5) and update t - -VVooL((
end for
task (e.g., "dogs" and "cats" are respectively labeled as and in a two-way classification task, and yet they are shuffled and 0 in another two-way task).
We will empirically show that our approach is less prone to over-fitting than MAML even without
class shuffling. possible reason that use longer chains of updates (øo. .... Øk k 10) to learn the functions that solve the individual tasks, making them harder to memorize.
Long-tailed classification emerges
inevitable challenge as object recognition makes progress toward large-scale, fine-grained classes (Van Horn et al. 2018; Weyand et al. 2020), which often exhibit a long-tailed distribution. To uplift infrequent classes, Jamal al. (2020) propose weigh each training example by two components, a fixed component W' to balance different classes (Cui al., 2019) and trainable component 6,. We improve their learning method by a "lazy" teacher, as described in Algorithm 2 It alternatively optimizes the per-example weight 6; (using balanced validation set) and a recognition network 0 (using the long-tailed training set), in the same spirit as meta learning (cf. Algorithm vs. L5- 12 in Algorithm 2). We insert "lazy" teacher model to L6, let it take a "leap" in L12, and then backpropagate the gradient with respect to the per-example weight e; through the "leap".
Meta-attack (Du et al. 2019) a query-efficient blackbox attack algorithm on deep neural networks. manipulate image recognition network's predictions by
Recent work has shown that adding very small perturbations to benign inputs. However, if the network's architecture and weights unknown (blackbox), takes a large number of queries into the network to find valid adversarial example. To improve the query efficiency, Du et al. (2019) propose to learn a meta-model from many whitebox neural networks and then generalize to blackbox attacks. They train this meta-model by using the same meta-leann veta-learning framework as Algorithm|] Therefore. it is straightforward to improve their inner loop by our "lazy" teacher, we postpone the detailed algorithm Appendix B
EXPERIMENTS
We evaluate the "lazy", long-horizon meta-learning approach by plugging it into different algorithms with applications to few-shot learning, toiled recognition, and meta-attack.
4.1 FEW-SHOT LEARNING
We experiment with four datasets for few-shot learning: Omniglot (Lake et al. 2011). Minilma- geNet (Vinyals et al. 2016b), TieredImageNet (Ren al. 201 and CIFAR- FF$ (Bertinetto et 2018). The experiment protocols and implementation details largely follow MAML (Finn et al. 2017 and Reptile (Nichol et al. 2018). Please refer to Appendices A.] and A.2 for more details.
5

---

Under review as a conference paper at ICLR 2021
Our approach permits long- horizon inner updates and involves a con- vex hull of the last few checkpoints. In Table T we first experiment with the last b=3 and b=1 check- points. We test them with two representative meta-learning algo- rithms: MAML (cf. Algorithm 1 and Reptile (replacing Line 13
Table 1: Our approach applied MAML and Reptile for five-
way few-shot classification on MinilmageNet (Accuracy + 95% confidence interval over 2000 runs)
MinilmageNet
Method
s-shot
5-shot
MAML (Finn et al.
2017
48.70 = 1.84
63.11 = 0.92
"Lazy" MAML (b- = =1
48.26 = 1.78
64.13 + 1.90
"Lazy" MAML. (b 3)
48.17 + 1.84
63.73 + 1.10
(L13) in Algorithm [ with 0 + 0-3Σ, (0)): The intervals are 0.05 in the grid search (L.10,, and the search range for the learn- ing rate y is between 0.75 and 0.95.
Reptile (Nichol al. 2018
49.97 = 0.32
65.99 + 0.58
"Lazy" Reptile (b- 1)
51.50 = 1.00
67.22 + 0.97
"Lazy" Reptile (b 3)
52.67 = 1.01
68.77 + 0.98
Table I shows that there is no significant difference between 6 = 3 and b == 1, so we shall employ b = 1 for the remaining experiments. Moreover, the "lazy" variation improves the vanilla Reptile, but not MAML, probably because the five-way one/five-shot learning too simple for MAML to take advantage of the long-horizon inner updates. We next study many-way few-shot learning tasks, which are arguably more complex.
4.1.1 MAML VS. "LAZY" MAML FOR MANY-WAY FEW-SHOT LEARNING
We switch to the Taeee dataset since there are only 20 classes in MinilmageNet's meta-test set. The left panel of Figure 2 shows the results of MAML and "Lazy" MAML for N-way-fiv-shoot learning, where N varies in [5, 20, 30, 50}, and the student runs for k = 10, 15, 20, 20 inner steps, respectively. The "lazy" variation on par with MAML for the five-way classification, and significantly outperforms MAMI. for 20-way, 30-way, and 50-way five-shot classifications. This trend indicates that the many-way few-shot learning problems desire more inner updates to the task-specific models, amplifying the benefit of the "lazy" teacher.
MAML
Ours (MAML)
70.6741.72
45
70
69.60m1.75
40
38.3710.4
Mean Accuracy (%)
60
Mean Accuracy
35
30
550
25
40
36:1110.42
20
30
31.9280
25.9589-33
26,0420.34
15
9.622.3
19.750.2
10
7.810.2
20
17.7210.2
5-way
20-way
30-way
50-way
MAML
TAML
Ours (MAML)
Figure 2: Left: Mean Accuracy (%) for N N-way-fivesshot classification on mageee Right:
Mean Accuracy (%) for 20-way-one-shot non-i.i.d (Yin et al. 2019) classification tasks on Omniglot.
4.1.2 "LAZY" MAML IS LESS PRONE TO FVER-FITTING BY MEMORIZATION TIIAN MAML
The right panel of Figure 2 shows some 20 20-way-on -ne-sote classification results on Omniglot when we learn from non-i.id. tasks. L... by maintaining a global order of all training classes. This global order creates a shortcut for meta-learning methods; they may memorize the order from the meta-training tasks and fail to generalize to meta-test tasks (Yin et al.. 2019). We can see that the "lazy" teacher boosts MAML by large margin and outperforms TAML (Jamal & Qil 2019), indicating that it is less prone to over-fitting by memorization. A plausible reason is that the k = 15 steps taken by the exploratory student make it harder to memorize than the one-step update in MAML or TAML.
In Appendix A we present more results of the few-shot learning. Section A.3
investigates the proposed "lazy" approach with Reptile-style update for N wwwy-five-shee learning TieredImageNet. Section A.4 further compares MAML and "lazy" MAML by their computation memory costs. Section contrasts "lazy" MAML and "lazy" Reptile prior arts all the four datasets.

---

Under review as a conference paper at ICLR 2021
4.2 LONG-TAILED CLASSIFICATION
Following the experiment setup in (Cui et al. 2019) and (Jamal et al. 2020), we use the CIFAR- LT-100 dataset (Cui al. 2019) compare Algorithm 2 with several long-tailed recognition methods. Cui et al. (2019) created multiple long-tailed datasets by removing training examples from CIFAR-100 100 (Krizhevsky & Hinton 2009) according different power law distributions, In each version, we compute an imbalance factor as the ratio between the sizes of the head class and the tail class. We run
steps the innermost loop of Algorithm 2
Table 2: Test top-1 errors (%) of ResNet-32 CIFAR-LT- 100 under different imbalance settings.
Method +
Imbalance factor +
200
100
50
20
Standard cross-entropy training Class-balanced cross-entropy training (Cui et
65.16
61.68
56.15
48.86
2019
64.30
61.44
55.45
48.47
Class-balanced fine-tuning (Cui et al. 2018
61.78
58.17
53.60
47.89
Learning to reweight (Ren al. 20185)
67.00
61.10
56.83
49.25
Meta-weight (Shu et al. 2019)
63.38
58.39
54.34
46.96
Two-component weighting (Jamal al. Lazy two-component weighting (ours)
2020)
60.69
56.65
51.47
44.38
58.67
53.46
48.24
43.68
Table 2 shows the test errors (%) under different imbalance factors. We can see that our teacher- student scheme boosts the original two-component weighting approach (Jamal al. (0200) under all the imbalance factors. The results are especially interesting in that Algorithm 2 is not exactly meta-learning method, though it shares the same framework as the gradient-basse meta-learning due to the two nested optimization loops. Besides, compared with the other competing methods, our results establish new state of the arts for the long-tailed object recognition.
4.3 META-ATTACH
We evaluate the "lazy" meta-attack MNIST (LeCun et al. 1998) and CIFAR-10 (Krizhevsky & Hinton 2009). We follow (Du et al. 2019) for the experiment setup and all training details, including the network architectures used to generate gradients for input images, the attack models, meta-attack models, and evaluation metrics for both the datasets, to name few. The learning rates in the inner and outer loops are both 0.01. We let the student run le 8 and Ic 10 steps in the innermost loop for MNIST and 11 respectively.
Table 3:
Untargeted adversarial attack results on MNIST and CIFARIO. We achieve comparable success rates and average & distortions with other methods by using a smaller number of queries.
Datasct Target model
Method
Success Rate
Avg. &y
Avg. Queries
Zoo (Chen al.
2017
1.00
1.61
21,760
Decision boundary al.
2018
13,630
Opt-attack (Cheng ct al. 2019
1.00
1.85
12,925
MNIST Net4
AutoZoom [Tu
2018
1.00
1.86
2,412
Bandits (Ilyas al.
2019
0.73
1.99
3,771
Meta-attack (Du al. 2019
1.00
1.77
749
Lazy meta-altack (ours)
1.00
1.65
566
700 (Chen al.
2017
1.00
0.30
8,192
Decision boundary
al.
2018
0.30
17,010
Opt-attack (Cheng et al.
2019
1.00
0.33
20,407
CIFARIO / Resnet18
AutoZoom [Tu al. 2018
1.00
0.28
3,112
Bandits (Ilyas al.
0.91
0.33
4,491
FW-black (Chen ct al.
201
1.00
0.43
5,021
Mcta-attack (Du et
2019
0.94
0.34
1,583
Lazy meta-allack (ours)
0.98
0.45
1,061
Table 3 shows the results of untargeted attack. namely, the attack is considered successful once it alters the recognition network's 's prediction to any incorrect class. Appendix B includes the results of targeted attack. In addition the original meta-attack (Du et al. 2019). Table also presents

---

Under review as a conference paper at ICLR 2021
several existing blackbox attack methods for comparison. We can that meta-attack and our "lazy" meta-attack yield about the same success rates as the other blackbox attacks. The second-to-the-e right column is about the average & distortion an attacker makes to an input, the lower the better. The rightmost column is about the number of queries an attacker makes into the recognition network, the lower the better. The "lazy" meta-attack is able achieve comparable success rates and lz distortion rates with the other methods yet by using a smaller number of queries. Both meta-attack and its "lazy" version significantly outperform the other methods in terms of the query efficiency, indicating the generalization capability of the meta-attack model from known whitebox neural networks to unknown blackbox networks.
RELATED WORK
Meta-learning has been a long-standing sub-field in machine learning (Schmidhuber 1987 Thrun &
Prati 1998 Naik& & Mammone 1992). Early approaches update a model's parameters by training a meta-learner (Bengio ct al.) 1995] Bengio et al. [991]] Schmidhuber 1992). This has been well studied in optimizing neural networks, and one such family meta-learning learns an optimizer (Ravi) & Larochelle| 2016 Li & Malik 2016 Andrychowicz et al. (016). A specialized neural network takes gradients as input and outputs an update rule for the learner. In addition to the update rule, Ravi & Larochelle (2016) also learn the weight initialization for few-shot learning. Finally, there are
several approaches (Metz et al. 2019, Wichrowska et al. 2017) for training generic optimizers that can be applied broadly to different neural networks and datasets.
Under the context of few-shot learning, another family of meta-learning involves metric-lcarning based methods (Vinyals et al., 2016al Snell et al.) 2017 Mishra al., 2017 Koch et al. [0155 Oreshkin et al. 2018). which learn a metric space to benefit different few-shot learning tasks. The goal is to find the similarity between two samples regardless of their classes using some distance metric so that the similarity function can be used to classify the unseen classes at the test stage. Some recent studies along this line include Matching Networks (Vinyals et al., 2016a), which employs the cosine similarity, Prototypical Networks (Snell et al. 2017). which uses the Euclidean distance to compute the similarity, Relation Network (Sung et al. (017]. which uses a relation module as the similarity function, ridge regression (Bertinetto et al. 2018). and graph neural networks (Satorras & Estrach 2018).
More recently, gradient- meta-learning gains its momentum, and a variety of methods have been proposed in this vein. The most notable one among them might be MAML (Finn et al., 2017)) where the goal is to learn the network weight initialization so that it can adapt to unseen tasks rapidly, There have been extensions to improve MAML. Meta-SGD (Li et al. 2017) learns the learning rates along with the weight initialization. Regularization techniques Yin et al. 2020 Jamal & Qi 2019) are introduced to MAML to mitigate over- fitting. (Park& & Oliva 2019) preconditions on the gradients in the inner loop by learning curvature. Despite MAML's popularity, it is still computationally expensive and consumes large memory due to the computation of high-order derivatives. The authors show that the first-order approximation, which neglects the gradients of the inner loop during meta-optimization, performs about the same as the original MAML. Another first-order meta- learning method is Reptile (Nichol et al. 2018), which decouples the inner and outer optimization steps. iMAML (Rajeswaran et al.) 2019) provides an approximate solution for meta- -rttdinggsnnts by using an algorithm based on conjugate gradients, and its low-level optimization is similar to Meta- MinibatchProx (Zhou et al. 2019). The idea is to add an &2 regularizer in the inner loop, allowing the updated parameters close to the initial parameters.
CONCLUSION
We propose a teacher-student scheme for the gradient- based meta-learning algorithms to allow them run more steps of inner updates to task-specific models while being immune to the risk of vanishing or exploding gradients. The student explores the tasks-specific model's feasible space up to many steps, and the "lazy" teacher takes one-step "leap" towards the region explored by the student. As result, the teacher defines a lightweight computation graph and yet it takes advantage of the adequately explored checkpoints by the student. This approach is generic; include few-shot learning, long- tail recognition, and meta-attack and various meta-learning methods. Experiments verify the benefit of long-horizon inner updates in gradient-based meta-learning.
apply it to different problems,
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Wieland Brendel ,, Jonas Rauber *, and Matthias Bethge. Docision-based adversarial attacks: Reliable attacks against black-box machine learning models. In International Conference on Learning Representations, 2018.
Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. In Advances in neural information processing systems, pp. 3981-3989, 2016.
Samy Bengio. Yoshua Bengio, Jocelyn Cloutier, and Jan Gecsei. On the optimization of a synaptic learning rule. In Optimality in Biological and Artificial Networks. 1995.
Y. Bengio, S. Bengio, and J. Cloutier. Learning synaptic learning rule. In IJCNN-9I-Seuttle International Joint Conference on Neural Networks, 1991.
Luca Bertinetto, João F. Henriques, Philip H. S. Torr, and Andrea Vedaldi. Meta-learning with differentiable closed-form solvers. CoRR, 2018.
Jinghui Chen, Dongruo Zhou, Jinfeng Yi, and Quanquan Gu. A frank-wolfe framework for efficient and effective adversarial attacks, 2018.
Pin- Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsich. Zoo. Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security - AlSec '17, 2017.
Minhao Cheng, Thong Le. Pin- Yu Chen, Huan Zhang, JinFeng Yi, and Cho-Jui Hsich. Query-
efficient hard-label black-box attack: An optimization-based approach. International Confer-
ence on Learning Representations, 2019. URL /tttts::/opeerevllle ew. net. www.:t/fouuettmmmemeeeeemmm ffrummrrrmmmo/o/
rJ1k6iRqKX
Benoît Colson, Patrice Marcotte, and Gilles Savard. An overview of bilevel optimization. Annals of operations research, (33(1)2335256 2007.
Yin Cui, Yang Song, Chen Sun, Andrew Howard, and Serge Belongie. Large scale fine- Ane-grained categorization and domain-specific transfer learning. In The IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2018.
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song. and Serge Belongie. Class- balanca loss based on effective number of samples. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 9268-92620 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Jiawei Du, Hu Zhang, Joey Tianyi Zhou, Yi Yang, and Jiashi Feng. Query-efficient meta attack to deep neural networks. arXiv preprint arXiv: 1rrXiv:9906..0 02398, 2019.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. agdel-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Bharning-Volumee 70, pp. 1126-1135. JMLR. org, 2017.
Sebastian Flennerhag, Andrei A Rusu, Razvan Pascanu, Hujun Yin, and Raia Hadsell. Meta-learning with warped gradient descent. arXiv preprint 1XXx:1909.00025 2019.
Marta Gamelo, Dan Rosenbaum, Chris J Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Eslami. Conditional neural processes. arXiv preprint 1rXiv:1807.01613. 2018
Jonathan Gordon, Wessel Bruinsma, Andrew YK Foong, James Requeima, Yann Dubois, and Richard E Turner. Convolutional conditional neural processes. In ICI.R, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattedrn recognition, pp. 770-778, 2016.

---

Under review as a conference paper at ICLR 2021
Andrew Ilyas, Logan Engstrom, and Aleksander Madry. Prior convictions: Black-box adversarial attacks with bandits and priors. In International Conference on Learning Representations. 2019.
Muhammad Abdullah Jamal and Guo-Jun Qi. Task agnostic meta-lcarning for few-shot learning. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Muhammad Abdullah Jamal, Matthew Brown, Ming-Hsuan Yang, Liqiang Wang, and Boqing Gong. Rethinking blass-balanced methods for long-tailed visual recognition from domain adaptation perspective, 2020.
Gregory Koch, Richard Zemel, and Ruslan Salakhutdinov. Siamese neural networks for one-shot image recognition. 2015.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009.
Brenden Lake, Ruslan Salakhutdinov, Jason Gross, and Joshua Tenenbaum. One shot learning of simple visual concepts. In Proceedings of the annual meeting of the cognitive science society, volume 33, 2011.
Yann LeCun, Corinna Cortes, and Christopher JC Burges. The mnist database of handwritten digits, 1998. URL htlp://yann, lecun. com/exdb/mnistt, 10:34, 1998.
Ke Li and Jitendra Malik. Learning optimize, 2016.
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few-shot learning, 2017.
Luke Metz, Niru Maheswaranathan, Jeremy Nixon, Daniel Freeman, and Jascha Sohl-dickstein. Learned optimizers that outperform on wall-clock and validation loss, 2019.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Picter Abbeel. A simple neural attentive meta- learner. arXiv preprint arXivv 2017.
D. K. Naik and R. Mammone. Meta-neural networks that leam by learning. In (Proceedings [992] IJCNN International Joint Conference on Neural Networks, 1992.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint 1rXiv:1803. 02999, 2018
Boris N. Oreshkin, Pau Rodriguez, and Alexandre Lacoste, Tadam: Task dependent adaptive metric for improved few-shot learning, 2018.
Eunbyung Park and Junier B Oliva. Meta-curvature. In Advances in Neural Information Processing Systems, pp. 3309-33111 2019.
Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-learning with implicit gradients. In Advances in Neural Information Processing Systems, pp. 113-124, 2019.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. 2016.
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua Tenenbaum, Hugo Larochelle. and Richard S Zemel. Meta-learning for semi-supervised few-shot classification. arXiv preprint 1803. 00676, 2018a.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning, arXiv preprint arXiv: 000.8000000000 2018b.
Victor Garcia Satorras and Joan Bruna Estrach. Few-shot learning with graph neural networks. In International Conference on Learning Representations, 2018.
Jurgen Schmidhuber. Evolutionary principles in self-referential learning. learning to learn: The meta-metta-meta.-hhook Diploma thesis, 1987.
Jürgen Schmidhuber. Learning to control weight memories: An alternative to dynamic recurrent networks. 1992.
10

---

Under review as a conference paper at ICLR 2021
Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Meta-weight- net: Learning an explicit mapping for sample weighting, 2019.
Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. 2017.
Flood Sung, Yongxin Yang. Li Zhang, Tao Xiang, Philip H. S. Torr, and Timothy M. Hospedales. Learning compare: Relation network for few-shot learning, 2017.
Sebastian Thrun and Lorien Pratt. Learning to Learn: Introduction and Overview. Springer US, 1998.
Chun-Chen Tu, Paishun Ting, Pin- Yu Chen, Sijia Liu, Huan Zhang, Jinfeng Yi, Cho-Jui Hsieh, and Shin-Ming Cheng. Autozoom: Autoencoder-basee zeroth order optimization method for attacking black-box neural networks, 2018.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pictro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 7779-8778 2018.
Ricardo Vilalta and Youssef Drissi. A perspective view and survey of meta-learning. Artificial intelligence review, 18(2):77-95, 2002.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, and Daan Wierstra. Matching networks for one shot learning. Advances in neural information processing systems, pp. 3630-3638, 2016a.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems, pp. 3630-3638 2016b.
Tobias Weyand, Andre Araujo, Bingyi Cao, and Jack Sim. Google landmarks dataset v2-a large-scale benchmark for instance-level recognition and retrieval. arXiv preprint arXiv:2004.01804 2020.
Olga Wichrowska, Niru Maheswaranathan, Matthew W. Hoffman, Sergio Gomez Colmenarejo. Misha Denil, Nando de Freitas, and Jascha Sohl-Dickstein. Learned optimizers that scale and generalize, 2017.
Mingzhang Yin, George Tucker, Mingyuan Zhou, Sergey Levine, and Chelsea Finn. Meta-learning without memorization. arXiv preprint arXiv: 1912:3912.008822 2019.
Mingzhang Yin, George Tucker, Mingyuan Zhou, Sergey Levine, and Chelsea Finn. Meta-learning without memorization. In International Conference on Learning Representations, 2020. URL
https://<penreviie wwt.:neeeooorreeeee
Michael Zhang, James Lucas, Jimmy Ba, and Geoffrey Hinton. Lookahead optimizer: k steps forward, step back. In Advances in Neural Information Processing Systems, pp. 9604,9604, 2019.
Pan Zhou, Xiaotong Yuan, Huan Xu, Shuicheng Yan, and Jiashi Feng. Efficient meta learning via minibatch proximal update. In Advances in Neural Information Processing Systems, pp. 1542, 2019.
APPENDICES
A FEW-SHOT LEARNING
A.1 DATASETS
We experiment with four datasets for few-shot learning: Omniglot (Lake al. 2011). Minilma- geNet Vinyals et al. 2016b). TieredmmageNet (Ren et al. 8a). and CIFAR-FF FS (Bertinetto et al. 2018). The Omniglot dataset consists of handwritten characters from 50 different alphabets and 1623 characters. There are 20 handwritten examples of each character. MinilmageNet contains 100 classes form ImageNet (Deng et al. 2009). which are split to 64, 16, and 20 classes for meta-training,
11

---

Under review as a conference paper at ICLR 2021
meta- valldaalii,tion, and meta-test, respectively. Taered has 608 classes from ImageNet, which are grouped into 34 higher-level categories following the ImageNet taxonomy. They are split into 20 meta-training categories, 6 meta-validation categories, and 8 meta-test categories. Due to this partition scheme, the meta-test classes are less similar to the meta-training classes in Tagred than in other datasets. CIFAR-FS re-purposes CIFAR-100 (Krizhevsky & Hinton 2009), splitting its 100 classes into 64, 16, and 20 classes for meta-training, meta-validation, and meta-test, respectively.
A.2 EXPERIMENT PROTOCOLS AND LYPER-PARAMETER
Our experiment protocols and implementation details largely follow MAML (Finn et al. 2017) and Reptile (Nichol ct al.) 2018). In particular, four modules in all the experiments. Each module has 3x3 convolutions, a batch normalization layer, 2x2 max-pooling, and the ReLU activation, and every convolutional layer contains 64 filters for the experiments on Omniglot and 32 filters for other datasets. For fair comparison, we also re-implement some of the existing methods using this network architecture. We report more details for the "Lazy" Reptile in Table 4 For "Lazy" MAML, we set k: = 10 for I 1-shot and 5-shot tasks, respectively, on both Minilmage Net and Tieee.
convolutional neural network that comprises
Table 4: Hyper-parameter details for few-shot learning in ours (Reptile). The "Eval inner batch" row shows the numbers for both 1-shot and 5-shot settings.
Hlyee-ppaameete
Omniglot
CIFAR-FS
Mini-ImageNet
TieredImageNet
Inner learning rate (n)
0.001
0.001
0.001
0.001
Inner iterations (R)
5
8
Inner batch size
10
10
Training shots
10
15
15
15
Outer step-size (3)
1.0
1.0
1.0
1.0
Total outer-iteration
100k
120k
120k
130k
Meta batch size
20
20
20
20
Eval. inner iterations
50
50
50
50
Eval. inner batch
5/15
5/15
5/15
5/15
A.3 MANY-WAY CLASSIFICATION
We have presented many-way results on TieredImageNet for "Lazy" MAML in section 4.1.1 Here, we describe the implementation details for these experiments. We set the inner learning rate (1)) to 0.005 and the outer learning rate (3) to 0.001 for all the settings, We let the student run A = 15, 15 and 18 steps for way, way and 50-way, respectively during meta meta-training.
GPU Memory
Compute Time ( sec/iter )
MAML Ours (MAML)
MMML Ours (MAML)
Memory (GB)
2
0
0.00
8
10
12
num of inner steps
num of inner steps
Figure 3: Left: Memory trade-off's with 4 layer CNN 20-way-5-shot MinilmageNet task. b). Computation time (sec per meta-iteration) w.r.t the number of inner gradient steps on 20-way-5-shot MinilmageNet task.
12

---

Under review as a conference paper at ICLR 2021
A.4 COMPUTATIONAL ANALYSIS
In our evaluation, we also want to answer the following question empirically. How does the memory and computation requirements of "Lazy" MAML compare with MAML? Figure3Jsowws the memory and compute trade-off for "Lazy" MAML and MAML on 5-shot 20-way MinilmageNet. "Lazy" MAML decouples the dependency inner and outer loop by teacher-student scheme which allows to define very lightweight computation graph. The left panel of the figure shows that the memory of the "Lazy" MAML. doesn't exceed beyond GB for many inner gradient steps while on the other hand, MAML reaches the capacity of 12 GB after 5 inner steps. The right panel shows the computation time per iteration with respect to multiple gradient inner steps. The time taken by "Lazy" MAML doesn't it increase exponentially compared to MAML which takes more compute time and reaches the maximum capacity of memory after inner steps.
A.5 ADDITIONAL RESULTS ON MANY-WA LEARNING
The left panel of Figure 4 compares the results Reptile and "Lazy" Reptile for N-way-five-shot learning Taeee where N varies in {5, 20, 30). Our approach outperforms Reptile. We emphasize that not all meta-learning algorithms can be approximated by first-order version; for example, it is not immediately clear how to do it for Algorithm 2, the two-component weighting method for long-tailed classification, in the main text.
Reptile
Ours (Reptile)
40
72.2150.36
18.75688.4
34.2800.44
70
Mean Accurscy (96)
30
29.1510.22
Accurccy
25.380.3
50
25
40
07.5558.46
20
34,4750.40
30
15
25:1040.25 32
20
10
5-way
20-way
30-way
Ours (MAML)
Reptile
(Repile)
Figure 4: Left: Mean Accuracy (%) for N N-way-five-sieee classification on TieredImageNet. b). Mcan Accuracy (%) for (0-wway--sshie classification MinilmageNet.
The right panel of Figure 4 shows some 20-way-5-she results on MinilmageNet. We can see that lazy strategy boosts both MAMI. and Reptile by significant margin, which is similar to what we see in Figure of the main paper. again indicates that more training data needs more steps of exploration for task-specific model and hence magnifies the benefit of our teacher-student scheme introduced to both MAML and Reptile.
A.6 COMPARISON RESULTS
We compare "our" approach with state-of-the-art meta-learning methods for five-way few-shot learn- ing problems on four datasets. The results are shown in Tables and For our own approach, we study both the MAML-style update to the meta-model (ours (MAML), L13 in Algorithm [ and the Reptile-style Nichol et al., 2018) update (ours (Reptile), L14 Algorithm T for MinilmageNet and TieredImageNet. On Omniglot and CIFAR-FS, we only report ours (Reptile) due to its low computa- tion cost. Batch normalization with test data yields about 2% improvement over the normalization with the training data only, and we report the results of both scenarios.
can be seen that our results are better than or comparable with those of the competing methods. In general, the improvements by our teacher-student scheme are more significant on 5-shot settings than on I-shot settings, verifying the trend in Section 4.1. that more training data can better leverage the exploratory student in our method. Besides, ours (Reptile) outperforms ours (MAML) probably for two reasons. One is that ours (Reptile) uses more than & shots of training examples per class for a k-shot learning problem during meta-train veta-training, following the experiment setup of Reptile (Nichol et al.) 2018). The other is that the second-order gradients ours (MAML) make the training procedure less stable than Reptile. We hypothesize that many-shot setting would be less sensitive both
13

---

Under review as a conference paper at ICLR 2021
Table 5: Five-way few-shot classification accuracies (%) on MinilmageNet and Tagred The
+ shows 95% confidence intervals computed over 2000 tasks.
BN w/
Mini -ing-emm
TieeedlmageNet
Method
Test
1-shot
5-shot
1-shot
5-shot
MAML Finn et al.
2017
X
46.21
1.76
61.12 + 1.01
49.60 + 1.83
66.58 + 1.78
MAML Funn et al.
2017
48.70 + 1.84
63.11 = 0.92
51.67 + 1.81
69.60 = 1.73
Meta-Curvature Park Oliva
2019
48.83 + 1.80
62.63 = 0.93
50.30 + 1.99
66.14 = 0.95
iMAMI. (Rajeswaran et al.
2019
49.30 + 1.88
Ours (MAML.)
48.26 + 1.78
64.13 + 1.90
51.03 + 1.70
70.67 + 1.72
FOMAML (Finn et al.
2017
45.53
61.02 1 1.12
48,01 1.74
64.07 1.72
Reptile (Niholl et al.. 2018
47.07
62.74 0.37
49.12
0,43 49.97
65.99 0.42
Meta-MiitttcchPro (Zhou et Ours (Reptile)
2019
47.81 1.00
63.18 1.00
66.60 0.91
48. 0.94
64.64 + 0.92
0.95 51.15
68.84 0.90
FOMAML (Finn et al.
2017
48.07 + 1.75
63.15
50.12 1,82
67.43 = 1.80
Reptile (Nicoll al
2018
49.97 0.32
65.99
51.34 1: 0.4
68.73 1 0.40
Meta-Mmbatcchroox Zhou et al. 2019
50.08 + 1.00
0.98 66.28 0.98
53.71 + 1.04
69.78 - 0.95
Ours (Reptile)
51.50 1.00
67.22 - 0.97
54.41 + 1.00
72.21 = 0.94
Table 6: Five-way few-shot classification accuracies (%) on Omniglot and CIFAR-FS. The shows 95% confidence intervals computed over 1000 tasks.
BN w/
Omniglot
CIFAR-FS
Method
Test
1-shot
5-shot
1-shot
5-shot
MAML Finn et al. 2017)
98.70 + 0.40
99.90 1 0.10
56.50 + 1.90
70.50 + 0.90
iMAMI.
(Kajeswaran ct al.
2019
99.16 + 0.35
99.67 + 0.12
Reptile Nichol al.
2018
95.39 + 0.09
98.90 0.10
53.12 1.34
69.40
1.30
Ours (Reptile)
95.44 + 0.57
98.92 + 0.29
54.64 = 1.30
70,56 + 1.20
FOMAML FFin al. 2017
98.30 = 0.50
99.20 + 0.20
Reptile Nichol al. 2018
97.68 + 0.04
99.48 0.06
57.50
0.45
71.88 + 0.42
Ours (Reptile)
98.20 = 0.38
99.70 0.16
59.36 1.44
74.90 + 1.28
factors. Indeed, we verified this hypothesis by another five-way-50-shol learning experiment with ours (Reptile), which yields 76.17 = 0.32% on MinilmageNet and is lower than 78.54 = 0.70 by ours (MAML).
B META-ATTACK
Here, we formally present the algorithm of our lazy meta-learning approach to training the meta- attacker Algorithm As described the main paper that the original meta-attack [Du al. (2019) uses Reptile to train the attacker, so it is straightforward to improve the approach by using "lazy" teacher. The inputs to the meta-attacker are images, and the desired outputs are their gradients during meta-training, the gradients generated from different classification models. Instead of the cross-entropy loss, meta-attack adopts a mean-square error (MSE) loss in the inner loop, i.c.,
C ( Gyll2
where the task T; is to find adversarial examples for the inputs to the i-th pre-trained classification network, Xi is an image sampled for the task, Gij are the gradients of the classification network with respect to (w.r.t)) the image, and ((-) is a task-specific model whose output is to approximate the gradients Gij- This model is useful because, given blackbox classification network, can use the task taek-fpe model to predict the gradients this network w.r.t. an image, followed by gradient ascent towards adversarial example (cf. Algorithm 4
Algorithm presents how to train this meta meta-attacker by applying our "lazy" teacher to Reptile, and then follow Algorithm 4 for attacking blackbox networks Du al. (209)).
14

---

Under review as a conference paper at ICLR 2021
Algorithm 3 Training algorithm of meta-attack using "lazy" Reptile
Require: A distribution over tasks PT
Require: Input Images X' gradients 9; generated from a classification network serving task T;
Require: Learning rates o,
Ensure: The meta attacker
1: Randomly initialize the meta-attacker 0
2: while not done do
3: Sample a batch tasks {T; ~ Pr}
4: for all } do
5: Sample data Drr and Dat for Ti 02.0 +- 0
the form of {Zig. 9
6:
7:
for j = 1, ... k do
8:
Oi f- Qi.j- a C (0i.51)
9:
end for
10:
Ni 6- arg min, C DDnt (70 + (1 - (00-)))m (),k))
11:
(0) t 7;0 + (1 - 7)oi
end for
13:
- 0 - 3 -0-)(()
end while
Algorithm Adversarial Meta-Attack
Require: Test image To with label i, meta-attacker fe. target model Mtars iteration interval j. selected top-q coordinates
1: for t = 0, 1, ... do
2: if ()+ mod
then
Perform zeroth-order gradient estimation on top-y coordinates, denoted as I, and 4: obtain St-
3:
5:
Fine-tune meta-attacker fo with (Fe.9t on 1, by L [[fo(xe]1, 190],,12
6:
else
7:
Generate the gradient map di directly from meta-attacker fo with dz. select coordinates It-
8:
9:
end if
10:
Update (zr]|,, = [xe]r, + AgghT..-
if MMMar (x') # then
12:
Ladv = x
break
14: else
F4+11 x
16:
end if
17:
end for
Ensure: adversarial example Tado-
15

---

Under review as a conference paper at ICLR 2021
B.1 RESULTS UNDER TARGETED ATTACK
In 3f the main paper, we report the results under untargeted attack. Here, we are presenting the results under targeted attack for both MNIST and CIFAR-IO Table Similar untargeted attack, we achieve comparable results on success rate and average &g distortion using a smaller number of queries.
Table Comparison of several methods under targeted attack on MNIST and CIFAR-10. 10. Similar to the untargeted attack, we reduce the number of queries for meta attack.
Datasct Target model
Method
Success Rate
Avg.
Avg. Queries
Zoo Chen al.
22017
1.00
2.63
23,552
Decision Boundary et al. 2018
0.64
2.71
19,951
MNIST Net4
AutoZoom Tu et al. 2018
0.95
2.52
6,174
Opt-attack Cheng el al.
2019
1.00
2.33
99,661
Meta attack Du et al. 2019
1.00
2.66
1,299
Lazy meta-attack (ours)
1.00
2.63
1,108
Zoo Chen et al.
22017
1.00
0.55
66,400
Decision Boundary et al.
2018
0.58
0.53
16,250
CIFARIO Resnet18
AutoZoom Tu al. 2018
1.00
0.51
9,082
Opt-attack Cheng el al. 2019
1.00
0.50
121,810
FW-hlack Chen et al. 2018
0.90
0.73
6,987
Meta attack Du ct al. 2019
0.93
0.77
3,667
Lazy meta-altack (ours)
0.92
0.69
3,092
16


Reviews:
[{"review_id": "ly2f_YYq7R", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\n\nThe paper proposes a method to address the short horizon limitation of gradient based meta-learning algorithms. To avoid the multiple problems of back-propagating the gradient through inner loops with many steps, the authors propose a teacher-student solution that decouples the exploration (done by the student) from the actual computation of the gradient (done by the teacher). In particular, the inner loop is run using the student on the training set. The teacher then optimises the validation loss using grid search in a reduced search space defined by the convex hull of last points explored by the student and the common initialization. Finally, the metagradient is taken with respect to the initialization avoiding the backpropagation through the inner loop. In the experimental section, the authors apply the methods in three different use cases (few shot learning, long-tail classification and meta-attack)\n\n## Comments\n\nThe paper is well motivated and the writing is clear.\n\nFrom the technical perspective, the model proposal seems reasonable: focus on the gradient information toward the end of the trajectories defined by the task specific inner loops. One of the confusing aspect of the paper is that despite the general description of the method based on the convex hull of the last \"b\" updates of the inner loop, in the end only the last update is used (according to the remark at the end of section 4.1). If there is not advantage in performance by taking b>1 and assuming there is a significant cost in increasing b (the computational complexity of grid search grows exponentially with the dimensionality b), why to include this extension in the first place? Unless I am missing something it may make sense to start with the description of the simpler algorithm that uses only the last update. \n\nI would also encourage the authors to elaborate on the similarities/differences between Reptile and their method in the same way they do it with MAML. In particular, when b=1 their method seems to be pretty close to Reptile (reptile uses \\theta \\leftarrow \\theta + \\epsilon(\\widehat{\\phi} -\\theta) = (1 - \\epsilon) \\theta + \\epsilon \\widehat{\\phi}), and so a deeper analysis of why the results are better would add value to the paper.\n\nRegarding the experimentation, i would recommend the authors to move some of the comparisons with state-of-the-art methods from the appendix back to the manuscript. In particular, given that it is mentioned in Figure 1, I would include iMAML results in the manuscript rather than the appendix. Also, in the experimental setting, it is unclear why iMAML and FOMAML are incomplete in tables 5 and 6.\n\nFinally, one of the relevant baselines that the authors missed is [1], which also addresses the short horizon bias by minimising the total length of the inner trajectories avoiding higher-order derivatives.\n\n## Minors\n\n* Figure 2: TAML is not defined\n* Figure 1: The figure is quite confusing and it is not clear what the authors try to convey (e.g. should not be the task specific gradient be pointing in the same direction across the three plots?). Why is iMAML included but not Reptile, when it is actually Reptile and not iMAML the one baseline that appears in the main manuscript.\n* At the end of page 4, the authors mention the importance of shuffling the N classes and then that their method performs better than MAML even without shuffling. Does that mean that all the experiments (for all methods) are done without shuffling? How do the results vary when shuffling is added?\n* Equation 4: k-b+1...k should be k-b+j for j in {1,k}\n\n## References\n[1] Transferring Knowledge across Learning Processes", "extra_scores": {}, "extra_text": {}}, {"review_id": "jVB_JazlDEn", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Computation of higher-order gradients is a major bottleneck in gradient-based meta-learning algorithms like MAML. This work proposes a simple approximation algorithm to alleviate the issue. They use a student network to explore the search space of task-specific models. Unlike MAML, the computation graph of the task-specific adaptation is not tracked and this allows the number of task-specific updates to be larger. Once the task-specific adaptation is done, they simply take a convex combination of the task-adapted parameters and the original parameters and evaluate the resulting model on the validation data of the task. The validation error is then backprogated for training. The proposed approach seems to be applicable to different meta-learning scenarios including few-shot learning, unbalanced classification and meta-attack. On the unbalanced classification and the meta-attack benchmarks, they showed marginal improvements over existing methods.\n\nPros:\n- The paper is well written and clear\n- They evaluated the algorithm on multiple different meta-learning problems\nStrong results on the unbalanced classification and the meta-attack benchmarks \n\nCons:\n- Although the approach is motivated by the look ahead optimization, it feels a bit unintuitive to take a simple convex combination of the adapted and the original parameters for the sake of skip connection\n- I am not fully convinced that the method alleviates the short-horizon biases of the gradient-based meta-learning algorithm. The meta-update still seems to favor a solution that is closest to the original model due to the skip connection.\n- Evaluation on multi-shot learning benchmarks can make the work strong\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "HkmeVtPZQAQ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the problem setting of gradient based meta learning (GBML) and presents a new algorithmic approach that is computationally tractable. The proposed approach follows a two time-scale algorithm, where the inner level takes multiple gradient steps of the task loss function, while the outer level takes smaller \"leaps\" following a weighted average of the inner level iterates. Experimental results are provided for few-shot classification and query efficient \"meta attack\" on deep networks which suggest that the proposed algorithms perform favorably when compared to baselines.\n\n**Assessment**\n\nThe proposed algorithm and motivations are interesting. The experimental results are convincing. However, the writing lacks clarity (some of these are below). Overall, my recommendation is borderline. I am happy to reconsider based on author response.\n\n**Questions and Comments**\n\n(1) Having very long inner loops can lead to divergence in the bi-level optimization setup, since the inner level parameters are no longer influenced by the meta-parameters. For example, if sufficient number of steps are used, the inner level will converge to a stable point regardless of initialization. This is related to the vanishing gradient problem and is explained in greater detail in the iMAML paper. This suggests that we either need early stopping (like in MAML) or some other explicit regularization (like in iMAML). How does your problem formulation consider this problem and how do you propose to use long inner loop training without any explicit or implicit regularization?\n\n(2) The paper writes $\\phi_i(\\theta) = \\hat{\\gamma}_i \\theta + (1-\\hat{\\gamma}_i) \\hat{\\phi}_i$ where $\\hat{\\phi}_i$ is a convex combinations of iterations in SGD for task $i$. Is $\\hat{\\phi}_i$ considered to be a function of $\\theta$, or do you pass stop-grad through the computation? This seems like a crucial detail that is missing, since if stop grad is not used, then one would still be differentiating through the optimization path. If stop grad is indeed used, why does the proposed algorithm solve the bi-level optimization?\n\n(3) \"The teacher’s role is similar to the skip connection in ResNet (He et al., 2016). They bridge two otherwise distant points such that gradients can effectively propagate between them.\" -- this seems like an unsubstantiated claim. In ResNets, gradients are passed through the skip connections. Do you also do this (see pt. 2 above)? Secondly,  ResNets perform skip connections from input to output, whereas GBML deals with optimization iterates. In my opinion, the correct analogy to ResNets would look something like Nesterov's momentum (where previous iterates are used to find the next iterate).\n\n(4) Lazy MAML requires hyper-parameter optimization over the different $\\alpha_i$ and $\\gamma$ for each **meta-training iteration**. This presumably would add significant computational overhead compared to MAML. This contradicts with the compute results in Appendix A.3. The cost of back-propagation through a graph is typically no more than 4-5 times as forward prop (regardless of the graph). If 4-5 hyperparameter choices are tried out, then the number of forward prop in case of lazy MAML would exceed this compute factor for backprop. Can the authors clarify as to why lazy MAML is more compute efficient? \n\n(5) Finally, I believe the authors may have a misunderstanding about MAML and related algorithms. Having a longer inner loop does not imply that higher order gradients are needed. As long as SGD is the inner loop optimizer, regardless of the size of the inner loop, only Hessian-vector products would be needed. See the iMAML paper for further discussion about these aspects. However, the memory footprint will increase linearly with the length of the inner loop. The authors of the submission point out the memory drawback correctly, but IMHO are a bit imprecise about the higher-order derivative claim. I recommend the authors to rephrase for correctness and better clarity.", "extra_scores": {}, "extra_text": {}}, {"review_id": "FG-L_hemg0l", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new method for gradient-based meta-learning that allows for multiple steps of inner-loop optimization while avoiding expensive backpropagation through these updates. The proposal is based on a teacher-student scheme, where the student does many steps of inner loop gradient descent, and the teacher does only a single update step where the single step is computed using the regions visited by the student. The authors propose variants of this setup for both Maml and Reptile. The meta-learning model is trained by back-propagating through the teacher's updates. Experiments are conducted on few-shot learning, long-tailed classification, and meta-attack benchmarks. \n\nPros\n* The paper considers a variety of different benchmarks (other than simply few-shot learning), including memorization in few-shot learning, long-tailed classification and meta-attack.\n\nCons\n* It is not straightforward to me what the benefit of the proposed method is compared to first-order maml (fomaml) or reptile. Both fomaml and reptile do not require expensive back-propogation through the inner-loop updates. It is mentioned in the paper that for their model, \"one can update the meta-model by backpropagating gradients through the teacher’s “leap”, not the student’s update chain (ignoring that the chain starts from the meta-model).\" This to me means that the second-order term is being ignored and so the update is indeed first-order.  Then, the only difference (as far as I can see), is that the proposed method uses the convex hull of the last b checkpoints and a linear combination that includes the initial parameters (which is a form of a lookahead optimizer) instead of simply the last checkpoint after k steps of gradient descent in the inner-loop. This could indeed offer some benefit but doesn't seem to offer the claimed benefit of \"enabling...exploring long horizons in the inner loop\" compared to fomaml or reptile as training with long inner loops seem similarly computationally expensive across all these first-order methods. If what I said above is not the case, I think the outer-gradient for the proposed method should be specifically stated and compared against the fomaml and second-order maml cases to highlight how this gradient is different and what benefit this offers (similar to the analysis that was done in the Reptile paper). In general, I believe a better argument needs to be made about the benefits of the proposed method compared to first-order methods such as fomaml and reptile, given that they all seem computationally similar in terms of cost.\n* My interpretation is that their \"lazy maml\" is essentially inserting a lookahead optimizer in the inner loop while having a first-order outer update; while \"lazy reptile\" is inserting a lookahead optimized in the inner loop and the outer-loop is also a lookahead optimizer by definition of reptile. However, in addition to the lookahead optimizer, they add calculating the convex hull of the last b checkpoints and hyperparmeters for this calculation and the lookahead optimizer are computed during meta-training by doing grid search on the perfomance on the corresponding validation set in each episode. It is unclear what the value of this hyperparameter search is and since it complicates the implementation of their method, it would be useful to have experiments to quantify the necessity of this hyperparameter search.\n* The experiments for TiredImageNet seem to only compare against second-order maml where the inner-loop consists of very few steps? I think a natural comparison is also fomaml where the number of inner-loop steps is the same as what was used for their model. Additionally, I believe results from other state-of-the-art models should also be shown to put the proposed model's results in context (a table is probably better for comparing across the metrics from different models). I had a similar observation about the few-shot memorization case - that fomaml with same number of inner-loop steps is a natural baseline. I would also be interested in knowing the metrics of iMAML for these benchmarks as it seems like a natural method for comparison.\n\nTo conclude, though some of the experimental results seem interesting, I believe the discussion of the method could be greatly improved as some of the claims seem a bit exaggerated or confusing. Additionally, I believe the setup of the experiments could be improved as stated above.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a variant of MAML or Reptile, where the meta-update along the long trajectory of the inner-loop optimization is bypassed to reduce the computational overhead appeared in MAML. The main idea is to use the look-ahed optimizer with careful tuning of relevant hyperparameters, which is done by a teacher-student scheme. Lazy MAML/Reptile are presented and experiments  demonstrated their validity. While the paper contains interesting ideas, most of reviewers have a few concerns which are not even resolved even after the author responses. First of all, ResNet analogy with respect to teacher update was claimed but it was never clearly shown in the paper. The method needs careful tuning of hyperparameters in the inner loop, but  the study about the computation requirements is not convincing yet.  Long inner loops are computationally feasible for both fomaml and reptile so its not clear in which way the proposed method is improving lengthy exploration in the inner loop other than the performance being better in the experiments. Improving the paper, taking these comments into account, will lead to a good work in near future. 
Decision: Reject