Meta-Learning with Warped Gradient Descent
Authors: Sebastian Flennerhag
Andrei A. Rusu
Razvan Pascanu
Francesco Visin
Hujun Yin
Raia Hadsell
Venue:  (2020)
Keywords: meta-learning
transfer learning

Abstract: Learning an efficient update rule from data that promotes rapid learning of new tasks from the same distribution remains an open problem in meta-learning. Typically, previous works have approached this issue either by attempting to train a neural network that directly produces updates or by attempting to learn better initialisations or scaling factors for a gradient-based update rule. Both of these approaches pose challenges. On one hand, directly producing an update forgoes a useful inductive bias and can easily lead to non-converging behaviour. On the other hand, approaches that try to control a gradient-based update rule typically resort to computing gradients through the learning process to obtain their meta-gradients, leading to methods that can not scale beyond few-shot task adaptation. In this work, we propose Warped Gradient Descent (WarpGrad), a method that intersects these approaches to mitigate their limitations. WarpGrad meta-learns an efficiently parameterised preconditioning matrix that facilitates gradient descent across the task distribution. Preconditioning arises by interleaving non-linear layers, referred to as warp-layers, between the layers of a task-learner. Warp-layers are meta-learned without backpropagating through the task training process in a manner similar to methods that learn to directly produce updates. WarpGrad is computationally efficient, easy to implement, and can scale to arbitrarily large meta-learning problems. We provide a geometrical interpretation of the approach and evaluate its effectiveness in a variety of settings, including few-shot, standard supervised, continual and reinforcement learning.

Published as a conference paper at ICLR 2020
WITH WARPED GRADIENT DESCENT
Sebastian Flennerhag," 1.2.3 Andrei A. RUsu, Razvan Pascanu, Francesco Visin, 3 Hujun Yin, Raia Hadsell
'The University of Manchester, 'The Alan Turing Institute, 'DeepMind (flennerhag] andrei rusu, razp, visin, raia @google. com jujunn www..yyieeannchstee.cc
ABSTRACT
Learning an efficient update rule from data that promotes rapid learning of new tasks from the same distribution remains an open problem in meta-learning. Typ- ically, previous works have approached this issue either by attempting train a neural network that directly produces updates or by attempting to learn better initialisations or scaling factors for a gradient- busdee update rule. Both of these approaches pose challenges. On one hand, directly producing an update forgoes a useful inductive bias and can easily lead non-converging behaviour. On the other hand, approaches that try to control a gradient-basee update rule typically resort to computing gradients through the learning process to obtain their meta- gradients, leading methods that can not scale beyond few-shot task adaptation. In this work, we propose Warped Gradient Descent (WarpGrad), method that intersects these approaches mitigate their limitations. WarpGrad meta-leams efficiently parameterised preconditioning matrix that facilitates gradient descent across the task distribution. Preconditioning arises by interleaving non-linear lay- ers, referred to as warp-layers, between the layers of a task-learner. Warp-layers are meta-learned without backpropagating through the task training process in a manner similar to methods that learn to directly produce updates. WarpGrad is computationally efficient, easy to implement, and can scale to arbitrarily large meta-learning problems. We provide a geometrical interpretation of the approach and evaluate its effectiveness in a variety of settings, including few-shot, standard supervised, continual and reinforcement learning.
INTRODUCTION
Learning (how) to learn implies inferring learning strategy from some set of past experiences via meta-learner that task-learner can leverage when learning a new task. One approach to directly parameterise update rule via the memory of recurrent neural network (Andrychowicz al., 2016; Ravi & Larochelle, 2016; Li & Malik, 2016; Chen et al., 2017). Such memory-base methods can, in principle, represent any learning rule by virtue of being universal function approximators (Cybenko, 1989; Hornik, 1991; Schäfer & Zimmermann, 2007). They can also scale to long learning processes by using truncated backpropagation through time, but they lack an inductive bias as what constitutes reasonable learning rule. This renders them hard to train and brittle to generalisation as their parameter updates have no guarantees of convergence.
An alternative family of approaches defines gradient- based update rule and meta-learns shared initialisation that facilitates task adaptation across a distribution of tasks (Finn al., 2017: Nichol et al., 2018: 8; Flennerhag et al., 2019). Such methods are imbued with a strong inductive bias gradient descent-but restrict knowledge transfer to the initialisation. Recent work has shown that it is beneficial to more directly control gradient descent by meta-learning parameterised matrix (Li et al., 2017; Lee & Choi, 2018; Park & Oliva, 2019) that preconditions gradients during task training, similarly to second-order and Natural Gradient Descent methods (Nocedal & Wright, 2006; Amari & Nagaoka, 2007). To meta- meta-learn preconditioning. these methods backpropagate through the gradient descent process, limiting them to few-shot learning.
approximation of

---

Published as a conference paper at ICLR 2020
Task Adaptation
Meta- Leta-Learning
f(x)
f(w) C(0; の)
((2)
(2(2)
12
(2)
((())
P(2) Vicas
min [T)]
PVL
(1)
(1)
Dw(1)
V L
(1))
(01)
p(l L
Task-learners
Shared Warp
Figure 1: Schematics of WarpGrad. WarpGrad preconditioning is embedded in task-learners f by in- terleaving warp-layers (w(1), (2) between each task-learner's layers (h(1), A2))). WarpGrad achieve preconditioning by modulating layer activations in the forward pass and gradients in the backward pass by backpropagating through warp-layers (Dw), which implicitly preconditions gradients by some matrix (P). Warp-parameter (Ø) are meta-learnee over the joint search space induced by task adaptation (Eg (o)) to form geometry that facilitates task learning.
In this paper, we propose novel framework called Warped Gradient Descent (WarpGrad)' that relies on the inductive bias of muddient-based meta-learnen by defining an update rule that preconditions gradients, but that is meta-learned using insights from memory- -bsee methods. In particular, we leverage that gradient preconditioning is defined point-wise in parameter space and can be seen as recurrent operator order 1. We use this insight to define trajectory agnostic meta-objective over a joint parameter search space where knowledge transfer is encoded in gradient preconditioning.
To achieve scalable and flexible form preconditioning, we take inspiration from works that embed preconditioning in task-learners (Desjardins et al., 2015; Lee & Choi, 2018), but we relax the assumption that task-learners are feed-forward and replace their linear projection with a generic neural network w, referred to warp layer. By introducing non non-linearity, preconditioning is rendered data data-dependent. This allows WarpGrad to model preconditioning beyond the block -iigonal structure of prior works and enables
meta-learn over arbitrary adaptation processes.
We empirically validate WarpGrad and show it surpasses baseline gradient- based meta-learners standard few-shot learning tasks (miniimageNNt,, tieredImageNet; Vinyals et al., 2016; Ravi & Larochelle, 2016; Ren et al., 2018), while scaling beyond few-shot learning to standard supervised settings the "multi"-shor Omniglot benchmark (Flennerhag al., 2019) and multi-shot version of tieredImageNet. We further find that WarpGrad outperforms competing methods in reinforce- ment learning (RL) setting where previous gradient-basee meta-learne fail (maze navigation with recurrent neural networks (Miconi et al., 2019)) and can be used to meta-learn optimiser that prevents catastrophic forgetting in a continual learning setting.
WARPED GRADIENT DESCENT
2.1 GRADIENT-BASER BASED LEATA-LEARNING
WarpGrad belongs to the family of optimisation-based meta-learners that parameterise an update rule 0 + (0; E) with some meta-parameters &. Specifically, gradient-based meta-learners define an update rule by relying on the gradient descent, (B: = 0 a L(Ø) for some objective L and learning rate a. A task is defined by a training set D and a test set Dis.i which defines learning objectives CD= (0) = E( (( (z, 0), y)] over the task-learner for some loss l. MAML (Finn
Open-source implementation available https: pectps:/ggitie http//ggtttuuuutt/// hatpo//cccc///eeeee/a///aaaaa
2

---

Published as a conference paper at ICLR 2020
V
Figure 2: Gradient-based meta-learning. Colours denote different tasks (r), dashed lines denote backpropagation through the adaptation process, and solid black lines denote optimiser parameter (Ø) gradients w.r.t. one step of task parameter (0) adaptation. Left: A meta-learned initialisation compresses trajectory information into a single initial point (fo). Middle: MAML-based optimisers interact with adaptation trajectories every step and backpropagate through each interaction. Right: WarpGrad is trajectory agnostic. Task adaptation defines an empirical distribution P(T, 0) over which WarpGrad learns geometry for adaptation by optimising for steepest descent directions.
et al., 2017) meta-learns a shared initialisation &g by backpropagating through steps of gradient descent across given task distribution p(T),
K-1 UD (6%:
MAMLL (5) =
Σ LD: Ao T-UP(T)
(1)
Subsequent works on gradient-based meta-learning primarily differ in the parameterisation of U. Meta-SGD (MSGD; Li & Malik, 2016) learns vector of learning rates, whereas (MC; Park Oliva, 2019) defines block-diagonal preconditioning matrix B, and T-Nets (Lee & Choi, 2018) embed block -ickonda preconditioning in feed-forward learners via linear projections,
U(0g; 00) = 02 a [((k)
MAML
(2)
U(0;; Φ) 0₂ a V
MSGD
(3)
U(0; S. Φ) = 82 aBir; @) C(Ax)
MC
(4)
(0,;00.00 O2 av L(Ox;
T-Nets.
(5)
These methods optimise for meta-parameters ę (0o, Φ} by backpropagating through the gradient descent process (Eq. 1). This trajectory dependence limits them few-shot learning as they become (I) computationally expensive, (2) susceptible to exploding/ vanddiin/ gradients. and (3) susceptible credit assignment problem (Wu al., 2018; Antoniou al., 2019; Liu al., 2019).
Our goal is to develop meta-learner that overcomes all three limitations. To do so, we depart from the paradigm of backpropagating to the initialisation and exploit the fact that learning to precondition gradients can be seen as a Markov Process of order that depends on the state but not the trajectory (Li et al., 2017). To develop this notion, we first establish a general-purpose form of preconditioning (Section 2.2). Based on this, we obtain a canonical meta-objective from a geometrical point of view (Section 2.3), from which we derive enjectory-agnossee meta-objective (Section 2.4).
2.2 DENERAL-PPRPOSE PRECONDITIONING
A preconditioned gradient descent rule, U(O: o) = - v L(0), defines geometry via P. To disentangle the expressive capacity of this geometry from the expressive capacity of the task- learner f. we take inspiration from T-Nets that embed linear projections T in feed-forward layers, oT' 66). This in itself is not sufficient to achieve disentanglement since the parameterisation o' is directly linked that of W, but can be achieved under non-linear preconditioning.
To this end, we relax the assumption that the task-learner is feed-forward and consider an arbitrary neural network, f = (5)) (!), We insert warp-layers that are universal function approximators parameterised by neural networks into the task-learner without restricting their form or how they
3

---

Published as a conference paper at ICLR 2020
Pr
Pr
<LC
W
W
G-'V
Figure 3: Left: synthetic experiment illustrating how WarpGrad warps gradients (see Appendix for full details). Each task p(f) defines distinct loss surface (W, bottom row). Gradient descent (black) on these surfaces struggles to find a minimum. WarpGrad meta-learns a warp w to produce better update directions (magenta; Section 2.4). In doing so, WarpGrad learns meta-geo reta-geometty P where standard gradient descent is well behaved (top row). Right: gradient descent in P is equivalent first-order Riemannian descent W under meta-learned Riemann metric (Section 2.3).
interact with f. the simplest case, we interleave warp-layers between layers of the task-learner to obtain f= (T.) (L.) o... o (1) h()) but other forms of interaction can be beneficial (see Ap- pendix A for practical guidelines). Backpropagation automatically induces gradient preconditioning. as in T-Nets, our case via the Jacobians of warp-layers:
(-(++) II
Oc 0000
Dowilh-i) Dabh-l-)) Dyw(l) Dah(
E Ve
where D, and Dg denote the Jacobian with respect to input and parameters, respectively. In the special case where f is feed-forward and each w a linear projection, we obtain an instance of WarpGrad that is akin to T-Nets since preconditioning is given by D2w = T. Conversely, by making warp-layers non-linear, model preconditioning beyond the block dlack-diagonal structure imposed by prior works. Further, this enables form of kask-oonditionig by making Jacobians of warp-layers data dependent. As we have made no assumptions on the form of the task-learner or warp-layers, WarpGrad methods can act on any neural network through any form of warping, including recurrence. We show that increasing the capacity of the meta-learner by defining warp-layers as Residual Networks (He et al., 2017) improves performance on classification tasks (Section 4.1). We also introduce recurrent warp-layers for agents in gradient-based meta meta-learner that is the first, to the best of our knowledge, to outperform memory- based meta-learners on maze navigation task that requires memory (Section 4.3).
induce interdependence between warp-layers, allowing WarpGrad to
Warp-layers imbue WarpGrad with three powerful properties. First, due to preconditioned gradients, WarpGrad inherits gradient descent properties, importantly guarantees of convergence. Second, warp-layers form distributed representation of preconditioning that disentangles the expressiveness of the geometry it encodes from the expressive capacity of the task-learner. Third, warp-layers are meta-learned across tasks and trajectories and can therefore capture properties of the task-distribution beyond local information. Figure illustrates these properties in a synthetic scenario, where we construct family of tasks f 2 R (see Appendix for details) and meta-learn across the task distribution. WarpGrad learns to produce warped loss surfaces (illustrated on two tasks 7 and r') that are smoother and more well-behaved than their respective native loss-surfaces.
2.3 THE GEOMETRY OF WARPED GRADIENT DESCENT
the preconditioning matrix P is invertible, defines a valid Riemann metric (Amari, 1998) and therefore enjoys similar convergence guarantees gradient descent. Thus, if warp-layers represent
4

---

Published as a conference paper at ICLR 2020
valid (meta-learned) Riemann metric, WarpGrad is well-behaved. For T-Nets, it is sufficient to require to be full rank, since T explicitly defines P as a block- diagonl matrix with block entries TT". In contrast, non-linearity in warp-layers precludes such an explicit identification.
Instead, we must consider the geometry that warp-layers represent. For this, we need a metric tensor, G, which is positive-definite, smoothly varying matrix that measures curvature
manifold W. The metric tensor defines the steepest direction of descent by - -G- (Lee, 2003), hence our goal establish that warp-layers approximate some G-1, Let L represent the effect of warp-layers by reparameterisation h h (z: (10; o) ( (z; ); d) Vz, i that maps from space P onto the manifold W with 7 = S2(0; o). We induce a metric G on W by push-forward (Figure 2):
A = V (Co.)) (0: = [D,2((0))] [ B o2)((00)) AB (
P -sspace
(7)
W W-space,
(8)
where G- [D [D," Provided Q is not degenerate (G is non-singular), G-1 is positive- definite, hence a valid Riemann metric. While this is the metric induced on W by warp-layers, it not the metric used to precondition gradients since
take gradient steps in P which introduces term (Figure 2). We can bound the error by first-order Taylor series expansion to establish first-order equivalence between the WarpGrad update in P (Eq. and the ideal update in W (Eq. 8),
( a20) L(> -a))))))))))- O(a²).
Consequently, gradient descent under warp-layers (in P-space) is first-order equivalent to warping the native loss surface under a metric to facilitate task adaptation. Warp parameters ó control the geometry induced by warping, and therefore what task-learners converge to. By meta-learning á can accumulate information that conducive to task adaptation but that may not be available during that process. This suggests that an ideal geometry (in W-space) should yield preconditioning that points in the direction of steepest descent, accounting for global information across tasks,
min Eccypp(er)) (1 -a(
(10)
In contrast to MAML-based approaches (Eq. 1), this objective avoids backpropagation through learning processes. Instead, defines task learning abstractly by introducing joint distribution over objectives and parameterisations, opening up for general-purposs meta-learning at scale.
2.4 META-LEARNING WARP PARAMETERS
The canonical objective in Eq. 10 describes a meta-objective for learning geometry on first principles that we can render into trajectory-agnosie update rule for warp-layers. To do so, we define task 7 (h' fi F(ssk) by task-learner î that is embedded with shared WarpGrad optimiser, a meta-training objective L and a task adaptation objective L We use Liss to adapt task parameters / and Cineta to adapt warp parameters ø. Note that we allow meta and task objectives to differ in arbitrary ways, but both are expectations over some data, as above. In the simplest case, they differ in terms of validation versus training data, but they may differ in terms of learning paradigm as well, as we demonstrate in continual learning experiment (Section 4.3).
To obtain our meje-objective, we recast the canonical objective (Eq. 10) in terms of using first-order equivalence of gradient steps (Eq. 9). Next, we factorise p(T, 0) into p(0 T)p(T). Since P(T) is given, remains to consider a sampling strategy for p(0 T). For meta-learning of warp-layers, we assume this distribution is given. We later show how to incorporate meta-learning of prior p(0o 7). While any sampling strategy valid, in this paper we exploit that task learning under stochastic gradient descent can be seen as sampling from an empirical prior p(0 7) (Grant et al., 2018); in particular, each iterate 0%. can be seen as a sample from p(0T (T-1 Thus, K K-steps of gradient descent forms Monte-Carke chain 07 ......, 0% and sampling such chains define an empirical distribution p(0 T) around some prior p(6n r), which we will discuss in Section 2.5. The joint distribution p(T, 0) defines joint search space across tasks. Meta-learning therefore learns
5

---

Published as a conference paper at ICLR 2020
geometry this space with the steepest expected direction of descent. This direction is however not with respect to the objective that produced the gradient, Lissk but with respect to metaa
Finet (0"
I(Ø)= E
Lisk (6'; :0):0)
(11)
Tmp(T) (>m(0(]=)
Decoupling the task gradient operator V V from the geometry learned by Siess lets us infuse global knowledge in warp-layers, promising avenue for future research (Metz al., 2019; Mendonca et al., 2019). For example, in Section 4.3, we meta-learn an update-rule that mitigates catastrophic forgetting by defining Line over current and previous tasks. In contrast to other gradient-based meta-lcarners, the WarpGrad meta-objective an expectation over gradient update steps sampled from the search space induced by task adaptation (for example, K steps of stochastic gradient descent; Figure 2). It is therefore trajectory agnostic and hence compatible with arbitrary task learning processes. Because the meta-gradient is independent of the number of task gradient steps, it avoids vanishing/exploding gradients and the credit assignment problem by design. It does rely on second-order gradients, requirement we can relax by detaching task parameter gradients (V (Ta))) in Eq. 11,
1())= = Σ
mett
sg
a LTssk (0"; :0)
(12)
Tmp(T]) (p((TT
where is the stop-gradient operator. In contrast to the first-order approximation of MAML (Finn et al., 2017), which ignores the entire trajectory except for the final gradient, this approximation retains all gradient terms and only discards local second-order effeects, which are typically dominated by first-order effect in long parameter trajectories (Flennerhag et al., 2019). Empirically, we find that our approximation only incurs minor loss of performance in ablation study Omniglot (Appendix F). Interestingly, this approximation is a form of multitask learning with respect to á (Li & Hoiem, 2016; Bilen & Vedaldi, 2017; Rebuffi al., 2017) that marginalises over task parameters 0".
Algorithm WarpGrad: online meta-training
Algorithm 2 WarpGrad: offline meta-training
Require: p(T): distribution over tasks
Require: P(T): distribution over tasks
Require: 8, A: hyper-parameters
Require: a, 13 X, 7/: hyper-parameter
1: initialise Φ and p(O. r)
1: initialise Φ. p(0 T)
2: while not done do
while not done do
3: sample mini-batch of tasks T from p(T)
3: initialise buffer B
4: 96. 90, f-
sample mini-batch of tasks T from p(T)
5: for all r E T do
5: for all
do
6: 8 ~ 10 r)
6: 0% ~ p(0g T)
7:
for all k in
K-- do
B[ = [0]]
8:
0 + 0f
a Lrssk (0) Φ)
8:
for all 1: in O.
KG- 1 do
9:
Js + 9s + L(o; Of
07 0- t of V (0%: d)
10:
90, 90, + V CC(u: 0
10:
B B|rlappendd 2T+))
11:
end for end for
11: end for
12:
end for
13:
④ + ò - 3ge
13:
i. 90, +0
- 0o X,500
14:
for all (T, E B do
15:
end while
15:
96 + + V L( 0%)
16:
90. <- 90g + VC 0%.
17:
iti+1
18:
if = 1 then
19:
Φ 6 3gg
20:
6n - 8 - X390.
21:
i, 1. 90. 0
22:
end if
23:
end for
24:
end while

---

Published as a conference paper at ICLR 2020
2.5 INTEGRATION WITH LEARNED INITIALISATIONS
WarpGrad is a method for learning warp layer parameters & over joint search space defined by plT, 0). Because WarpGrad takes this distribution as given, we can integrate WarpGrad with methods that define or learn some form of "prior" p(0p 7) over 0% For instance, (a) Multi-task solution: in online learning, we can alternate between updating a multi-task solution and tuning warp parameters. We use this approach in our Reinforcement Learning experiment (Section 4.3): (b) Meta-leamedd point-estimate: when task adaptation oceurs in batch mode, we can meta-learn a shared initialisation to- Our few-shot and supervised learning experiments take this approach (Section 4.1); (c) Meta- learned prior: WarpGrad can be combined with Bayesian methods that define a full prior (Rusu et al., 2019; Oreshkin et al., 2018; Lacoste et al., 2018; Kim et al., 2018). We incorporate such methods by some objective C' (potentially vacuous) over 9g that we optimise jointly with WarpGrad,
00, = (0) + +XC((0))
(13)
where L can be substituted for by L and À E [0, 00) is a hyper-parametee We train the Warp- Grad optimiser via stochastic gradient descent and solve Eq. 13 by alternating between sampling task parameters from p(T, 8) given the current parameter values for á and taking meta-gradien steps over these samples to update Φ. As such, our method can also be seen as generalised form of gradient descent in the form of Mirror Descent with a meta-learned dual space (Desjardins et al., 2015; Beck & Teboulle, 2003). The details of the sampling procedure may vary depending on the specifics of the tasks (static, sequential), the design of the task (feed--orward, recurrent). and the learning objective (supervised, self-supervised. reinforcement learning). In Algorithm I we illustrate a simple online algorithm with constant memory and linear complexity in K, assuming the same holds for C. drawback of this approach is that it is relatively data inefficient; in Appendix B we detail a more complex offline training algorithm that stores task parameters in a replay buffer for mini-batched training of Φ. The gains of the offline variant can be dramatic: in our Omniglot experiment (Section 4.1), offline meta- meta-traini training allows us to update warp parameters 2000 times with each meta- batch, improving final test accuracy from 76.3% to 84.3% (Appendix F).
3 RELATED WORK
Learning to learn, or meta-learning, has previously been explored in variety of settings. Early work focused on evolutionary approaches (Schmidhuber, 1987; Bengio et al., 1991; Thrun & Pratt, 1998). Hochreiter et al. (2001) introduced gradient descent methods to meta-learning, specifically for recurrent meta-learning algorithms, extended to RL by Wang et al. (2016) and Duan et al. (2016). A similar approach was taken by Andrychowicz et al. (2016) and Ravi & Larochelle (2016) to meta-learn parameterised update rule in the form of Recurrent Neural Network (RNN).
A related idea is to separate parameters into "slow" and "fast" weights, where the former captures meta-information and the latter encapsulates rapid adaptation (Hinton & Plaut, 1987; Schmidhuber, 1992; Ba et al., 2016). This can be implemented by embedding a neural network that dynamically adapts the parameters of main architecture (Ha et al., 2016). WarpGrrad can be seen learning slow warp-paramen parametersss that precondition adaptation of fast weights. Recent meta-learning focuses almost exclusively on few-shot learning, where tasks are characterised by severe data scarcity. In this setting, tasks must be sufficiently similar that new task can be learned from a single or handful of examples (Lake et al., 2015; Vinyals al., 2016; Snell et al., 2017: Ren et al., 2018).
Several meta-learners have been proposed that directly predict the parameters of the task- learner (Bertinetto et al.. 2016; Munkhdalai et al., 2018: Gidaris & Komodakis, 2018; Qiao et al., 2018). To scale, such methods typically pretrain a feature extractor and predict a small subset of the parameters. Closely related to our work are gradient-basee few-shot learning methods that extend MAML by sharing some subset of parameters between task-learners that is fixed during task training but meta-learner across tasks, which may reduce overfitting (Mishra et al., 2018; Lee & Choi, 2018; Munkhdalai et al., 2018) or induce more robust convergence (Zintgraf et al., 2019). It can also be used to model latent variables for concept or task inference, which implicitly induce gradient modulation (Zhou et al., 2018; Oreshkin et al., 2018; Rusu et al., 2019; Lee et al., 2019). Our work is also related to gradient- based meta-lcarning of a shared initialisation that scales beyond few-shot learning (Nichol et al., 2018; Flennerhag et al., 2019).
7

---

Published as a conference paper at ICLR 2020
Meta-learned preconditioning is closely related to parallel work on second-order optimisation methods for high dimensional non-couvex loss surfaces (Nocedal & Wright, 2006; Saxe et al., 2013; Kingma & Ba, 2015; Arora et al., 2018). In this setting, second-order optimisers typically struggle to improve upon first-order baselines (Sutskever et al., 2013). As second-order curvature is typically intractable compute, such methods resort low-rank approximations (Nocedal & Wright, 2006; Martens, 2010; Martens & Grosse, 2015) and suffer from instability (Byrd et al., 2016). In particular, Natural Gradient Descent (Amari, 1998) is method that uses the Fisher Information Matrix as curvature metric (Amari & Nagaoka, 2007). Several proposed methods for amortising the cost of estimating the metric (Pascanu Bengio, 2014; Martens & Grosse, 2015; Desjardins et al., 2015). As noted by Desjardins et al. (2015), expressing preconditioning through interleaved projections can be seen as a form of Mirror Descent (Beck& Teboulle, 2003). WarpGrad offers a new perspective on gradient preconditioning by introducing generic form of model-embedded preconditioning that exploits global information beyond the task at hand.
4 EXPERIMENTS
We evaluate WarpGrad in a set of exper- iments designed to answer three ques- tions: (1) do WarpGrad methods retain the inductive bias of MAML-based few- shot learners? (2) Can WarpGrad meth- ods scale to problems beyond the reach of such methods? (3) Can WarpGrad generalise to complex meta-learning problems?
Table 1: Mcan test accuracy after task adaptation on held out evaluation tasks. -Mulei-headed. !No meta-nraining; see Appendix E and Appendix H.
minilmageNet
5-way 1-shot
5-way 5-shot
Reptile
50.0 + 0.3
66.0 + 0.6
Meta-SGD
50.5 = 1.9
64.0 = 0.9
(M)T-Net
51.7 = 1.8
CAVIA (512)
51.8 + 0.7
65.9 + 0.6
MAML
48.7 = 1.8
63.2 + 0.9
4.1 FEW-SHOT LEARNING
Warp-MAML
52.3 = 0.8
68.4 0.6
For few-shot learning, test whether WarpGraa retains the inductive bias of bradient-based meta-learners while avoiding backpropagation through the gradient descent process. To isolate the effect of the WarpGrad objective, we use linear warp-layers that we train using online meta-training (Algo-
tieredImageNet
5-way 1-shot
5-way 5-shot
MAML Warp-MAML
51.7 + 1.8 57.2 = 0.9
70.3 + 1.8 74.1 0.7
fieredImageNet 10-way 640-shot
Omniglot 20-way 100-shot
rithm 1) to make WarpGrad as close to T-Nets as possible. For fair comparison, we meta-learn the initial- isation using MAML (Warp-MAML) with [6p, Φ) = T(Ø) + XCMAML (0o). We evaluate the importance of meta- learning the initialisation in Appendix G and find that WarpGrad achieves similar performance under random task param- eter initialisation.
SGD
58.1 + 1.5
51.0
KFAC!
56.0
Finetuning
76.4 + 2.2
Reptile
76.52 +
70.8 1.9
Leap Warp-Leap
73.9 - 2.2
75.5 2.6
80.4 = 1.6
83.6 + 1.9
All task-learners use a convolutional architecture that stacks 4 blocks made of a 3 x 3 convolution, max-pooling batch-norm, and ReLU activation. We define Warp-MAMLL by inserting warp-layers in the form of 3 x 3 convolutions after each block in the baseline task-learner. All baselines are tuned with identical and independent hyper-parameter searches (including filter sizes full experimental settings in Appendix H), and we report best results from our experiments or the literature. Warp- MAML outperforms all baselines (Table 1), improving 1- and 5-shot accuracy by 3.6 and 5.5 percentage points minilmageNet (Vinyals al., 2016; Ravi & Larochelle, 2016) and by 5.2 and 3.8 percentage points on tieredImageNet (Ren et al., 2018), 18), which indicates that WarpGrad retains the inductive bias of MAML- meta-learners.
8

---

Published as a conference paper at ICLR 2020
Warpel.cp
Leap
Reptile
SGDE
KPACI
Warp INN
Bebb INN
Bebh RNN
RNN
175
00.8
Test aceuracy held
125
Rowwar
100
10.6
50
25
0.4
5
15
20
25
500000
80000 100000
Number of tasks in meta-training set
Number Epteades
Figure 4: Left: Omniglot test accuracies on held-out tasks after meta-traint meta-training on varying number of tasks. Shading represents standard deviation across 10 independent runs. We compare Warp- Leap. Leap. and Reptile. multi-headed finetuning, as well as SGD and KFAC which used random initialisation but with 4x larger batch size and 10x larger learning rate. Right: On a RL maze navigation task, mean cumulative return is shown. Shading represents inter-quartile ranges across 10 independent runs. Simple modulation and Fretroactive modulation are used (Miconi al., 2019).
4.2 MULTI-SHOT LEARNING
Next, we evaluate whether WarpGrad can scale beyond few-shot adaptation on similar supervised problems. We propose new protocol for tieredImageNet that increases the number of adaptation steps to 640 and use 6 convolutional blocks in task lea--leaners,, which are otherwise defined as above. Since MAML-based approaches cannot backpropagate through 640 adaptation steps for models of this size, we evaluate WarpGrad against two gradient- based meta-leamers that meta-learn an initialisation without such backpropagation, Reptile (Nichol al., 2018) and Leap (Flennerhag et al., 2019), and we define a Warp-Leap meta-learner by J(O.. Φ) = L(e) + AC (0o). Leap is an attractive complement as minimises the expected gradient descent trajectory length across tasks. Under WarpGrad, this becomes joint search for geometry in which task adaptation defines geodesics (shortest paths, see Appendix C for details). While Reptile outperforms Leap by 2.6 percentage points on this benchmark, Warp-Leap surpasses both, with margin of 3.88 to Reptile (Table 1).
We further evaluate Warp-Leap on the multi shot Omniglot (Lake et al., 2011) protocol proposed by Flennerhag et al. (2019), where each of the 50 alphabets is 20-way classification task. Task adaptation involves 100 gradient steps on random samples that are preprocessed by random affine transformations. We report results for Warp-Leap under offline meta-training (Algorithm 2), which updates warp parameters 2000 times per meta step (see Appendix E for experimental details). Warp- Leap enjoys similar performance on this task well, improving over Leap and Reptile by 8.1 and 12.8 points respectively (Table 1). We also perform an extensive ablation study varying the number of tasks in the meta-training set. Except for the case of single task, Warp-Leap substantially outperforms all baselines (Figure 4), achieving a higher rate of convergence and reducing the final test error from ~30% -15% Non-linear warps, which go beyond block-diagonal preconditioning, reach ~11 111 % test error (refer to Appendix and Table 2 for the full results). Finally, we find that WarpGrad methods behave distinctly different from Natural Gradient Descent methods in an ablation study (Appendix G). It reduces final test error from -42% to -19%, controlling for initialisation, while its preconditioning matrices differ from what the literature suggests (Desjardins et al., 2015).
4.3 COMPLEX META-LEARNING. REINFORCEMENT AND CONTINUAL LEARNING
To illustrate how WarpGrad may be used both with recurrent neural networks and in meta-reinforcement learning, we evaluate it in a maze navigation task proposed by Miconi (2018). The environment is fixed maze and task defined by randomly choosing a goal location. The agent's objective is to find the location as many times as possible, being teleported to random location each time finds it. We use advantage actor-critic with a basic recurrent neural HyperNetwork (Ha
(c.1) Reinforcement Learning
network (Wang et al., 2016) as the lask-learner, and we design Warp-RNN et al., 2016) that uses an LSTM that is fixed during task training. This LSTM modulates the weights of the task-learning RNN (defined in Appendix I), which in turn is trained mini-batches of 30 episodes for 200 000 steps. We accumulate the gradient of fixed warp-parament continually (Algorithm 3, Appendix B) at each task parameter update. Warp parameters updated every 30th

---

Published as a conference paper at ICLR 2020
Figure 5: Continual learning experiment. Average log-loss over 100 randomly sampled tasks, each comprised of 5 sub-tasks. Left: learned sequentially as seen during meta- meta-trainie training, Right: learned in random order [sub-task 3, 4, 2, 0].
step on task parameters (we control for meta-LSTM, capacity in Appendix I). We compare against Learning Reinforcement Learn (Wang et al., 2016) and Hebbian meta-learning (Miconi al., 2018; 2019); see Appendix for details. Notably, linear warps (T-Nets) do worse than the baseline RNN on this task while the Warp-RNN converges to mean cumulative reward -160 in 60 000 episodes, compared to baselines that reach at most mean cumulative reward of - -125 after 100 000 episodes (Figure 4), reaching -150 after 200 000 episodes (I).
(c.2) Continual Learning We test if WarpGrad can prevent catastrophic forgetting (French, 1999) in continual learning scenario. To this end, design continual learning version of the sine regression meta-learning experiment Finn et al. (2017) by splitting the input interval [-5, 5] C R into 5 consecutive sub-tasks (an alternative protocol was recently proposed independently by Javed & White, 2019). Each sub-task is a regression problem with the target being a mixture of two random sine waves. We train 4-layer feed-forward task- with interleaved warp-layers incrementally on one sub-task at a time (see Appendix for details). To isolate the behaviour of WarpGrad parameters, we use a fixed random initialisation for each task sequence. Warp parameters are meta-learned to prevent catastrophic forgetting by defining fi to be the average task loss over current and previous sub-tasks, for each sub-task in a task sequence. This forces warp-paran warp-parameters to disentangle the adaptation process of current and previous sub-tasks. We train each sub-task for 20 steps, for total of 100 task adaptation steps. We evaluate WarpGrad on 100 random tasks and find that it learns new sub-tasks well, with mean losses performance immediately deteriorates to -10 but is stable for the remainder training (Figure 5). Our results indicate that WarpGrad can be effective mechanism against catastrophic forgetting. promising avenue for further research. For detailed results, see Appendix J.
order of magnitude 10-3 When switching sub-task,
CONCLUSION
We propose WarpGrad, a novel meta-learner that combines the expressive capacity and flexibility of memory-based meta-learners with the inductive bias of gradient- meta-learners. WarpGrad meta-learns to precondition gradients during task adaptation without backpropagating through the adaptation process and we find empirically that retains the inductive bias of MAML-based few-shot learners while being able to scale to complex problems and architectures. Further, by expressing preconditioning through warp-layers that are universal function approximators, WarpGrad can express geometries beyond the block- diigonaa structure of prior works.
WarpGrad provides principled framework for general general-purpose purpose meta-learning that integrates learning paradigms, such as continual learning, an exciting avenue for future research. We introduce novel means for preconditioning. for instance with residual and recurrent warp-layers. Understanding how WarpGrad manifolds relate second-order optimisation methods will further understanding of gradient- based meta-learning and aid us in designing warp-layers with stronger inductive bias.
In their current form, WarpGradshare some of the limitations of many popular meta-learning ap- proaches. While WarpGrad avoids backpropagating through the task training process, as in Warp-Leap, the WarpGrad objective samples from parameter trajectories and has therefore linear computational complexity in the number of adaptation steps, currently an unresolved limitation of gradient- based meta-learning. Algorithm 2 hints at exciting possibilitties for overcoming this limitation.
10

---

Published as a conference paper at ICLR 2020
ACKNOWLEDGEMENTS
The authors would like to thank Guillaume Desjardins for helpful discussions on an early draft as well as anonymous reviewers for their comments. SF gratefully acknowledges support from North West Doctoral Training Centre under ESRC grant ES/J500094/1 and by The Alan Turing Institute under EPSRC grant EP/N510129/1.
REFERENCES
Amari, Shun-Ichi. Natural gradient works efficiently in learning. Neural computation, 10(2):511276. 1998.
Amari, Shun-ichi and Nagaoka, Hiroshi. Methods of information geometry, volume 191. American Mathematical Society, 2007.
Andrychowicz, Marcin, Denil, Misha, Gomez, Sergio, Hoffman, Matthew W, Pfau, David, Schaul, Tom, Shillingford, Brendan, and De Freitas, Nando. Learning to learn by gradient descent by gradient descent. In Advances in Neural Information Processing Systems, 2016.
Antoniou, Antreas, Edwards, Harrison, and Storkey, Amos J. How to train your MAML. In International Conference Learning Representations, 2019.
Arora, Sanjeev, Cohen, Nadav, and Hazan, Elad. On the optimization of deep networks: Implicit acceleration by overparameterizaaion. International Conference on Machine Learning. 2018.
Ba, Jimmy, Hinton, Geoffrey E, Mnih, Volodymyr. Leibo, Joel Z, and Ionescu, Catalin. Using fast weights to attend to the recent past. In Advances in Neural Information Processing Systems, 2016.
Beck, Amir and Teboulle, Mare. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters, 31:167-175, 2003.
Bengio, Yoshua, Bengio, Samy, and Cloutier, Jocelyn. Learning a synaptic learning rule. Université de Montréal, Département d' d'informatique et de recherche opérationnelle, 1991.
Bertinetto, Luca, Henriques, João F, Valmadre, Jack. Torr, Philip, and Vedaldi, Andrea. Learning feed-forward one-shot learners. In Advances in Neural Information Processing Systems, 2016.
Bilen, Hakan and Vedaldi, Andrea. Universal representations: The missing link between faces, text, planktons, and cat breeds. arXiv preprint arXiv: urXiv:1701.07277: 2017.
Byrd, R., Hansen, S., Nocedal, J., and Singer, Y. A stochastic quasi-newton method for large-scale optimization. SIAM Journal on Optimization, 26/2):1008-1031, 2016.
Chen, Yutian, Hoffman, Matthew W, Colmenarejo, Sergio Gómez, Denil, Misha, Lillicrap. Timothy P, Botvinick, Matt, and de Freitas, Nando. Learning to learn without gradient descent by gradient descent. In International Conference Machine Learning, 2017.
Cybenko, George. Approximation by superpositions of a sigmoidal function. MCSS, 1989.
Deng, Jia, Dong, Wei, Socher, Richard, Li, Li-Jia, Li, Kai, and Fei-Fei, Li. Imagenet: A large- scale hierarchical image database. In International Conference on Computer Vision and Pattern Recognition, 2009.
Desjardins, Guillaume, Simonyan, Karen, Pascanu, Razvan, and kavukcuoglu, koray. Natural neural networks. In Advances in Neural Information Processing Systems, 2015.
Duan, Yan, Schulman, John, Chen, Xi, Bartlett, Peter L., Sutskever, Ilya, and Abbeel, Pieter. RI$22$: Fast reinforcement learning via slow reinforcement learning. arXiv preprini arXiv: 1611,02779, 2016.
Finn, Chelsea, Abbeel, Pieter, and Levine, Sergey. Model-Agnostic Meta-Learning for Fast Adapta- tion of Deep Networks. In International Conference on Machine Learning, 2017.
Flennerhag, Sebastian, Yin, Hujun, Keane, John, and Elliot, Mark. Breaking the activation function bottleneck through adaptive parameterization. In Advances in Neural Information Processing Systems, 2018.
11

---

Published as a conference paper at ICLR 2020
Flennerhag, Sebastian, Moreno, Pablo G., Lawrence, Neil D., and Damianou, Andreas. Transferring knowledge across learning processes. In International Conference on Learning Representations. 2019.
French, Robert M. Catastrophic forgetting in connectionist networks, Trends in cognitive sciences, (4):128-135, 1999.
Gidaris, Spyros and Komodakis, Nikos. Dynamic few-shot visual learning without forgetting. In Internationa Conference on Computer Vision and Pattern Recognition, 2018.
Grant, Erin, Finn, Chelsea, Levine, Sergey, Darrell, Trevor, and Griffiths, Thomas L. Recasting gradient-based meta-lcarning as hierarchical bayes. In International Conference on Learning Representations, 2018.
Ha, David, Dai, Andrew M., and Le, Quoe V. Hypernetworks. In International Conference on Learning Representations, 2016.
He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun, Jian. Deep residual learning for image recognition. International Conference on Computer Vision and Pattern Recognition, 2016.
He, Kaiming, Gkioxari, Georgia, Dollár, Piotr, and Girshick, Ross. Mask r-cnn. In International Conference on Computer Vision, pp. 2980-2988, 2017.
Hinton, Geoffrey E and Plaut, David C. Using fast weights deblur old memories. In 9th Annual Conference of the Cognitive Science Society, 1987.
Hochreiter, Sepp and Schmidhuber, Jürgen. Long short-term memory. Neural computation, 9: 1735-80, 80, 12 1997.
Hochreiter, Sepp. Younger, A. Steven, and Conwell, Peter R. Learning to learn using gradient descent. In International Conference on Articial Neural Networks, 2001.
Hornik, Kurt, Approximation capabilities of multilayer feedforward networks. Neural Networks, (2):251-257, 1991.
loffe, Sergey and Szegedy, Christian. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, 2015.
Javed, Khurram and White, Martha. Meta-leariing representations for continual learning, arXiv preprint arXiv: 1tt.//99...288.... 2019.
Kim, Taesup, Yoon, Jaesik, Dia, Ousmane, Kim, Sungwoong, Bengio, Yoshua, and Ahn, Sungjin. Bayesian model-agnostic meta-learning. arXiv preprint arXiv:1806.03888 2018.
Kingma, Diederik P. and Ba, Jimmy. Adam: A Method for Stochastic Optimization. In International
Conference on Learning Representations, 2015.
Lacoste, Alexandre, Oreshkin, Boris, Chung, Wonchang, Boquet, Thomas, Rostamzadeh, Negar, and Krueger, David. Uncertainty in multitask transfer learning. In Advances in Neural Information Processing Systems, 2018.
Lake, Brenden, Salakhutdinov, Ruslan, Gross, Jason, and Tenenbaum, Joshua. One shot learning of simple visual concepts. Proceedings of the Annual Meeting of the Cognitive Science Society, 2011.
Lake, Brenden M., Salakhutdinov, Ruslan, and Tenenbaum, Joshua B. Human-level concept learning through probabilistic program induction. Science, 135066266: 2015.
John M. Introduction to Smooth Manifolds. Springer, 2003.
Lee, Kwonjoon, Maji, Subhransu, Ravichandran, Avinash, and Soatto, Stefano. Meta-learning with differentiable convex optimization. In CVPR, 2019.
Lee, Yoonho and Choi, Seungjin. Meta-Learning with Adaptive Layerwise Metric and Subspace. In International Conference Machine Learning, 2018.
Li. Ke and Malik, Jitendra. Learning to optimize. International Conference on Machine Learning, 2016.
12

---

Published as a conference paper at ICLR 2020
Li, Zhenguo, Zhou, Fengwei, Chen, Fei, and Li. Hang. Meta-sgd: Learning to learn quickly for few-shot learning. arXiv preprint arXiv:1707.09835, 2017.
Li, Zhizhong and Hoiem, Derek. Learning without forgetting. In European Conference Computer Vision, 2016.
Liu, Hao, Socher, Richard, and Xiong. Caiming. Taming MAML: Efficient unbiased meta- reinforcement learning. In International Conference on Machine Learning, 2019.
Martens, James. Deep learning via hessian-free optimization. In International Conference on Machine Learning, 2010.
Martens, James and Grosse, Roger. Optimizing neural networks with kromecker-factored approximate curvature. In International Conference on Machine Learning, 2015.
Mendonca, Russell, Gupta, Abhishek, Kralev, Rosen, Abbeel, Pieter, Levine, Sergey, and Finn, arXiv 00946, 2019.
Chelsea. Guided meta-policy search. arXiv preprint arXiv: 1ww..99440000
Metz. Luke, Maheswaranathan, Niru, Cheung, Brian, and Sohl-Dickstein, Jascha. Meta-learning update rules for unsupervised representation learning. In International Conference on Learning Representutions, 2019.
Miconi, Thomas, Clune, Jeff, and Stanley, Kenneth O. Differentiable plasticity: training plastic neural networks with backpropagation. In International Conference Machine Learning, 2018.
Miconi, Thomas, Clune, Jeff, and Stanley, Kenneth O. Backpropamine: training self-modifying
neural networks with differentiable neuromodulated plasticity. In International Conference on Learning Representations, 2019.
Mishra, Nikhil, Rohaninejad, Mostafa, Chen, Xi, and Abbeel, Pieter. A Simple Neural Attentive Meta-Leamer. In International Conference on Learning Representations, 2018.
Mujika, Asier, Meier, Florian, and Steger, Angelika, Fast-slow recurrent neural networks. In Advances in Neural Information Processing Systems, 2017.
Munkhdalai, Tsendsuren, Yuan, Xingdi, Mehri, Soroush, Wang, Tong, and Trischler, Adam. Learning rapid-temporal adaptations. In International Conference Machine Learning. 2018.
Nichol, Alex, Achiam, Joshua, and Schulman, John. On First-Order Meta-Learning Algorithms. arXiv preprint ArXiv: 02999, 2018.
Nocedal, Jorge and Wright, Stephen. Numerical optimization. Springer, 2006.
Oreshkin, Boris N, Lacoste, Alexandre, and Rodriguez, Paul. Tadam: Task dependent adaptive metric for improved few-shot learning. In Advances in Neural Information Processing Systems, 2018.
Park, Eunbyung and Oliva, Junier B. Meta-curvature. arXiv preprint arXiv: 1902:09900.0356,, 2019.
Pascanu, Razvan and Bengio, Yoshua. Revisiting natural gradient for deep networks. In International Conference on Learning Representations, 2014.
Perez, Ethan, Strub, Florian, De Vries, Harm, Dumoulin, Vincent, and Courville, Aaron. Film: Visual reasoning with a general conditioning layer. In Association for the Advancement of Artificial Intelligence, 2018
Qiao, Siyuan, Liu, Chenxi, Shen, Wei, and Yuille, Alan L. Few-shot image recognition by predict- ing parameters from activations. In International Conference on Computer Vision and Pattern Recognition, 2018.
Ravi, Sachin and Larochelle, Hugo. Optimization as a model for few-shot learning. In International Conference on Learning Representations, 2016.
Rebuffi, Aylvestre-Alvise, Bilen, Hakan, and Vedaldi, Andrea. Learning multiple visual domains with residual adapters. In Advances in Neural Information Processing Systems, 2017.
Ren, Mengye, Triantafillou, Eleni, Ravi. Sachin, Snell, Jake, Swersky, Kevin, Tenenbaum, Joshua B., Larochelle, Hugo, and Zemel, Richard S. Meta-learning for semi-supereiieed few-shot classification. In International Conference on Learning Representations, 2018.
13

---

Published as a conference paper at ICLR 2020
Rusu, Andrei A., Rao, Dushyant, Sygnowski, Jakub, Vinyals, Oriol, Pascanu, Razvan, Osindero, Simon, and Hadsell, Raia. Meta-learning with latent embedding optimization. In International Conference Learning Representations, 2019.
Saxe, Andrew M, McClelland, James L. and Ganguli, Surya. Exact solutions to the nonlinear
dynamics of learning in deep linear neural networks. arXiv preprint arXiv: 13/2.6/3/, 2013.
Schäfer, Anton Maximilian and Zimmermann, Hans-Georg. Recurrent neural networks are universal approximators. International Journal of Neural Systems, 17(04):253-28) 263, 2007.
Schmidhuber, Jürgen. Evolutionary principles in self-referential learning. PhD thesis, Technische Universität München, 1987.
Schmidhuber, Jürgen. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation, 4(1):131-13, 1992.
Snell, Jake, Swersky, Kevin, and Zemel, Richard S. Prototypical Networks for Few-shot Learning. In Advances in Neural Information Processing Systems, 2017.
Suarez, Joseph. Language modeling with recurrent highway hyperneworks.. In Advances in Neural Information Processing Systems, 2017.
Sutskever, Ily,, Martens, James, Dahl, George, and Hinton, Geoffrey. On the importance of ini- tialization and momentum in deep learning. In International Conference on Machine Learning, 2013.
Thrun, Sebastian and Pratt, Lorien. Learning to learn: Introduction and overview. In In Learning To Learn. Springer, 1998.
Vinyals, Oriol, Blundell, Charles, Lillicrap, Timothy, Kavukcuoglu, Koray, and Wierstra, Daan. Matching Networks for One Shot Learning. In Advances in Neural Information Processing Systems, 2016.
Wang, Jane X., Kurth-Nelson, Zeb, Tirumala. Dhruva, Soyer, Hubert, Leibo, Joel Z., Munos, Rémi, Blundell, Charles, Kumaran, Dharshan, and Botvinick, Matthew. Learning to reinforcement learn. In Annual Meeting of the Cognitive Science Society, 2016.
Wu, Yuhuai, Ren, Mengye, Liao, Renjie, and Grosse, Roger Understanding short-horizon bias in stochastic meta-optimization. In International Conference on Learning Representations, 2018.
Zhou, Fengwei, Wu, Bin, and Li, Zhenguo. Deep meta-learning: Learning learn in the concept space. arXiv preprint arXiv: 1802. 035.03396, 2018
Zintgraf, Luisa M., Shiarlis, Kyriacos, Kurin, Vitaly, Hofmann, Katja, and Whiteson, Shimon. Fast Context Adaptation via Meta-Learning International Conference on Machine Learning, 2019.
14

---

Published as a conference paper at ICLR 2020
APPENDIX
A WARPGRAD DESIGN PRINCIPLES FOR NEURAL NETS
task
warp
conv block
h
3
4x linear
4x linear
conv
LSTMM
BN
LSTM
conv block
h
h
COTVV
BN
linear
linear
(a) Warp-ConvNet
(b) Warp-ResNet
(c) Warp-LSTM
(d) Warp-HlypenNewwork
Figure 6: Illustration of possible WarpGrad architectures. Orange represents task layers and blue represents warp-layers. @ denotes residual connections and @ any form of gating mechanism. We can obtain warped architectures by interleaving task- and warp-layers (a, c) or by designating some layers in standard architectures as task-adaptable and some warp-layers (b, d).
WarpGrad is a model-embedded meta-learned optimiser that allows for several implementation strategies. To embed warp-layers given a task-learner architecture, we may either insert new warp- layers in the given architecture or designate some layers as warp-layers and some as task layers. We found that WarpGrad can both be used in high-capacity mode, where task-learners weak to avoid overfitting, as well as a low-capacity mode where task-learners are powerful and warp-layers relatively weak. The best approach depends on the problem at hand. We highlight three approaches to designing WarpGrad optimisers, starting from a given architecture:
relatively
(a) Model partitioning. Given a desired architecture, designate some operations as task-adaptable and the rest as warp-layers. Task layers do not have interleave exactly with warp-layers as gradient warping arises both through the forward pass and through backpropagation. This was how we approached the magee and minilmageNNe experiments.
(b) Model augmentation. Given model, designate all layers task-adaptable and interleave warp- layers. Warp-layers can be relatively weak as propaggtnnt through non-linear activations ensures expressive gradient warping. This was our approach to the Omniglot experiment; our main architecture interleaves linear warp-layers in a standard architecture.
(c) Information compression. Given model, designate all layers warp and interleave weak task layers. In this scenario, task-learners are prone to overfitting. Pushing capacity into the warp allows it to encode general information the task-learner can draw during task adaptation. This approach is similar to approaches in transfer and meta-learning that restrict the number of free parameters during task training (Rebuffi al., 2017: Lee & Choi, 2018: Zintgraf et al., 2019).
Note that in either case, once warp-layers have been chosen, standard backpropagation automatically warps gradients for us. Thus, WarpGrad is fully compatible with any architecture, for instance, Residual Neural Networks (He et al., 2016) or LSTMs. For convolutional neural networks, we may any form of convolution, learned normalization (e.g. Ioffe & Szegedy, 2015), adaptor module (e.g. Rebuffi et al., 2017; Perez et al., 2018) to design task and warp-layers. For recurrent networks, stacked I. STMS to interleave warped layers, as well any type of HyperNetwork
15

---

Published as a conference paper at ICLR 2020
architecture (e.g. Ha et al., 2016; Suarez, 2017; Flennerhag et al., 2018) partitioning of fast and slow weights (e.g. Mujika et al., 2017). Figure illlsstrates this process.
B WARPGRAD META- TRAAIINGINING ALGORITTMS
In this section, provide the variants of WarpGrad training algorithms used in this paper. Algo- rithm describes simple online algorithm, which accumulates veta-gradients online during task adaptation. This algorithm has constant memory and scales linearly in the length of task trajectories. In Algorithm& 2, we describe an offline meta-training algorithm. This algorithm is similar to Algo- rithm in many respects, but differs in that we do not compute meta-gres gradients online during task adaptation. Instead, we accumulate them into replay buffer of sampled task parameterisations. This buffer is Monte-Carlo sample of the expectation in the meta objective (Eq. 13) that can be thought of as dataset in its own right. Hence, can apply standard mini-batching with respect to the buffer and perform mini-batch gradient descent on warp parameters. This allows us to update warp parameters several times for given sample of task parameter trajectories, which greatly improve data efficiency. our Omniglot experiment, we found offline meta veetatraiiiing to converge faster: in fact, mini-batch size of (i.e. 7 1 in Algorithm 2 converges rapidly without any instability.
Finally, Algorithm 3, we present a continual meta-training process where meta-training occurs throughout stream of learning experiences. Here, C represents multi-task objective, such as the average task loss, C Σ-<p(>) LTss. Meta-lcarning arises by collecting experiences continuously (across different tasks) and using these to accumulate the meta-gradient online. Warp parameters are updated intermittently with the accumulated meta-gradient We use this algorithm in our maze navigation experiment, where task adaptation is internalised within the RNN task- learner.
C WARPGRAD OPTIMISERS
In this section, we detail WarpCirad methods used in our experiments.
Warp-MAML We use this model for few-shot learning (Section 4.1). We use the full warp- objective in Eq. 11 together with the MAML objective (Eq. 1),
Warp MAML I(O)) + ACMAML (0)).
(14)
where MAMI. L MAMI. under the constraint P = I. In experiments, we trained Warp-MAML using the online training algorithm (Algorithm 1).
Warp-Leap We this model for multi-shot meta-learning. It defined by applying Leap (Flen- nerhag et al., 2019) to & (Eq. 16),
J Warp- L(d) + XC
(15)
where the Leap objective is defined by minimising the expected cumulative chordal distance,
K-
(lo)
sg [+]]
0%
12-1112
0r = -ff......,
0 Link (0f; Φ)).
(16)
Tru((T) k=
Note that the Leap meta-gradiem makes a first-order approximation to avoid backpropagating through the adaptation process. It given by
K- A CT
CT (0%_1i0) + A
vClepp(eu) x
(17)
10 07-1112
Trup(T)
where ACT.& (0]:0)= = LT (0]: Φ) - Siunk (0) and A = - - - In our experiments,
train Warp-Leepp using Algorithm in the multi-shot lieredImageNet experiment and Algorithm!
16

---

Published as a conference paper at ICLR 2020
in the Omniglot experiment. We perform ablation study for training algorithms, comparing exact (Eq. 11) versus approximate (Eq. 12) meta-objectives, and several implementations of the warp-layers on Omniglot in Appendix F.
Warp-RNN For our Reinforcement Learning experiment, we define WarpGrad optimiser by meta-learning an LSTM that modulates the weights of the task-learner (see Appendix for details). For this algorithm, we face a continuous stream of experiences (episodes) that we meta-leam using our continual meta-training algorithm (Algorithm 3). In our experiment, both Lusk and L are the advantage actor-critic objective (Wang al., 2016); C is computed on one batch of episodes, whereas I. is accumulated over y/ = 30 such batches, for a total of 900 episodes. As each episode involves 300 steps in the environment, we cannot apply the exact meta objective, but use the approxi- mate meta objective (Eq. 12). Specifically, let E {so, 01, $1, ..... ST, (T, IT, ST-11 } denote an episode on task т, where s denotes state, a action, and r instantaneous reward. Denote a mini- batch of randomly sampled task episodes by E {E"}r<p(r) and ordered set of & consecutive mini-batches by cK = (E k-1 Then (( c = 1/n EE,Eg* ΣE, F (6: 0, Ei,) and
Algorithm 1 WarpGrad: online meta-training
Algorithm WarpGrad: offline meta-training
Require: p(r): distribution over tasks
Require: P(T): distribution over tasks
Require: a, X: hyper-parameters
Require: 3, x, hyper-parameters
1: initialise Φ and 0g
initialise & and 0g
2: while not done do
while not done do
3: Sample mini-batch of tasks B from p(r)
3: Sample mini-batch of tasks B from p(r)
4: Jé. 90,
4: T + {r: [0)] for T in B}
5:
for all r E B do
6: 0 t 0g
5:
for all E B do
7:
for all k in 0, K--1k do
0ī + 6
8:
0 0% a V VT Lick (0%: Φ)
7:
for all l: in 0,
K--10k do
9:
9o Is + L( 0%)
0 - 0
a V V (0T
10:
90, + 90. + VC(6,; 0
appea
end for end for
12:
10:
end for end for
Φ 5- Φ - 3ge
11:
14:
Oo 0o X
12:
i, H6s 96. +-
15: end while
13:
while T not empty do
14:
sample t, without replacement
Algorithm 3 WarpGrad: continual meta-tres training
15:
Jo + 90 + ( 0
Require: p(7): distribution tasks
16:
go. 90, + VC 0
Require: a, 8, >, 17. syper-parameters
17:
< i+1
initialise & and 9
18:
if i n then
2: i, 9. f- 0
19:
@ - Φ 8gg 0o 0n X
3: while not done do
20:
4: Sample mini-batch of tasks B from p(r)
21:
i, go. 90. + 0
5: for all r E B do
end
6: 96 + 96 v o; 0)
23:
end while
7: ge + ge + (C (0:
24:
end while
8:
end for
9:
f +- XB38
10:
go. i +- 0,
11:
if = " then
12:
3g
13:
i. 1e + 0
end if
15: end while
17

---

Published as a conference paper at ICLR 2020
C.multi (8; Ek) 1/n' E/n''ET,, EB, EB, (0: Φ, Ek,y). where and n' are normalising constants. The
Warp-RNN objective defined by
[[(o; Edh + AC Et) AC (0;Es) E&)
if k: =
JWarp-NN
(18)
otherwise.
WarpGrad for Continual Learning For this experiment, we focus on meta-learning warp- parameters. Hence, the initialisation for each task sequence fixed random initialisation, (i.e. XC(09) 0). For the warp meta-objective, we take expectations over N task sequences, where each task sequence a sequence of T sub-tasks that the task-learner observes one at a time; thus while the task loss is defined over the current sub-task, the meta-loss averages of the current and all prior sub-tasks, for each sub-task in the sequence. See Appendix for detailed definitions. Importantly, because WarpGrad defines task adaptation abstractly by probability distribution, plement a continual learning objective by modifying the joint task parameter distribution p(T, 0) that use in the meta-objective (Eq. 11). A task defines sequence of sub-tasks which we generate parameter trajectories 0" Thus, the only difference from multi-task meta-learning is that parameter trajectories are not generated under a fixed task, but arise as a function of the continual learning algorithm used for adaptation. We define the conditional distribution p(O () as before by sampling sub-task parameters H from a mini-batch of such task trajectories, keeping track of which sub-task it belongs to and which tasktasks came before it in the given task sequence T. The meta meta-objective is constructed, for any sub-task parameterisation d as (0") = Ei= fiosk (0": Di; ②), where D, data from sub-task (Appendix J). The meta-objective is an expectation over task parameterisations,
readily im-
i==1
T
(0) = Σ trup(T) 1=1 (T)mp((0)7z)
Lineta (0):
(19)
D SYNTHETIC EXPERIMENT
To build intuition for what means warp space, we construct a simple 2-D problem over loss surfaces. A learner is faced with the task of minimising an objective function of the form x2) = ge (xx) exp(gg (x2)) 9 (ax) exp(g] (D((g[(x,,1))) (2), g exp(ge (z1)). where each task f" is defined by scale and rotation functions g" that are randomly sampled from predefined distribution. Specifically, each task defined by the objective function
x2)) (ā x2)²² exp(-x7 (x2 + a5)²)
b5(21/8" x - -x1) expl x3)) b3 exp(- hap(-((e (z, + az)² z+)),
where each a, and s are randomly sampled parameters from
8* ~ Cat (1, .., ,, 10) a. ~ Cat(-1,0,,]
6ī ~ Cat(-5, -4...,,,
The task is minimise the given objective from a randomly sampled initialisation. ((==12)) 3-3,33) During meta-training, we train on a task for 100 steps using learning rate 0.1. Each task has unique loss-surface that the learner traverses from the randomly sampled initialisation. While each loss-surface is unique, they share an underlying structure. Thus, by meta- learning a warp over trajectories on randomly sampled loss surfaces, we expect WarpGrad to learn a warp that is close invariant spurious descent directions. In particular, WarpGrad should produce a smooth warped space that is quasi quasi-convex for any given task to ensure that the task-learner finds minimum as fast as possible regardless of initialisation.
To visualise the geometry, we use an explicit warp Ω defined by 2-layer feed-forward network with a hidden-state size of 30 and tanh ooneaiirasss.es. We train warp parameters for 100 meta-training steps;
18

---

Published as a conference paper at ICLR 2020
Figure 7: Example trajectories on three task loss surfaces. We start Gradient Descent (black) and WarpGrad (magenta) from the same initialisation; while SGD struggles with the curvature, the WarpGrad optimiser has learned warp such that gradient descent in the representation space (top) leads to rapid convergence model parameter space (bottom).
in each meta-step we sample new task surface and mini-batch of 10 random initialisations that we train separately. We train to convergence and accumulate the warp meta-grade geota-gradeent online (Algorithm 1). We evaluate against gradient descent randomly sampled loss surfaces (Figure 7). Both optimisers start from the same initialisation, chosen such that standard gradient descent struggles; we expect the WarpGrad optimisers to learn geometry that is robust the initialisation (top row). This is indeed what we find; the geometry learned by WarpGrad smoothly warps the native loss surface into well-behaved space where gradient descent converges to local minimum.
E OMNIGLOT
We follow the protocol of Flennerhag al. (2019), including the choice of otherwise noted. In this setup, each of the 50 alphabets that comprise the dataset constitutes distinct task. Each task is treated as a 20-way classification problem. Four alphabets have fewer than 20 characters in the alphabet and are discarded, leaving us with 46 alphabets in total. 10 alphabets are held-out for final meta-testing; which alphabets are held out depend on the seed account for variations across alphabets; we train and evaluate all baselines on 10 seeds. For each character in an alphabet, there are 20 raw samples. Of these, are held out for final evaluation on the task pre-processed by random affine transformations in the form of (a) scaling between [0.8, 1.2), (b) rotation [0, 360), and (c) cropping height and width by factor of [-0.2, 0.2] in each dimension. This ensures tasks too hard for few-shot learning. During task adaptation, mini-batches are sampled at random without ensuring class-balance (in contrast few-shot classification protocols (Vinyals et al., 2016)). Note that benchmarks under this protocol are not compatible with few-shot learning benchmarks.
unless
while the remainder is used to construct a training set. Raw samples
We use the same convolutional neural network architecture and syper-parameters as Flennerhag et al. (2019). This learner stacks a convolutional block comprised of a 3 x 3 convolution with 64 filters, followed by 2 max-pooling, Datch-normalliatioon, and ReLU activation, four times. All images are down-sampled to 28 x 28, resulting in 1 x 1 64 feature map that is passed on to final linear layer. We create Warp Leap meta-leaman that inserts warp-layers between each convolutional block, W w o 1 .... o - ht,, where each J is defined as above. In our main experiment, each we simply 3 x convolutional layer with zero padding: in Appendix we consider both simpler and more sophisticated versions. We find that relatively simple warp-layers do quite well. However, adding capacity does improve generalisation performance. We meta-learn the initialisation of task parameters using the Leap objective (Eq. 16), detailed in Appendix C.
Both Civ and Tis are defined as the negative log-likelihood loss; importantly, we evaluate them on different batches of task data to ensure warp-layer layers encourage generalisation. We found additional benefit in this experiment from using held-out data to evaluate L We use the offline meta-training algorithm (Appendix B, Algorithm 2); in particular, during meta-training, we sample mini-batches
19

---

Published as a conference paper at ICLR 2020
Warp-Leap
Leap
Reptile
FT
SGD*
KFACE
Test accuracy held-out tasks
0.8
0.7
0.5
0.4
1
5
10
15
20
25
Number of tasks meta- trainriinnng set
Warp-Leap
Leap
Reptile
SGD
KFAC
Train Acc AUC held-out, tasks
0.7
0.6
0.5
0.4
0.3
10
15
20
25
Number of tasks in teatt-trainnng set
Figure 8: Omniglot results. Top: test accuracies held-out tasks after meta-training varying number of tasks. Bottom: AUC under accuracy curve on held-out tasks after meta-training on varying number of tasks. Shading represents standard deviation across 10 independent runs. We compare between Warp-Leap. Leap, and Reptile, multi-heades finetuning, as well as SGD and KFAC which used random initialisation but with 10x larger learning rate.
of 20 tasks and train task-learners for 100 steps to collect 2000 task parameterisations into a replay buffer. Task-leamnrs share common initialisation and warp parameters that held fixed during task adaptation. Once collected, we iterate over the buffer by randomly sampling mini-batcher of task parameterisations without replacement. Unless otherwise noted, we used batch size 1) For each mini-be batch, we update é by applying gradient descent under the canonical meta-objection data from the corresponding
(Eq. 11), where we evaluate C. on a randomly sampled mini-bater task. Consequently, for each meta-batch, we take (up to) 2000 meta-gran meta-gradient steps warp parameters 6. We find that this form of mini-batching causes the meta-training loop to converge much faster and induces discernible instability.
We compare Warp-Leap against no meta-learning with standard gradient descent (SGD) or KFAC (Martens & Grosse, 2015). We also benchmark against baselines provided in Flennerhag et al. (2019); Leap, Reptile (Nichol et al., 2018), MAML, and multi-headed fine-tuning. All learners benefit substantially from large batch sizes as this enables higher learning rates. To render no- pretrraiiing competitive option within a fair computational budget, we allow SGD and KFAC to use 4x larger batch sizes, enabling 10x larger learning rates.
20

---

Published as a conference paper at ICLR 2020
Table 2: Mean test error after 100 training steps on held out evaluation tasks. Multi-headed. No
meta-training, but 10x larger learning rates).
Method
WarpGraa
Leap
Reptile
Finetuning"
MAML
KFAC!
SGD!
No. Meta-training tasks
49.5
7.8
37.6
1 4.8
40.4
+ 4.0
40.0
56.0
51.0
68.8
2.8
53.4
3.1
53.1
4.2
64.6 +
48.6 2.5
56.0
51.0
75.0
3.6
59.5
3.7
58.3
3.3
67.7
2.8
51.6
56.0
51.0
10
81.2 2.4
67.4
2.4
65.0
2.1
71.3
71.3120 2.0
54.1128
56.0
51.0
15
82.7 + 3.3
70.0
+ 2.4
66.6+22
73.5
+
54.8
+
56.0
$1.0
20
82.0
2.6
73.3
+: 2.3
+.4
75.4
3.2 3.2
56.6
2.0
56.0
51.0
25
83.8
1.9
74.8
70.8 1.9
76.4 =
56.7
+ 2.1
56.0
51.0
F ABLATION STUDY: WARP LAYERS, META-OBJECTIVE, AND META-TRAINING
WarpGrad provides a principled approach for model-informed meta-learning and offers several degrees of freedom. To evaluate these design choices, we conduct an ablation study Warp-Leap where we vary the design of warp-layers as well as meta-training approaches. For the ablation study, we fixed the number of pretraining tasks to 25 and report final test accuracy over 4 independent runs. All ablations use the same hyper- parameeras,,trss, except for online meta-training which uses a learning rate of 0.001.
First, we vary the meta-training protocol by (a) using the approximate objective (Eq. 12), (b) using online meta-training (Algorithm 1), and (c) whether meta-learning the learning rate used for task adaptation is beneficial in this experiment. We meta-learn a single scalar learning rate (as warp parameters can learn layer-wise scaling). Meta-gradients for the learning rate are clipped at 0.001 and we use learning rate of 0.001. Note that when using offline meta-training, we store both task parameterisations and the momentum buffer in that phase and use them in the update rule when computing the canonical objective (Eq. 11).
Further, vary the architecture used for warp-layers. We study simpler versions that channel- wise scaling and more complex versions that use non-linearities and residual connections. We also evaluate a version where each warp-layer has stacked convolutions, where the first warp convolution outputs 128 filters and the second warp convolution outputs 64 filters. Finally, in the two-layer warp-architecture, we evaluate a version that inserts a FiLM layer between the two warp convolutions. These are adapted during task training from O initialisation; they amount task embeddings that condition gradient warping on task statistics. Full results are reported in Table 3.
G ABLATION STUDY: WARPGRAD AND NATURAL GRADIENT DESCENT
Here, we perform ablation studies to compare the geometry that WarpGrad optimiser learns to the geometry that Natural Gradient Descent (NGD) meth- ods represent (approximately). For con- sistency, we run the ablation Om- niglot. As computing the true Fisher In- formation Matrix intractable, we can compare WarpGrad against two com- mon block-diagonal approximations, KFAC (Martens & Grosse, 2015) and Natural Neural Nets (Desjardins et al., 2015).
Table 4: Ablation study: mean test error after 100 training steps on held out evaluation tasks from a random initialisa- tion. Mean and standard deviation over 4 seeds.
Method
Preconditioning
Accuracy
SGD KFAC (NGD)
None
40.1 + 6.1 58.2 3.2
Linear (block-diagonal)
WarpGrad WarpGrad
Linear (block-diagonal)
68.0 + 4.4
Non-linear (full)
81.3 4.0
First, we isolate the effect of warping task loss surfaces by fixing random initialisation and only meta-learning warp parameters. That is. in this experiment, set XC(0") O. We compare against two baselines, stochastic gradient descent (SGD) and KFAC, both trained from a random initialisation. We use task mini-batch sizes of 200 and task learning rates of 1.0, otherwise
21

---

Published as a conference paper at ICLR 2020
Table Ablation study: mean test error after 100 training steps on held out evaluation tasks. Mean and standard deviation over 4 independent runs. Offline refers to offline meta-training (Appendix B), online to online veta-training Algorithm 1: full denotes Eq. 11 and approx denotes Eq. Batch Normalization (loffe & Szegedy, 2015); 'equivalent to FiLM layers (Perez et al., 201 2018); Residual connection (He et al., 2016), when combined with BN, similar to the Residual Adaptor architec- ture (Rebuffi et al., 2017); FiLLM task embeddings.
Architecture
Meta-training
Meta-objective
Accuracy
None (Leap) x 3 conv (default)
Online
None
74.8
Offline
full (L, Ey. II)
84.4
x 3 conv
Offline
approx (f, Eq. 12)
83. 2.7
x 3 conv
Online
76.3
3
Offline
full, learned a
83.1
Scaling! conv
Offline
full
77.5 + 1.8
Offline
79.4
3 ReLU
Offline
3 conv
BN
Offline
3 conv BN
ReLU
Offline
85.0
3 conv BN' Res + ReLU
Offline
86.3
2-layer x 3 conv BNT Res
Offine
88.0 1.0
2-layer x 3 conv BNT Res TA
Offine
full
88.1 1.0
0.5
pee verp past warp
pre-warp
044
Shattenn- Norm of Cor
0.3
Inpacted
0.2
+
Layer
Figure 9: Ablation study, Left: Comparison of mean activation value B[h(z)] across layers, pre- and post-warping. Right: Shatten-1 gathered on held-out test set and averaged over tasks and adaptation steps.
of Cov(M()), 1, pre- and post-norm. Statistics
the same hyper-paramen parameters in the main experiment. For WarpGrad, we meta-train with these hyper-parameters well. We evaluate two WarpGrad architectures, one, we use linear warp-layers. which gives a block-diagonal preconditioning. as in KFAC. In the other, we use our most expressive warp configuration from the ablation experiment in appendix F, where warp-layers are two-layer convolutional block with residual connections, batch normalisation, and ReLU activation. We find that warped geometries facilitate task adaptation on held-out tasks to a greater degree than either SGD or KFAC by significant margin (table 4). We further find that going beyond block-diagonal preconditioning yields a significant improvement in performance.
Second, explore whether the geometry that meta-leam under in the full Warp-Leap al- gorithm is approximately Fisher. In this experiment, we use the main Warp-Leap architecture. We use meta-leamer trained 25 tasks and that evaluate 10 held-out tasks. Because warp-layers are linear in this configuration, if the learned geometry is approximately Fisher, post-warp activations should be zero-centred and the layer-wise covariance matrix should satisfy (ov(w(() (((())))))))))))))))))))) (x)), (9) (H() (z))) = where is the identity matrix (Desjardins et al., 2015). If true, Warp-Leap would learn a block diagonal approximation to the Inverse Fisher Matrix, as Natural Neural Nets.
22

---

Published as a conference paper at ICLR 2020
To test this, during task adaptation held-out tasks, compute the mean activation in each convolutional layer pre- and post-warping. We also compute the Shatten-1 norm of the difference between layer activation covariance and the identity matrix pre- and post-warping, described above. We average statistics over task and adaptation step (we found no significant variation in these dimensions).
Figure 9 summarise our results. We find that, in general, WarpGrad- Leap has zero-centered post-warp activations. That pre-warp activations are positive is an artefact of the ReLU activation function. However, we find that the correlation structure is significantly different from what we would expect Warp-Leap were to represent the Fisher matrix; post-warp covariances are significantly dissimilar from the identity matrix and varies across layers.
These results indicate that WarpGrad methods behave distinctly different from Natural Gradient Descent methods. One possibility is that WarpGrad methods do approximate the Fisher Information Matrix, but with higher accuracy than other methods. A more likely explanation is that WarpGrad methods encode different geometry since they can learn to leverage global information beyond the task at hand, which enables them to express geometries that standard Natural Gradient Descent cannot.
H MiniIMAGENET AND riered
minilmageNNet This dataset is subset of 100 classes sampled randomly from the 1000 base classes in the ILSVRC-12 training set, with 600 images for each class. Following (Ravi & Larochelle, 201 6), classes are split into non-overlapping meta-training, meta- vela-validation and meta-tests sets with 64, 16, and 20 classes in each respectively.
mageed As described in (Ren et al., 2018), this dataset is a subset of ILSVRC-12 that stratifies 608 classes into 34 higher-level categories in the ImageNet human-curated hierarchy (Deng et al., 2009). In order to increase the separation between meta-train and meta-evaluation splits, 20 of these categories are used for meta-training, while and 8 are used for meta-validation and meta- testing respectively. Slicing the class hierarchy closer to the root creates more similarity within each split, and correspondingly more diversity between splits, rendering the meta-leariig problem more challenging. High-level categories are further divided into 351 classes used for meta-training, 97 for meta-validation and 160 for meta-testing, for a total of 608 base categories. All the training images in ILSVRC-12 12 for these base classes are used to generate problem instances for fiered of which there are a minimum of 732 and a maximum of 1300 images per class.
For all experiments, N-way K-shot classification problem instances were sampled following the standard image classification methodology for meta-learning proposed in Vinyals et al. (2016). A subset of N classes sampled at random from the corresponding split. For each class, K arbitrary images were chosen without replacement to form the training dataset of that problem instance. As usual, disjoint set of T. images per class selected for the validation set.
Few-shot classification In these experiments we used the established experimental protocol for evaluation in meta-validation and meta-testing: 600 task instances were selected, all using N K = or K = 5, as specified, and L = 15. During meta-training we used N 5, K 5 or K- 15 respectively, and L 15.
Task-learners used 4 convolutional blocks defined by with 128 filters (or less, chosen by hyper- parameter tuning). 3 x kernels and strides set to 1, followed by batch normalisation with learned scales and offsets. a ReLU non-lincarity and 2 x 2 max- max-pooling. The output of the convolutional stack (5 128) was flattened and mapped, using a linear layer, to the output units. The last 3 convolutional layers were followed by warp-layers with 128 filters each. Only the final 3 task-layer parameters and their corresponding scale and offset batch-norm parameters were adapted during task-training, with the corresponding warp-layers and the initial convolutional layer kept fixed and meta-learned using the WarpGrad objective. Note that, with the exception of CAVIA, other baselines do worse with 128 filters they overfit; MAML and T-Nets achieve 46% and 49 % 5-way-1-sher test accuracy with 128 filters, compared to their best reported results (48.7% and 51.7%, respectively).
23

---

Published as a conference paper at ICLR 2020
Hyper-pane sapper-praaeterrs were tuned independently for each condition using random grid search for highest test accuracy on meta-validation left-out tasks. Grid sizes were 50 for all experiments. We choose the optimal hyper-parametee parameters (using early stopping at the meta-level) in terms of meta-validatee test set accuracy for each condition and we report test aceuracy on the meta-test set of tasks. 60000 meta- training steps were performed using meta-gradients over single randomly selected task instances and their entire trajectories of 5 adaptation steps. Task-specific adaptation was done using stochastic gradient descent without momentum. We use Adam (Kingma & Ba, 2015) for meta-updates.
Multi-shot classification For these experiments we used N = 10, K = 640 and L = 50. Task- learners are defined similarly, but stacking 6 convolutional blocks defined by x kernels and strides set to 1, followed by batch normalisation with learned scales and offsets, a ReLU non- non-linearity and 2 x max-pooling (first 5 layers). The sizes of convolutional layers were chosen by hyper-parameter tuning to {64, 64, 160, 160, 256, 256). The output of the convolutional stack (2 x 2 x 256) was flattened and mapped, using a linear layer, to the output units.
Hyper-parameters parameters were tuned independently for each algorithm, version, and baseline using random grid search for highest test aceuracy on meta-validation left-out tasks. Grid sizes were 200 for all multi-shot experiments. We choose the optimal syper--prameters in terms of mean veta-validatton test set accuracy AUC (using early stopping at the meta-level) for each condition and we report test accuracy the meta-test set of tasks. 2000 meta-training steps were performed using averaged meta-gradients over random task instances and their entire trajectories of 100 adaptation steps with batch size 64, inner-loops. Task-specific adaptation with momentum (0.9). Meta-gradients were passed to Adam in the outer loop.
done using stochastic gradient descent
We test WarpGrad against Leap, Reptile, and training from scratch with large batches and tuned mo- mentum. We tune all meta-learners for optimal performance on the validation set. WarpGirad outper- forms all baselines both in terms of rate of convergence and final test performance (Figure 10).
Momentum
Reptile
Leop
Warp-Lapp
Test accuracy on held out tasks
0.8
0.7
0.6
0.4
0.3
20
60
80
100
Inner-loop steps
Momentum
0.85
0.80
0.75
0.70
0.65
accuracy
0.60
0.55
0.50
0.45
Meta-training steps (x100)
Figure 10: Multi-shot fieredImageNet results. Top: mean learning curves (test classification accuracy) on held-out meta-test tasks. Bottom: mean test classification performance on held-out meta-test tasks during meta-training. Training from scratch omitted as it is not meta-trained.
24

---

Published as a conference paper at ICLR 2020
1 MAZE NAVIGATION
To illustrate both how WarpGrad may be used with Recurrent Neural Networks in an online meta- learning setting, well as in Reinforcement Learning environment, evaluate it in maze navigation task proposed by Miconi et al. (2018). The environment is a fixed maze and a task is defined by randomly choosing goal location in the maze. During task episode of length 200, the goal location is fixed but the agent gets teleported once it finds it. Thus, during an episode the agent must first locate the goal, then return
as many times as possible, each time being randomly teleported to new starting location. We use an identical setup as Miconi et al. (2019), except our grid x 9. We compare our Warp-RNN to Learning to Reinforcement
of size x as opposed Learn (Wang et al., 2016) and Hebbian meta-learners (Miconi et al., 2018; 2019).
The task-learner in all cases is an advantage actor-critic (Wang et al., 2016), where the actor and critic share an underlying basic RNN, whose hidden state is projected into a policy and value function by two separate linear layers. The RNN has a hidden state size of 100 and tanh non-linearities. Following (Miconi et al., 2019), 9). for all benchmarks, we train the task-learner using Adam with learning rate of le - for 200 000 steps using batches of 30 episodes, each of length 200. Meta- learning arises in this setting each episode encodes different task, the goal location moves, and by learning across episodes the RNN is encoding meta-information in its parameters that it can leverage during task adaptation (via its hidden state (Hochreiter & Schmidhuber, 1997; Wang et al., 2016)). See Miconi et al. (2019) for further details.
We design& a Warp-RNN by introducing warp-layer in the form of LSTM that is frozen for most of the training process. Following Flennerhag et al. (2018), we use this meta-LSTM, to modulate the task RNN. Given an episode with input vector 2i,, the task RNN is defined by
h4 tanh (UZ,VUL, h1-1+ + U2, U2, 2,6 Fe + 6).
(20)
where W, V, b are task-adee parameters; each j,t is diagonal warp matrix produced by projecting from the hidden state of the meta-LSTM, U(6) diag(tanh((p)) ())nnn(((()))))))))))) 2))). where is the
hidden-state of the meta-L-TTM. See Flennerhag et al. (2018) for details. Thus, our Warp-RNN is a form of HyperNetwork (see Figure 6, Appendix A). Because the meta-LSTMM is frozen for most of the training process, task adaptable parameters correspond to those of the baseline RNN,
To control for the capacity of the meta-LSTM, we also train a HyperRNN where the LSTM is updated with every task adaptation; we find this model does worse than the WarpGradRRNN, We also compare the non-linear preconditioning that we obtain in our Warp-RNi RNN linear forms of preconditioning defined in prior works. We implement a T-Nets-RNY -NNN meta-learner, defined by embedding linear projections T,, T2 and Ti that are meta-learned in the task RNN, h, tanh (T,, Vh, + T, W2, + b). Note that we cannot backpropagate to these meta- meta-parameters as per the T-Nets (MAML) framework. Instead, we train Th, Tz, Tr with the meta-objective and veta-ntaining algorithm we use for the Warp-RNN. The T-Nets-RNN does worse than the baseline RNN and generally fails to learn.
We meta-train the Warp-RNN using the continual meta-training algorithm (Algorithm 3, see Ap- pendix B for details), which accumulates meta- petaiggradeent continuously during training. Because task training is a continuous stream of batches of episodes, we accumulating the meta mretiegradient using the approximate objective (Eq. 12, where L and L are both the same advantage actor-critic objec- tive) and update warp-parameters on every 30th task parameter update. We detail the meta-objective in Appendix C (see Eq. 18). Our implementation of a Warp-RNN can be seen meta-learning "slow" weights to facilitate learning of "fast" weights (Schmidhuber, 1992; Mujika et al., 2017). Implementing Warp-RNN requires four lines of code on top of the standard training script. The task-learner is the same in all experiments with the same number of learnalee parameters and hidden state size. Compared to all baselines, we find that the Warp-RNN converges faster and achieves a higher cumulative reward (Figure 4 and Figure 11).
J META-LEARNING FOR CONTINUAL LEARNING
Online SGD and related optimisation methods tend to adapt neural network models to the data distribution encountered last during training, usually leading what has been termed "catastrophic
25

---

Published as a conference paper at ICLR 2020
160
Warp-RNN
HyperRNN
200
Hohb--NN!
Hebb-RNN
T-Not&RNN
50
20000
40000 Number Egiiodes
50000
90000
100000
190
WaYp-RNN
HypeRRNN
100
HoblsRNNi
HNNb
RNN
T-NctsRNN
50
Namber of Pgisodes
Mean cumulative return for maze navigation task, for 200000 training steps. Shading represents inter nnterquurrtle ranges across 10 independent runs. modulation, respectively (Miconi et al., 2019).
Figure 11:
runs,"Simple Simple modulation and retroacctvve
forgetting" (French, 1999). In this experiment, we investigate whether WarpGrad optimisers can meta-learn to avoid this problem altogether and directly minimise the joint objective over all tasks with every update in the fully online learning setting where no past data retained.
Continual Sine Regression We propose continual learning version of the sine regression meta- learning experiment in Finn et al. (2017). We split the input interval [-5, 5] C R evenly into 5 regression tasks. These presented
consecutive sub-intervals, corresponding to a task-learner, which adapts to each sub-task using 20 gradient steps on data from the given sub-task only. Batch sizes task sequence is defined by a target function composed of two randomly mixed sine functions of the form fayd, (x) a; sin(æ bi) each with randomly sampled amplitudes a; E [0.1, 5) and phases b; E [0, T]. A task (a1, b1, be, o) is therefore defined by sampling the parameters that specify this mixture; task specifies target function 9- by
at time
samples. Sub-tasks thus differ in their input domain. A
(x(x) = (o(x))a by (x) + (1 -co(x))) (I))
(21)
where (z) = al + 0) for randomly sampled offset E [[5,5), 5), with being the sigmoid activation function.
Model We define task-learmer as 4-layer feed-forward networks with hidden layer size 200 and ReLU non-linearities to learn the mapping between inputs and regression targets, f(. 0, d). For each task sequence т, task-leamer is initialised from fixed random initialisation Og (that is not
26

---

Published as a conference paper at ICLR 2020
meta-leamed). Each non-linearity is followed by residual warping block consisting of 2-layer feed- forward networks with 100 hidden units and tanh non non-lincarities, with meta-learned parameters ø which are fixed during the task adaptation process.
Continual learning as task adaptation The task target function 9- is partitioned into 5 sets of sub-tasks. The task- sees one partition at a time and is given 20 gradient steps to adapt, for a total of K = 100 steps of online gradient descent updates for the full task sequence; recall that every such sequence starts from fixed random initialisation 0g- The adaptation is completely online since at step k = 1..... K we sample a new mini- bannh-bachh D of 5 samples from single sub-task (sub-interval). The data distribution changes after each = 20 steps with inputs x coming from the next sub-interval and targets form the same function (z). During meta-training we always present tasks in the same order, presenting intervals from left to right. The online (sub-)task loss is defined on the current mini-batch D mask at step k:
LTukk (0% D o)
Σ task De DED
-
(22)
2 D
Adaptation to each sub-task uses sub-task data only to form task parameter updates 02+1 4 0% - a. fask (0f, ) v). We used constant learning rate 0.001. Warp-parameters 6 are fixed across the full task sequence during adaptation and are meta-learned across random samples of task sequences, which describe next.
Meta-learning an optimiser for continual learning To investigate the ability of WarpGrad to learn optimiser for continual learning that mitigates catastrophic forgetting, fix random initialisation prior to meta-training that is not meta-learned; every task-learner initialised with these parameters. To meta-learn optimiser for continual learning, need a meta-objective that encourages such behaviour. Here, we take a first step towards a framework for meta-learned continual learning. We define the meta-objective L as an incremental multitask objective that, for each sub-task!T, in a given task sequence T, averages the validation sub-task losses (Eq, 22) for the current and every preceding loss in the task sequence. The task meta-objective is defined by summing over all sub-tasks in the task sequence. For some sub-task parameterisation f", we have
C (8:
Σ (T - -t++1 Clusk (0 (P.tttt))
(23)
As before, the full meta-objective is an expectation over the joint task parameter distribution (Eq. 11); for further details on the meta-objective, see Appendix C, Eq. 19. This meta-objective gives equal weight to all the tasks in the sequence by averaging the regression step loss over all sub-tasks where a prior sub-task should be learned remembered. For example, losses from the first sub-task, defined using the interval [-5.-3], will appear nT times in the meta-objective. Conversely, the last sub-task sequence, defined on the interval [3. 5], is learned only the last n 20 steps of task adaptation, number of appearances corrects for
and hence appears times in the meta-objective. Normalising this bias. We trained warp-pane parpmpraeeters using Adam and a meta-learning rate of 0.001, sampling random tasks to form meta-batel and repeating the process for 20 000 steps of meta- meta-training.
Results Figure 12 shows a breakdown of the validation loss across the 5 sequentially learned tasks over the 100 steps of online learning during task adaptation. Results are averaged over 100 random regression problem instances. The meta-learned WarpGrad optimiser reduces the loss of the task currently being learned in each interval while also largely retaining performance previous tasks. There is an immediate relatively minor loss of performance, after which performance on previous tasks is retained. We hypothesise that this is because the meta-objectives averages over the full learning curve, as opposed to only the performance once task has been adapted to. As such, the WarpGrad optimiser may allow for some degree of performance loss. Intriguingly. in all cases, after an initial spike in previous sub-task losses when switching to new task, the spike starts to revert back some way towards optimal performance, suggesting that the WarpGrad optimiser facilitates positive backward transfer, without this being explicitly enforced in the meta-objective. Deriving a principled meta-objective for continual learning is an exciting area for future research.
27

---

Published as a conference paper at ICLR 2020
411
EEEE E
(a) Task order during mela-training.
(b) Random task order.
Figure 12: Continual learning regression experiment. Average log-loss over 100 randomly sampled tasks. Each task contains sub-tasks learned (a) sequentially seen during meta-training or (b) in random order (sub-task 1, 3, 4, 2, 0J. We train on each sub-task for 20 steps, for total of K 100 task adaptation steps.
b354
ta5k3
task2
(a) Task order seen during meta-training.
(b) Random task order.
Figure 13: Continual learning regression: evaluation after partial task adaptation. We plot the ground truth (black), task-learner prediction before adaptation (dashed green) and task-learner prediction after adaptation (red). Each row illustrates how task-learner predictions evolve (red) after training on sub-tasks up and including that sub-task (current task illustrate in plot). (a) sub-tasks presented the same order as seen during meta-training; (b) sub-tasks are presented in random order meta-test time in sub-task order [1, 3, 4, and 0].
28


Reviews:
[{"review_id": "r1g_Ddk0FS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose warped gradient descent (WarpGrad) an optimisation framework for facilitating gradient-based meta-learning. WarpGrad interleaves within the learner meta-learned warp-layers that implicitly precondition the gradients of the task-specific parameters during backpropagation. In contrast to the linear projection layers employed in T-Nets, warp-layers are unrestricted in form and induce a full Jacobian preconditioning matrix. The warp layers are meta-learned in a trajectory-agnostic fashion, thus obviating the need to backpropagate through the gradient steps to compute the updates of their parameters. The framework is readily applicable to standard gradient-based meta-learners, and is shown to yield a significant boost in performance on both few-shot and multi-shot learning tasks, as well as to have promising applications to continual learning.\n\nThe paper is well-structured and well-motivated: the problem statement is clearly laid out from the outset, with appropriate context, and explanations supported well diagramatically. The idea, and perhaps more so the applications thereof, is seemingly novel and its explanation is given straightforwardly while avoiding getting bogged down in technical details. Clear comparisons and distinctions with previous work are drawn - for instance with the update rules for several gradient-based methods - MAML and its derivatives - being laid out in standard form (though it might also be nice to echo this with the WarpGrad update rule).\n\nThe experiments are logically ordered with the initial set covering the standard few-shot learning benchmarks with appropriate baselines (though the results for few-shot tieredImageNet are lacking in this respect), with most essential details given in the main text and full details, including those related to the datasets in question and hyperparameter selection, documented in Appendix H. Meta-learning does seem uniquely well-positioned for tackling the task of continual learning and it's heartening to see this being explored here with a degree of success - it would be interested to see how its performance compares with standard continual learning methods (such as EWC) on the same task. Particularly impressive is the depth into which the Appendices regarding the experiments, both elaborating on the details given in the main text as well as additional ablation studies.\n \nMinor errors:\n\n- Page 7: \"a neural network that dynamically **adapt** the parameters...\" - should be \"adapts\"\n- Page 22: \"where $I$ is the **identify** matrix\" - should be \"identity\" \n- Page 27: \"The task target function $g_\\tau$ is **partition** into 5 sets of **sub-task**\" - should be \"partitioned\" and \"sub-tasks\", respectively", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HkgHvrYptS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a learning strategy to precondition gradients for meta-learning. I really enjoyed reading the paper though I admit that I couldn't fully grasp all the details yet (paper is dense). My comments below are mostly to improve the readability of the paper for readers like me (knowing a thing or two in optimization and meta-learning)\n\n\n\n1- The authors emphasize on the method being trajectory-agnostic. Can you explain why this is very important? What methods are not trajectory-agnostic?\n\n2 - Also in various places, the authors claim the method does not suffer from vanishing/exploding gradients and credit-assignment problem. This needs to be properly verified (and explained as I do not see the connections clearly)\n\n3- Some claims are based on the Omniglot experiments (eg., the effect of the stop-gradient). It would be good if this can be done on Mini-imagenet instead.\n\n4- I am not sure I understand the stop-gradient operator, can you be more explicit there?\n\n5- I read the conversation regarding linear units on openreview and I disagree with your statement. A cascade of linear layers does not necessarily match one linear layer unless some constraints on the rank of layers are envisaged, a bottleneck in the middle ruin everything. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1eAh22icH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe current paper deals with meta-learning and essentially proposes a generalization of MAML (a popular gradient-based meta-learning algorithm) that mostly builds upon two main recent advances in meta-learning: 1) an architectural one (see e.g. T-Nets), which consists in optimizing the parameters of additional layers during the meta-learning outer loop (as opposed to only optimizing the initial conditions of the original parameters like in MAML), and 2) a theoretical one (see e.g. Meta-SGD, Meta-curvature), which is based on the geometrical observation that one set of parameters can precondition a second set of parameters that are consequently being optimized in a \"warped\" geometry, possibly speeding up learning.\nThe authors provide a great and thorough overview of the literature, in particular for gradient-based meta-learning methods, which helps putting all this in perspective.\nThe way they obtain the mentioned \"warped\" geometry in practice is by adding additional so-called warp-layers to an architecture that is being trained with meta-learning. Such warp-layer are generic deep learning modules (such as convolutions followed by BatchNorm, or LSTM layers), which are being trained in the outer-loop of the meta-learning optimization. In this sense, WarpGrad extend T-Nets, which only allowed for linear layers.\nThe second main innovation of WarpGrad is the proposal of a new meta-learning objective, which incorporates a meta-learning internal loop of only one step of (preconditioned) SGD, meaning that, as the authors notes, \"in contrast to MAML-based approaches (Eq. 1), [...] avoids backpropagation through learning processes\".\nThe authors test their algorithm on several meta-learning benchmarks, including few- and multi-shot learning tasks demonstrating very competitive performance when their algorithm is combined with MAML or Leap. They then deploy WarpGrad on a maze navigation reinforcemente learning task to demonstrate training of recurrent architectures, and on a continual learning toy dataset to show that their objective can be adapted to mitigate catastrophic forgetting. \n\nDecision:\nThis is a good paper which proposes an interesting generalization of previous gradient-based meta-learning methods like MAML and T-Net, with an impressive number of experiments. However, some of the statements regarding the advantages of WarpGrad over previous algorithms seem a little bit misleading, in particular in situations where WarpGrad needs to be combined with these same algorithms. For instance (and I might have completely misunderstood things here), it seems that when the WarpGrad objective is being combined with MAML (which requires backpropagation through multiple-step gradient descent trajectories), then also the resulting combined objective will necessarily need to backprop through the same multi-step trajectory, defeating the stated advantage of the WarpGrad algorithm (i.e. that its objective avoids backpropagating through the learning processes).\nIn general, even if one only considers the WarpGrad objective eq. (10), that comprises a meta-learning inner loop which consists of one step of (preconditioned) gradient descent. However, it seems like an arbitrary (and limiting) choice of the authors to only perform one step, as opposed to multiple ones. As a matter of fact, even very sophisticated second order gradient descent methods like natural gradient descent typically require more than one step to reach a local minimum. That is to say, that the main advantage showcased by the authors (the fact that the WarpGrad objective avoids backprop through a whole learning trajectory) seems like a limitation, rather than the result of a principled derivation.\nIt would be beneficial if the authors could clarify this points. In particular, whether combining WarpGrad with MAML does not indeed negate the stated advantages of WarpGrad over MAML, and whether there is a principled way of demonstrating that executing only one step in the inner loop of the WarpGrad objective is completely general (i.e., additional steps do not help the inner loop).\n\nMinor:\n- The authors use the wrong citation key when referring to the T-net paper: it should be Lee et al 2018, instead of Lee et al. 2017\n- I believe that when the authors mention Fast and slow weights, they are being described in the opposite way: slow weights should be in charge of meta-learning information, while fast ones are in charge of task-specific information.\n- Line 3 and 4 of Algorithm 1 and 2: shouldn't it say \"mini-batch of tasks\" (plural), instead of \"mini-batch of task\", since several tasks are being sampled? Otherwise, it might be erroneously interpreted as \"mini-batch of (samples belonging to) task T\".\n- The comment that \"learning to precondition gradients can be seen as a Markov Process of order 1\" is never clearly elucidated or developed. It would help to develop this.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: A strong paper reporting improved approaches to meta-learning.
Decision: Accept (Talk)