Generalized Inner Loop Meta-Learning
Authors: Edward Grefenstette
Brandon Amos
Denis Yarats
Phu Mon Htut
Artem Molchanov
Franziska Meier
Douwe Kiela
Kyunghyun Cho
Soumith Chintala
Venue:  (2020)
Keywords: meta-learning

Abstract: Many (but not all) approaches self-qualifying as "meta-learning" in deep learning and reinforcement learning fit a common pattern of approximating the solution to a nested optimization problem. In this paper, we give a formalization of this shared pattern, which we call GIMLI, prove its general requirements, and derive a general-purpose algorithm for implementing similar approaches. Based on this analysis and algorithm, we describe a library of our design, unnamedlib, which we share with the community to assist and enable future research into these kinds of meta-learning approaches. We end the paper by showcasing the practical applications of this framework and library through illustrative experiments and ablation studies which they facilitate.

Under review as a conference paper at ICLR 2020
GENERALIZED INNER LooP META-LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Many (but not all) approaches self-qualifying as "meta-learming in deep learning and reinforcement learning fit common pattern of approximating the solution to nested optimization problem. In this paper, we give a formalization of this shared pattern, which we call GIMLI, prove its general requirements, and derive a general-pe ceneral-purpose algorithm for implementing similar approaches. Based on this analysis and algorithm, describe a library of our design, unnamedl li,, which we share with the community to assist and enable future research into these kinds of meta-learning approaches. We end the paper by showeasing the practical appli- cations of this framework and library through illustrative experiments and ablation studies which they facilitate.
INTRODUCTION
Although it is by no means subfield of machine learning research (see e.g. Schmidhuber 1987 Bengioo 2000, Hochreiter et al.) 200T), there has recently been a surge interest in meta- learning (e.g. Maclaurin al., 2015 Andrychowicz al., 2016, Finn et al. 2017). This is due to the methods meta-learning provides, amongst other things, for producing models that perform well beyond the confines of single task, outside the constraints of static dataset, or simply with greater data efficiency or sample complexity. Due to the wealth of options in what could be con- sidered "meta-" learning problem, the term itself may have been used with some degree of underspecification. However, it turns out that many meta-learning approaches, in particular in the recent literature, follow the pattern of optimizing the "eeetapprameters" of the training process by nesting one or more inner loops in an outer training loop. Such nesting enables training model for several steps, evaluating it, calculating or approximating the gradients that evaluation with respect to the paatmpparameees,, and subsequently updating these meta vetaa-aaramttes.
This paper makes three contributions. First, we propose a formalization of this general process, which we call Generalized Inner Loop Meta-Learning (GIMLI), and show that it subsumes several recent approaches. The proposed formalism allows us to describe the meta-optimization process in general terms and analyse its requirements. Second, we derive general algorithm that supports the implementation of various kinds of meta-learning fitting within the GIMLI framework and its requirements. Third, based on this analysis and algorithm, brary that enables the straightforward implementation of any meta-learning approach that fits within the GIMLI framework in canonical PyTorch, such that existing codebases require minimal changes, supporting third party module impiementations and a variety of optimizers. Through a set of indica- tive experiments, we showcase the sort of research directions that are facilitated by our formalization and the corresponding library.
describe lightweight Py li-
The overarching aim of this paper is not-eaphatically--tt purport some notion of ownership or precedence in any sense over existing efforts by virtue of having proposed a unifying formulation. Rather, in pointing out similarities under such unification, provides theoretical and practical tools for facilitating further research in this exciting domain.
2 GENERALIZED INNER LOOP META-LEARNING
Whereby "meta-learnin" is taken to mean the process of "learning learn", describe it as nested optimization problem according to which an outer loop optimizes meta-variables controlling the optimization of model parameters within inner loop. The aim of the outer loop should be to

---

Under review as a conference paper at ICLR 2020
improve the meta-variables such that the inner loop produces models which are more suitable ac- cording to some criterion. In this section, we will formalize this process, which we call Generalized Inner Loop Meta- Meta-LLarning (GIMLI). We will define our terms in Section 2.1 give formal descrip- tion of the inner and outer loop's 's respective optimization problems in Section and 2.. followed by description and proof of the requirements under which the process can be in Section 2.4 Finally, in Section (2.5 we describe an algorithm which permits an efficient and exact implementation of GIMLI, agnostic to the model and optimizers used.
2.1 DEFINITIONS
Let us assume we wish to train a model parameterized by 0. Let ip describe some collection of possi- bly unrelated mheet-parameters" describing some aspect of the process by which we train our model. These could be, for example the learning rate or some other real-valued hyperparameters, task loss weights in multi-task learning, or perhaps the initialization of the model weights the beginning of training. For disambiguation, we will refer to the subset of şp describing optimization parameters by goop, and those meta- perampaaraeteess which parameterize aspects of the training loss calculation every step (e.g. the loss itself, the task mixture, regularization terms, etc) by gloss, i.c. P = gope U ploss. We will give several concrete examples in Section 3
Let Ctrain be a function of 0 and corresponding to the overall training objective we seek to minimize directly or indirectly (e.g. via an upper bound, a Monte Carlo estimate of an expectation, etc). Other elements of the objective such as the dataset, and hyperparameters or other aspects of training not covered by (p, are assumed to be implicit and held fixed in Ltrein for notational simplicity. With this, the process by which we train model to obtain test-time parameters 0* can be formalized as shown in Equation
argmin (0; girain (0,4)) 4p)
(1)
We assume that this search will be performed using an iterative gradient-based method such as some form of gradient descent as formalized in Section 2.2. and thus that the derivative V Cfrtrain (0, s) exists, as an obvious requirement.
Furthermore, let Lval be a function exclusively of 0 (except the specific case where 3o describes some aspect of the model, c.g. initialization, as is done in MAML (Finn et al. 2017)) describing some objective we wish measure the post-training performance of the model against. This could be a validation set from the same task the model was trained on, from collection of other tasks, or some other measure of the quality of the model parameterized by d". Typically, the aforementioned iterative process is not run to convergence, but rather to the point where an intermediate set of model parameters considered satisfactory according to some criterion (e.g. best model under Lval for a fixed budget of training steps, or after no improvement is seen for certain number of training steps, etc.), and these intermediate parameters will serve as l*
2.2 TRAINING
The process by which we typically approximate 6* against Ltrain decomposes into a sequence of updates of 9. At timestep t, we compute 0.+1 from 0r by first computing step-specific loss &t etrain (orain(e,, (loss) Note that this loss may itself also be function of step-specific factors, such as the training data used for that timestep, which we leave implicit here for notational simplic- ity. Following this, we typically compute or approximate (e.g. as in reinforcement learning) the gradients Vo, gtrain (0,, glose) of this loss with regard to O., by using algorithms such as backpropa- gation (Rumelhart et al. 1985). We then typically use an optimizer to "apply" these gradients to the parameters 9, to obtain updated parameters 0 As such, optimization processes such as e.g. Ada- grad (Duchi et al.) 2011) may be stateful, in that they exploit gradient history to produce adaptive "local" learning rates. Because some aspects of the optimization process may be covered by our
I 'Throughout this paper, we will write arg min_ in functional form as follows to facilitate formalizing nested optimization, with æ a variable and E' an expression (presumably a function of r):
( E) == arg min E
2

---

Under review as a conference paper at ICLR 2020
choice of ip, denote this optimization step at time-step t by opt, as shown in Equation 2 where snrestep-specffc attributes of the optimizer are left implicit by our use of the step subscript.
0++1 opt, (0z, o G+)
where
G. Vo, ptrain (0:, (lse)
(2)
For example, where we might use SGD as optimizer, with the learning-rate being meta-variable soope we could instantiate equation 2 as follows:
opt,(0, "0T G+) = Oe up Gt where G. = Ve, Vel
grrain (0z, ploss)
The estimation of 6" from Equation 1 Jusing T + 1 training updates according to Equation 2 yields a double recurrence in both 0 and G. it is function of 8t, outlined in Equation 3
opty (OT, T0th GT) = =pty nm Vor
eT Etrain (Pr; sploss)
where Pr opt (0 0 1)) = optr-1 11 mm Vor 1 ( lles))
where Pi opt (00, o0th Go) opt. oor Vo etrain (fo, gloos)
From this see that the optimization process used to train variety of model types, with a variety of optimization methods, can be described as a function yielding test-time model parameters 8" potentially function of parameter history .1...., Oy and of meta-parameters yp (if any exist). While this may seem like a fairly trivial formalization of ubiquitous training process, we will see in Section 2.4 that if few key requirements are met, this process can be nested the inner loop within an outer loop containing-amonn
other things-a meta-training process. From this process, described in Section 2.3 we estimate values of sp which improve our training process against some external metric.
2.3 META- CETAATRAINING
We now describe the outer loop optimization problem which wraps around the inner loop described in Section 2.2 Through this process, we seek a value &e which ensures that the training process Gtrain (f,, 40" produces parameters 0" which perform best against some metric L which we care about. The formalization and decomposition of this "meta-training process" into nested opti- mization problems is shown in Equation
argmin (f;; Cual(()))
argmin (4p; Coal (argmin (0; Ctrann (0,+))))
(4)
In this section, introduce formalization of iterative process allowing to approximate this nested optimization process. Furthermore, we describe a general iterative algorithm by which the process in Equation can be approximated by gradient-based methods while jointly estimating 0"
according to the process in Equation]] [
An iterative process by which we can estimate :p" given E the sort of decomposi- tion of the training process described Equation innto the training process described in Equations Z 3 is described below. Essentially, an estimate of 0" obtained following T steps training as outlined in Equation 3 which is then evaluated against L The gradient of this evaluation,
vgC (6*) then obtained through backpropagation, and used to update yp Using 7 as time-step
counter to symbolize the time-scale being different from that used in the "inner loop", we formalize this update in Equation 5 where metaopt, denotes the optimization process used update yr using M, at that time-step.
4-+1 metaopt, (yr, M,) where M, V Lual = metaopt, (f,, V eval (Ppt,((T,ypp) (GT))
(5)
where Pr optr-1 o
Vor- drain (PT-1, (oso))
:
where Pj opt, opt Go)
3

---

Under review as a conference paper at ICLR 2020
2.4 KEY REQUIREMENTS
For gradient-basee based meta-learning as formalized in Section 2.3 in particular through the process formalized in Equation 3 to be possible, few key requirements must be met. We enumerate them here, and then discuss the conditions under which they are met, either analytically or through choice of model, loss function, or optimizer.
L is a differentiable function of its input, i.e. the derivative V I--Luol (f*) exists.
II. The optimization process opt in Equation 2 is differentiable function of 0 and cP
III. Either or both of the following conditions hold:
(a) there exist continuous optimization hyperparameters (e.g. learning rate a) covered by uptt (e.g. a S (PPt) and opt in Equation 2 is differentiable function of your, or (b) the gradient Ge for one or more time-steps in Equations 3 is function of gloss (i.e. the derivative gloss gtrain (trainn (0, (ooos)) exists).
In the remainder of this section, we show that based on the assumptions outlined Section 2.1 namely the existence of gradients on the validation objective gool with regard model parameters 0 and of gradients on the training objective Lr with regard to both model parameters @ and meta- parameters sp, there exist gradients on the validation objective geal with regard to y. We will then, in the next section, demonstrate how to implement this process by specifying an update algorithm for meta-variables.
Implementing an iterative process as described in Equation 5 will exploit the chain rule for partial derivatives in order to run backpropagation. The structure of the recurrence means we need to ensure that Lual (0*) exists for E {O..... T}} in order to compute, for all such 6,, gradient paths (Vo, Loal (0*) (.). We can prove this exists in virtue of the chain rule for partial derivatives and the requirements above:
1. By the chain rule, Vo, Vo, Lual ( (0.) (Vor Loal (0.)) (Vo,0*) exists if V (0 ex- ists (and does, by Requirement D and Vo Vo, f11 exists.
2. Idem, Ve, Vo,0T+1- = (Ve..0r44)) 1) (Vo,0.4 exists by recursion over if for all i [1,7], Vo, Opm1 exists, which is precisely what Requirement guarantees.
Therefore Va, (8") exists for fO...., T), leaving us to demonstrate that V. is defined for relevant values of as a consequence of requirements I I|I We separately consider the case of gose and spopt as defined in Section 2.1
1. For V Voptt the gradients trivially exist as a consequence of Requirement II
2. For V VetonaDe, by the chain rule, V Vein O2 (Va,, 01) (V gloze G From Re- quirement M. it follows that VG.,, Or exists, and from Requirement Mil it follows that Vilors Gt-1 exists, therefore does V Vionn
Putting these both together, and having covered the union of sont and loss by exhaustion, we have shown that the gradients V Cn (8*) can be obtained by composition over the gradient paths (Ve, ((0aa(())) (V (.) for all t E [1, In Section 2.5 we show how to implement the exact and efficient calculation of V o Cval((
To complete this section, we indicate the conditions under in practice.
which requirements [1
Requirement M is simply a function of the choice of evaluation metric used to evaluate the model after training as part of Leal If this not a differentiable function of 8*, e.g. BLEU (Papineni et al. 2002 in machine translation, then proxy metric can be selected for meta-training (e.g. negative
For stateful optimizers. we assume that, where the state is itself a function of previous values of the network parameters (e.g. moving averages of parameter weights), it is included in the computation of gradients of the meta-loss with regard those parameters.
'This requirement is typically only satisfied by deterministic optimizers. However, for stochastic optimizers, if there exists reparameterization trick Kingma & Welling) 2013) for the distribution from which gradients other components of the update sampled, the requirement may also he satisfied.
4

---

Under review as a conference paper at ICLR 2020
sigllikelihodd of held out data), gradient estimation methods such REINFORCE (Williams 1992) can used.
Requirement [ is function of the choice of optimizer, but is satisfied for most popular optimizers. We directly prove that this requirement holds for SGD (Robbins & Monro I951) and for ADA- GRAD (Duchi et al. (011) in Appendix A and prove it by construction for wider class of common optimizers in the implementation and tests of the software library described in Section [
Requirement Ia is function of the choice of hyperparameters and optimizer, but is satisfied for least the learning rate in most popular optimizers. Requirement (Ilbls is function of the choice of loss function frrai (or class thereof), in that Vo, ftraix e (0ee)) needs to exist and be differentiable function of Usually. this requirement is held where is a multiplicative modifier of 0, For algorithmss such as Model Agnostic Meta-Learning (Finn al., 2017 MAML), this requirement is equivalent to saying that the Hessian of the loss function with regard to the parameters exists.
2.5 THE GIMLI UPDATE ALGORITHM
In Algorithm T we present the algorithm which permits the computation of updates to meta- variables through the nested iterative optimization process laid out above. To clearly disentangle different gradient paths, we employ the mathematical fiction that is the "stop-gradient operator, which is defined operator which maintains the truth of the following expression:
( f(x)
H : vf
(v=[((>)]
-
stop(z)
stop((z)
stop(z)
As will be shown below, this will allow us to decompose the computation of updates to the meta- variables through an arbitrarily complex training process, agnostic to the models and optimizers used (subject to the requirements of Section 2.4 being satisfied), into series of local updates pass- ing gradient with regard to the loss back through the inner loop steps. This is akin to backpropagation through time (BPTT; Rumelhart et al.) [9855. method which has been adapted other nested op- timization processes with various constraints or restrictions (Andrychowicz et al. 2016 Franceschi al. 2017 2018).
Algorithm The GIMLI update loop
Require: Current model parameters 6,, meta-parameters ffr
Require: Number of meta-parameter updates 7, length of unrolled inner loop J
1: for i 0 to / 1 do
Segment meta- parameters 4p looss +- split((pres)
3: Copy model state 03 +- Ot, optimizer state opti +- opt.
4: for = 0 to J 1 do
etrein (8), plort) and retain gradient graph state 6: Update inner model 1, + (pt 10 ,PP,, end for Initialize accumulators: A +- zerosTikee Alosa +- zerosLikeep!!00)
5:
Compute inner gradient G, + Vo; Fogttss
7:
8:
9:
Compute B, < Vor, gual (0,)
10:
for = J 1 to do Compute optimizer-gradient derivative O,, +- VG, opt p0th .G,,)
11:
12:
Update APP +- A By (V p opt [0 popl [G ))
stop(s
stop(e.Pt)
13:
Update Aloss +- Aloss + By-1 O2' (V pions
V strain
I0 (o00))
stop(yfunn)
14:
Compute By +- By441 ((Vo, opt opt (0/ opt [G,] )) Og (V (0,,
stop(e,r)
end for
16:
Update meta-parameters P74441 + metaopt((p... join ((500))
17:
end for
18:
return Updated meta- pentt-parameterss.oe 1
Each iteration through the loop defined by lines does one gradient- based update meta-
parameters y using the optimizer employed in line 16 Each such iteration, first (line copy
5

---

Under review as a conference paper at ICLR 2020
the model and optimizer state (generally updated through some outer loop within which this update
loop sits). We then (lines 7 compute a series of updates on a copy of the model, preserving the intermediate gradient computations Go...., GJ-1, intermediate model parameters 8/, ....... 8') (sometimes confusingly referred to as "fast weights", following Hinton & Plaut (1977). within the meta-learning literature), and all associated activations. These will be reused in the second
stage (lines (0H15) to backpropagate higher-order gradients of the meta-loss, computed on line P through the optimization process that was run in lines 4 In particular, in lines 12 and 13 local
(i.e. time step-specific) gradient calculations compute part of the gradient of V "p
and
Velora (8'),, which is stored in accumulators which contain the exact respective gradients by
the end of loop. What permits this efficient local computation is the dynamic programming calcula-
tion of the partial derivative By Ve, as function of only By-1 and timestep-specifte gradients, implementing a second-order variant of BPTT through reverse- mode differentiation.
3 EXAMPLES AND RELATED WORK
In this section, we highlight some instances of meta-learning which are instances of GIMLI, before discussing related approaches involving support for nested optimization, with applications to similar problems. The aim is not to provide a comprehensive literature review, which space would not permit. Rather, in pointing out similarity under our GIMLI formulation, we aim to showcase that rich and diverse research has been done using this class of approaches, where yet more progress indubitably remains to be made. This is, we believe, the strongest motivation for the development of libraries such as the one we present in Section support the implementation of algorithms that fall under the general algorithm derived in Section
3.1 EXAMPLES
Many of the papers referenced below contain excellent and thorough reviews of the literature most related to the type of meta-lcarning they approach. In the interest of brevity, we will not attempt such a review here, but rather focus on giving examples of few forms of meta-learning that fit the GIMLI framework (and thus are supported by the library presented in Section 4. and briefly explain why.
One popular meta-learning problem is that of learning to optimize hyperparameters through gradient-based methods (Bengio) 2000) Maclaurin et al.] 2015 Luketina et al. 2016 Franceschi al. 2017), as an alternative to grid/random search (Bergstra mization (Mockus et al. 1978 Pelikan et al.] 1999. |Bergstra et al., [011] Snoek et al., 2012). Here, select continuous-valle valued hyperparameters are meta-optimized against a meta-objective, subject to the differentiability of the optimization step, and, where relevant, the loss function. This corresponds to Requirements and IIIa of Section 2.4 being met, i.e. GIMLI being run with select optimizer hy- perparameters part of ooM To give simple concrete example, in the approaches of [Bengio 2000) and Maclaurin et al. (2015). the only meta-variable is the learning rate or i.e. P spopt a.
Bengio 2012) or Bayesian Opti-
M1
A related problem is that of learning the optimizer wholesale let al. 2001]] (Andrychowicz et al., 2016)] Duan et al.) 2016). typically based on recurrent architec- tures. Again, here the optimizer's own parameters are the optimizer hyperparameters, and constitute the entirety of gopt as used within GIMLI. Requirements/ [I and Ial are trivially met through the pre- condition that such optimizers models have parameters with regard to which their output (and losses that are a function thereof) is differentiable. As a concrete example, in the work of Andrychowicz al. (016), an RNN with parameters ő is meta-leamed, and models the updates made to parameters during training. In our formalism, this would correspond to setting as meta-variable the parameters of this update network, i.e. op Φ.
parametric model (Hochreiter
More recently. meta-learning approaches such MAML (Finn et al. 2017| Antoniou et al. 2018) and its variants/extensions have sought to use higher-order gradients to meta-learn model/policy initializations in few-shot learning settings. In GIMLI, this corresponds to setting 0g = ploss which then is not an explicit function of ptrain in Equation 3 but rather is implicitly its argument through updates to the inner model over the unrolled optimization. All requirements in Section 2.4 must be satisfied (save Ia with Requirement [Ilb further entailing that gtrain be defined such that the second derivative of the function with regard to 0 exists (i.e. non-zero).

---

Under review as a conference paper at ICLR 2020
Finally, recent work by Chebotar al. (019 has introduced the ML framework for learning unconstrained loss functions as parametric models, through exploiting second-order gradients of meta-loss with regard to the parameters of the inner loss. This corresponds, in GIMLI, to learning a parametric model of the loss parameterized by glose.
3.2 RELATED WORK
In a sense, many if not all of the approaches discussed in Section but here we will briefly discuss approaches to the general problem of formalizing and supporting implementations of problems that fit within the nested optimization specified by GIMI.I
3.1
quality as "related work",
The first is work by Franceschi et al. (2018) which describes how several meta-leamiig and hyper- parameter optimization approaches can be cast as bi-level optimization process, akin our own formalization in 2.3 This fascinating and relevant work is highly complementary to the formal- ization and discussion presented our paper. Whereas we focus on the requirements according to which gradient- based solutions to approaches based on nested optimization problems can be found in order to drive the development of a library which permits such approaches to be easily and scal- ably implemented, their work focuses analysis of the conditions under which exact gradient-based solutions to bi-level optimization processes can be approximated, and what convergence guarantees exist for such guarantees. In this sense, this is more relevant those who wish to analyze and extend alternatives to first-order approximations of algorithms such as MAML, e.g. see work of Nichol & Schulman 2018) or Rajeswaran et al. (2019).
On the software front, the library learn2 learn (Arold et al. 2019) addresses similar problems that which we will present in Section 4 This library focuses primarily on providing implemen- tations of existing meta-learning algorithms and their training loops that can be extended with new models. In contrast, the library we present in Section 4 is "closer to the metal", aiming sup- port the development of new meta-learning algorithms fitting the GIMLI definitions with as little resort to non-canonical PyTorch as possible. recent parallel effort, Torchmeta (Deleu et al. al. 2019) also provides a library aiming to assist the implementation of meta-learning algorithms, supply- ing useful data-loaders for meta-training. However, unlike our approach described in 4 it requires ce-implementation of models using their functional/stateless building blocks, and for users to reim- plement the optimizers in a differentiable manner.
4 THE unnamed1 ib LIBRARY
In this section, we provide a high-level description of the design and capabilities of unnamedlib PyTorch (Paszke et al., 2017) library aimed at enabling implementations of GIMLI with as little reliance on non-vanilla Py Torch as possible. In this section, we first discuss the obstacles that would prevent us from implementing this in popular deep learning frameworks, how overcame these in PyTorch to implement GIMLI. Additional features, helper functions, and other considerations when usinglextending the library are provided in its documentation.
4.1 OBSTACLES
Many deep learning frameworks offer the technical functionality required to implement GIMLI, namely the ability to take gradients of gradients. However, there are two aspects of how we imple- ment and train parametric models in such frameworks which inhibit our ability to flexibly implement Algorithm
The first obstacle is that models are typically implemented statefully (e.g. torch. nn in PyTorch, keras layers in Keras (Chollet et al.) 2015). etc.). meaning that the model's parameters are encapsulated in the model implementation. and are implicitly relied upon during the forward pass. Therefore while such models can be considered as functions theoretically, they are not pure functions practically, as their output is not uniquely determined by their explicit input, and equivalently the parameters for a particular forward pass typically cannot be trivially overridden or supplied at call
"We will publicly release the library described in this section in the coming weeks under the Apache li- cense. References to the library name have heen anonymized throughout the paper to preserve the double-blind reviewing process.

---

Under review as a conference paper at ICLR 2020
time. This prevents us from tracking and backpropagating over the successive values of the model parameters 8 within the inner loop described by Equation 3]
through an implicit or explicit graph.
The second issue is that, as discussed at the end of Section 2.4 and in Appendix A even though the operations used within popular optimizers are mathematically differentiable functions of the param- eters, gradients, and select hyperparameters, these operations are not tracked in various framework's implicit or explicit graph/gradient tape implementations when an optimization step is run. This is with good reason: updating model parameters in-place is memory efficient, as typically there is no need to keep references to the previous version of parameters. Ignoring the gradient dependency formed by allowing backpropagation through an optimization step essentially makes it safe re- lease memory allocated to historical parameters and intermediate model states once an update has been completed. Together, these obstacles essentially prevent us from practically satisfying Require- of Section 2.4
4.2 MAKING STATEFUL MODULES STATELESS
As we wish to track and backpropagate through intermediate states of parameters during the inner loop, we keep a record of such states which can be referenced during the backward pass stage of the outer loop in Algorithm [I] The typical way this is done in implementations of meta-learning algorithms such as MAML is to rewrite a "stateless" version of the inner loop's model, permitting the use, in each invocation of the model's forward pass, of weights which are otherwise tracked on the gradient graph/tape. While this addresses the issue, it is an onerous and limiting solution, as exploring new models within such algorithms invariably requires their reimplementation in a stateless style. This typically prevents the researcher from experimenting with third-party codebases, complicated models, or those which requiring loading pre-trained weights, without addressing a significant and unwelcome engineering challenge.
more generic solution, permitting the use of existing stateful modules (including with pre-loaded activations), agnostic to the complexity or origin of the code which defines them, is to modify the run-time instance of the model's parent class to render them effectively function, a technique of- ten referred to as "monkey mannkey-pptthing... The high-level function unnamedl ib. monke soannkypptett () does this by taking as argument a torch. nn. Module instance and the structure of its nested aub-modules. As it traverses this structure, it clones the parent classes of submodule instances, leav- ing their functionality intact save for that of the forward rd method which implements the forward pass. Here, it replaces the call to the forward method with one which first replaces the stateful pa- rameters of the submodule with ones provided as additional arguments to the patched forward, before calling the original class's bound forward method, which will now used the parameters provided at call time. This method is generic and derived from first frst-principles analysis of the terch. nn. Module e implementation, ensuring that any first or third-party implementation of para- metric models which are subclasses of torch. nn. Modul le and do not abuse the parent class at runtime will be supported by this recursive patching process.
4.3 MAKING OPTIMIZERS DIFFERENTIABLE
Again, as part of our need to make the optimization process differentiable in order to satisfy Re- quirement of Section the typical solution is write version of SGD which does not modify parameters in-place, but treated as differentiable function of the input parameters and hyperpa- rameters akin any other module in the training process. While this, again, is often considered a satisfactory solution in the meta-learning literature due to its simplicity, it too is limiting. Not only does the inability to experiment with other inner loop optimizers prevent research into the ap- plicability of meta-learning algorithms to other optimization processes, the restriction to SGD also means that existing state-of-the-at methods used in practical domains cannot be extended using meta-learning methods such as those described in Section 3 trained with SGD.
lest they perform competitively when
Here, while less generic, the solution provided by the high-level func- tion unnamedl ib. by mapping its parent class to differentiable reimplementation of the instance's parent class. The reimplementation is typically a copy of the optimizer's 's step logic, with il-place operations being replaced with gradient-tracking ones (a process which is syntactically simple execute in PyTorch).
() is to render a PyTorch optimizer instance differentiable
8

---

Under review as a conference paper at ICLR 2020
To this, add wrapper code which copies the optimizer's state, and allows safe branching off of it, to permit "unrolling" of the optimization process within an inner loop (ef. the recurrence from Equation B) without modifying the initial state of the optimizer (e.g. permit several such unrolls, or to preserve state if inner loop optimizer is used elsewhere in the outer loop). Most of the optimizers in torch - opt peptim
covered by this method. Here too, runtime modification of the parent optimizer class could possibly be employed as was done for torch. nn MMdules, but this would involve modifying Python objects far finer level of granularity. We find that supporting wide and varied class of optimizers is sufficient compromise to enable further research
5 EXPERIMENTS
In this section, we briefly present some results of experiments using select existing methods from Section[..]] show case how unnamedlib ib can be used to simply implement ablation studies and searches over model architectures, optimizers, and other aspects of an experiment. This could, natu- rally, be done without appeal to the library. However, in such cases, changing the model architecture or optimizer requires re-implementing the model functionally or optimization step differentiably. Here such changes require writing no new code (excluding the line where the model is defined).
5.1 META-LEARNING LEARNING RATES WITH HIGHER GRANULARITY
As the first application for unnamed b, we up simple meta-learning task where meta- optimize the learning rate. Employing a handcrafted annealing schedule for learning rates has been the de facto approach to improving learning algorithm. While scheduled annealing can help to boost the final performance, it usually requires significant hand-tuning of the schedule. In con- trast, we adjust learning rates automatically using meta-learning, which unnamedl ib enables for arbitary models and optimizers.
We take an image classification model DenseNet- (k=12) (Huang et al. 2016), that provides competitive state-of-the-an results on CIFAR10, 10, and modify its training procedure by replacing the multi-step anncaling schedule of learning rate with a meta-optimizer. Specifically, we treat learning rate for each inner optimizer's parameter group learn. Doing so allows individual model parameters to have finer learning rates, that are adjusted automatically. The GIMLI algorithm provides for clean implementation of such training procedure.
separate meta-parameter that we will meta-
Train accuracy
Test accuracy
Learning
105
100
0.15
100
95
0.10
90
90
85
85
80
80
0.00
0
100
200
300
100
200
300
100
200
300
epoch
epoch
opoch
Fixed
Multi-step annealing
Meta-leaming
Figure 1: Comparison of meta-learned learning rates against fixed and multi-step annealed for train- ing 1enseNet-BC((=11)) on CIFARIO. We observe convergence near state-of-the-art with better sample complexity that using a lann-designed annealing schedule.
We first train two baseline variants of DenseNet-BC (k=12) using setup from (Huang et al. 2016). In the first variant keep the learning rate fixed to 0.1 for all 300 epochs, while the second configuration takes advantage of a manually designed multi-step annealing schedule, which drops learning rate by 10 after 150 and 225 epochs. For the meta-learning variant, split the training set into two disjoint pieces, one for training and another for validation, in proportion of 99 : 1. We then use per parameter group learning rates (299 in total) for mnta-oppimization, initializing each learning rate to 0. We perform one meta-update step after each epoch, where we unroll inner optimizer for
5 We also provide documentation as to how to write differentiahle third party optimizers and supply helper functions such as unnamed1 ib. regi ster optim () register them for use with the library runtime.

---

Under review as a conference paper at ICLR 2020
Table 1: Results from ablating MAML++ architecture and inner optimizers. "Our base" results are from our VGG model trained with SGD in our MAML+ variant, and "our best" results show the best test accuracy found, with the best model/optimizer combination shown in the text below.
omniglot test accurncy
minilmage test necurney WWa
5 Way
20 Way
Appronch
1 Shot
5 Shot
Shot
5 Shot
Sbot
Shot
MAML++
(Antoniou et al. 2018
99.53 = 0.26%
99.93 = 0.09%
97.65 + 0.05%
99.33
0.03%
52.15 = 0.26%
68.32
0.44
MANIL (Our nice!
99.62
0.08%
99 0.02%
97.21
11%
99.13
0.13%
56.33
75.13
0.67%
MAML++ (Our besi)
99.91
0.05%
99.87
0.03%
99.00
0.33%
99.76 L 0.01%
$6.33 1 0,27%
76.73 0.52%
resnet-4FSDD
resner-1245CH
resner-S-SGD
vEE+SGD
resnet-8+SGDD
15 steps using batches from the training set, and compute meta-test error on the validation set over 10 batches from the validation set. We use batch size of 16 for meta-update, rather than 64 in the base training loop. We use Adam (Kingma & Ba 2014) with default parameters as a choice for meta-optimizer. We average results over 3 random seeds. Figure demonstrates that our method is able reach the state-of-the the--o-tthherrtt art performance faster.
5.2 ABLATING MAML'S MODEL ARCHITECTURE AND INNER OPTIMIZER
The unnamedl ib library enables the exploration of new MAML-like models and inner-loop opti- mizers, which historically has required non-trivial implementations of the fast-weights for the model parameters and inner optimizers as done in Antoniou et al. (2018); Deleu et al. (2019). These ab- lations can be important for squeezing the last few bits of accuracy on well-established tasks and baselines that are already near-optimal as shown in Chen et al. (2019), and is even more important for developing new approaches and tasks that deal with different kinds of data and require adaptation to be done over non-standard operations.
To illustrate how easy unnamedl ib makes these ablations, in this section we take a closer look at different model architecture and optimizer choices for the MAML++ approach (Antoniou et al.) 2018). MAML+ uses a VGG network with a SGD inner optimizer for the the Omniglot (Lake ett al. 2015) and Mini-lmagene (Vinyals et al., 2016 Ravi & Larochelle, 2016) tasks. We start with the official MAML++ code and evaluation procedure and use unname dlib to ablate across VGG, ResNet, and DenseNet models and SGD) and Adam optimizers. We provide more details about these in Appendix B We denote the combination of model and inner optimizer choice with <model>+<opt>> One finding of standalone interest is that we have kept most of the features from MAML++, except we significantly increase the base VGG+ SGD performance by using batch normalization in training mode everywhere in (Finn al. 2017) instead of using per-timestep statistics and parameters as MAML+ proposes. In theory, this enables more inner optimization steps to be rolled out at test time, which otherwise is not possible with MAML++ because of the per- timestep information, for simplicity in this paper we have not explored this and keep every algorithm, optimizer, and mode to five inner optimization steps. When using Adam as the inner optimizer, we initialize the first and second moment statistics to the statisties from the outer optimizer, which is also Adam, and learn per-parameter-gron results in Table [ show that are able to push the accuracy of MAML+ slightly up with this ablation. We note that this pushes the performance of MAML+ closer to that of sfate-of-the methods such as LEO (Rusu et al. (018)) Appendix B shows our full experimental results, and note that in some cases Adam slightly outperforms SGD for a particular model.
learning rates and rolling average 3 coefficients. Our
6 CONCLUSION
To summarize, we have presented GIMLI, a general formulation of a wide class of existing and potential meta-learning approaches, listed and proved the requirements that must be satisfied for such approaches to be possible, and specified a general algorithmic formulation of such approaches. We'v ve described a lightweight library, unnamedl ib, which extends PyTorch to enable the easy and natural implementation of such meta-learning approaches at scale. Finally we' ve demonstrated some of its potential applications. We hope to have made the case not only for the use of the mathematical and software tools we present here, but have also provided suitable encouragement for other researchers to use them and explore the boundaries of what can be done within this broad class of meta-learning approaches.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. In Advances in neural information processing systems, pp. 3981-3989, 2016.
Antreas Antoniou, Harrison Edwards, and Amos Storkey. How train your maml, arXiv preprint arXiv:18/0.09502. 2018.
Séb Amold, Praateek Mahajan, Debajyoti Datta, and Ian Bunner. learn2leann. gi t hub com http.//wwweeaaaaaaeeeaaaaaeeeee.
https://
2019.
Yoshua Bengio. Gradient-based optimization of hyperparameters. Neural computation. 12(8))1889-
1900, 2000.
James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13(Feb):281-305 2012.
James S Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper-parameter optimization. In Advances in neural information processing systems, pp. 2546-2554, 2011.
Yevgen Chebotar, Artem Molchanov, Sarah Bechtle, Ludovic Righetti, Franziska Meier, and Gaurav Sukhatme. Meta-learning via learned loss. arXiv preprint WxXv:/906.005374 2019.
Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification, 2019.
François Chollet et al. Keras, 2015.
Tristan Deleu, Tobias Würfl, Mandana Samiei, Joseph Paul Cohen, and Yoshua Bengio. Torch- meta: A Meta-Learning library for PyTorch, 2019. URLhttps: URt/itttps:///xxiiii
06576 Available at:
Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, llya Sutskever, and Pieter Abbeel. RI2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv: 1.02779, 2016.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 120uu):2121-2159, 2011.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1126-1135. JMLR. org, 2017.
Luca Franceschi, Michele Donini, Paolo Frasconi, and Massimiliano Pontil. Forward and reverse gradient-based hyperparameter optimization. In Proceedings of the 34th International Conference on Machine Learning Volume 70, pp. 1165-1173. JMLR. org, 2017.
Luca Franceschi, Paolo Frasconi, Saverio Salzo, Riccardo Grazzi, and Massimilano Pontil. Bilevel programming for hyperparameter optimization and meta-learning. arXiv preprint arXiv:1806.0m 040.000900. 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Geoffrey E Hinton and David C Plaut. Using fast weights to deblur old memories. In Proceedings of the ninth annual conference of the Cognitive Science Society, pp. 177-186, 1987.
Sepp Hochreiter, A Steven Younger, and Peter Conwell. Learning to learn using gradient descent. In International Conference on Artificial Neural Networks, pp. 87-94. Springer, 2001.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks. CoRR, 2016.
11

---

Under review as a conference paper at ICLR 2020
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.6114, 2013.
arXiv preprint
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 133066266:13322113333 2015.
Jelena Luketina. Mathias Berglund. Klaus Greff, and Tapani Raiko. Scalable gradient- baased tuning of continuous regularization hyperparameters. In International conference on machine learning, pp. 2002-2992699 2960, 2016.
Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient-basee hyperparameter optimiza- tion through reversible learning. In International Conference Machine Learning, pp. 2113- 2122, 2015.
Jonas Mockus, Vytautas Tiesis, and Antanas Zilinskas. The application of bayesian methods for seeking the extremum. Towards global optimization, 2(117-129):2, 1978.
Alex Nichol and John Schulman. Reptile: a scalable metalearning algorithm. arXiv preprint vrXiv:1803.02000 2, 2018.
Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. In Advances in Neural Information Processing Systems, pp. 721-731, 2018.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting association for computational linguistics, pp. 311-318. Association for Computational Linguistics, 2002.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop, 2017.
Martin Pelikan, David E Goldberg, and Erick Cantú-Paz. Boa: In Proceedings the 1st Annual Conference on Genetic and Evolutionary Computation- Volume 1, pp. 525-532 Morgan Kaufmann Publishers Inc., 1999.
The bayesian optimization algorithm.
Aravind Rajeswaran, Chelsea Finn, Sham Kakade, and Sergey Levine. Meta-learning with implicit gradients. arXiv preprint UrXiv:1909004633 2019.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. CoRR, 2016.
Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathemati- cal statisties, pp. 400-407, 1951.
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning internal representations by error propagation. Technical report, California Univ San Diego La Jolla Inst for Cognitive Science, 1985.
Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osin- dero, and Raia Hadsell. Meta-learning with latent embedding optimization. arXiv preprint arXiv:1807.05960. 2018.
Jürgen Schmidhuber.
Evolutionary principles in self-neferential learning, or learning how to learn: the meta-meta.... hook. PhD thesis, Technische Universität München, 1987.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
12

---

Under review as a conference paper at ICLR 2020
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. Advances in neural information processing systems. pp. 2951-2959, 2012.
Oriol Vinyals, Charles Blundell, Timothy Lillicrpp, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems, pp. 3630-3638, 2016.
Ronald J Williams. Simple statistical wradient-followigg algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-25) 256, 1992.
13

---

Under review as a conference paper at ICLR 2020
A
OPTIMIZER DIFFERENTIABILITY
A.1
SGD
00+1 = G, = v - aGi - - -
where a is the learning rate, and V v is defined iff C yp clse va=a
Vo,0 = (GD G.) = Vo, aGr = Vg,0,
av. 1 - V. et
prrain (0,, sp)
A.2 ADAGRAD
Vole+1= = Addagrad((,,GG) = Vol0,-
Grl ru Ei G.V.G.
VE G.
v VaGe V Cf
G.V
V Cf
E Gf
GiV 0
where 17 is the global learning rate, and V is defined iff 1 S 4p else
/ G/
Vo. 0.1 = Vo,Adggrad (c. G1) = Ve,
G.l
VEL C2
-o Vo Ge
Vo Gt
E G7
EE G?
=1 V Arain (0e, x)
G/ V3, grain (rraim(ee,4 sp)
-i=1 G?
E G?)
14

---

Under review as a conference paper at ICLR 2020
B MAML++ EXPERIMENTS: ADDITIONAL INFORMATION
Table shows all of the architectures and optimizers we ablated. The table missing some rows as we only report the results that successfully completed running three seeds within three days our cluster.
We the following model architectures, which closely resemble the vanilla PyTorch examples for these urchitectures but are modified to be smaller for the few-shot classification setting:
vgg is the VGG architecture (Simonyan & Zisserman 2014 variant used MAML+
(Antoniou et al.) 2018)
resnet-N is the ResNet architecture (He et al. 2016). resnet -4 corresponds to the
four blocks having just a single layer, and resnet and resnet-1 12 have respectively
2 and layers in each block
densenet- 8 is the DenseNet architecture (Huang et al. 2017) with 2 layers in each block
We follow TADAM (Oreshkin et al. 2018 and do not use the initial convolutional projection layer common in the full-size variants of the ResNet and DenseNet.
We can also visualize how the learning rates and rolling momentum terms (with Adam) change over time when they are being learned. We show this in Figure 2 and Figure for the 20-way 1-shot Omniglot experiment with the resnet-4 architecture, which is especially interesting as Adam outperforms SGD in this case. We find that most of the SGD learning rates decreased to near-zero except for a few select few that seem especially important for the adaptation. Adam exhibits similar behavior with the learning rates where most except for a select for are zeroed for the adaptation, and the rolling moment coefficients are particularly interesting where the first moment coefficient 31 becomes relatively evenly spread throughout the space while 82 splits many parameter groups between low and high regions of the space.
15

---

Under review as a conference paper at ICLR 2020
Table Full MAML++ model and inner optimizer sweep search results.
mean
std
dataset
nway
kshot
model
inner-optim
mini imagenet_fulllsin
densenet-8 8
SGD
46.08
1.40
resnet-12 12
SGD
51.06
1.51
resnet-4
Adam
49.71
3.71
SGD
54.36
0.23
resmet-8 vgg
SGD
54.16
1.35
Adam
47.93
11.64
SGD
56.33
0.27
5
densenet-8
SGD
65.29
0.98
resnet- 12
Adam
37.40
3.64
SGD
69.14
3.19
resnet-4
Adam
76.33
0.71
SGD
74.48
0.77
resnet-8
Adam
68.03
15.19
SGD
76.73
0.52
vgg
Adam
72.82
2.36
SGD
75.13
0.67
omniglot dataset
densenet-8
SGD
99.54
0.33
resnet-4
SGD
99.91
0.05
vgg
Adam
99.62
0.08
SGD
99.62
0.08
densenet-8
SGD
99.86
0.05
resnet-4
SGD
99.87
0.03
vgg
Adam
99.86
0.04
SGD
99.86
0.02
20
densenet-8
SGD
93.20
0.32
resnet- 12
SGD
99.00
0.33
resnet-4
Adam
98.31
0.09
SGD
96.31
0.15
resnet-8
SGD
98.50
0.15
vgg
Adam
96.15
0.16
SGD
97.21
0.11
5
densenet-8
SGD
97.24
0.26
resnet-12 12
SGD
99.69
0.17
resnet-4
Adam
99.44
0.23
SGD
99.71
0.03
resnet-8
SGD
99.76
0.01
vgg
Adam
98.74
0.04
SGD
99.13
0.13
16

---

Under review as a conference paper at ICLR 2020
Learning Rates
2.5
0.5
6.0
Epoch
Figure 2: The learning rates during training of VGG network with SGD the inner optimizer for 20-way I-shot mini-imagenet classification. The colors show the parameter groups within the model.
Learning Rates
1.50
1.00
0.75
20
40
60
80
100
120
140
Epoch
1.3
0.8
0.5
0.2
0.2
Epoch
Figure 3: The learning rates during training of a VGG network with Adam the inner opti- mizer for 20-way 1-shot mini-imagenet classification. The colors show the parameter groups within the model.
17


Reviews:
[{"review_id": "ryeVrc4XcS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe authors present a PyTorch based framework for performing second-order\nreverse mode autodiff for meta-learning.\n\nFirst, the authors present a formalization of a general prototypical\nmeta-learning setting.\nThey then provide an algorithm that solves this problem via gradient based\noptimization.\nFinally, perhaps the main contribution is a specific PyTorch implementation\nof said algorithm.\n\nThe type of meta-learning setting the authors consider is one where a gradient\nbased inner loop optimizer finds $\\theta^\\star$ by performing a finite number of steps.\nThe inner loop optimizer is parameterized through $\\varphi$ that consists of\ntwo parts $\\varphi^\\text{loss}$ and $\\varphi^\\text{opt}$.\nThe parameters $\\varphi^\\text{loss}$ are somehow part of the loss used for\ntraining in the inner loop. Example: Regularization paramter.\nThe parameters $\\varphi^\\text{opt}$ do not occur in the loss but in the\noptimizer step. Example: Learning rate.\n\nExample of an inner loop step:\n$\\theta^{k+1} := \\theta^k - \\alpha (\\nabla L(\\theta^k) + \\lambda \\nabla R(\\theta^k))$\nwhere $\\theta$ are the parameters of a neural network, $L$ is the training loss,\n$R$ is the regularizer, $\\alpha$ is the learning rate, $\\lambda$ is the regularization parameter.\nIn this example we would have $\\varphi = (\\alpha \\lambda)^T$.\n\nThe authos assume $\\theta^K$, the output of the inner loop after $K$ steps to\nbe differentiable wrt $\\varphi$.\nFurthermore, the meta-learning loss is assumed to be differentiable wrt $\\theta$\nso that a gradient of the meta-learning loss wrt to $\\varphi$ can be computed.\nThe authors also assume the meta-learning loss to be sufficient smooth in $\\varphi$\nsuch that a gradient based optimization can even be used for meta learning to\na local optimum.\n\nThe authors explicitly write down the reverse mode auto differentiation of\nthe inner loop and show how to, in that way, compute the gradient of the\nmeta-learning loss wrt to $\\varphi$.\n\nThe reverse (adjoint) mode auto differentiation of the above example inner loop step\nis the following step (iterated over in reverse down from $k = K$ to $k = 1$:\n$\\bar \\theta^k := (I - \\alpha (\\nabla^2 L(\\theta^k) + \\lambda \\nabla^2 R(\\theta^k)))^T \\bar \\theta^{k+1}$\n$\\bar \\alpha := \\bar \\alpha - (\\nabla l(\\theta^k) + \\lambda \\nabla R(\\theta^k))^T \\bar \\theta^{k+1}$\n$\\bar \\lambda := \\bar \\lambda - \\alpha \\nabla R(\\theta^k)^T \\bar \\theta^{k+1}$\nwhere $\\bar \\alpha$ accumulates the gradient of $\\theta$ wrt $\\alpha$ and\n$\\bar \\lambda$ accumulates the gradient of $\\theta$ wrt $\\lambda$.\n\nThe authors give some implementation details specific to some frameworks necessary\nfor implementing such \"gradient of an inner loop\".\n\nThe authors present experiments where they show how to meta-learn learning rates\nwith their framework.\nThey also how their framework can be used to quickly implement a MAML type\nmeta-learning optimizer ablation study comparing various combinations of\narchitecture, optimizer and inner loop steps etc..\n\nRecommendation:\nI propose to reject the paper.\nIn my eyes the only contribution is the implementation of a meta-learner in\nPyTorch based on well known methods.\nThe provided unifying formalization is theoretically inaccurate (see below)\nand to me come across as merely a motivation for their framework\n(but no value added compared to existing literature).\nThere is no new insight provided on the software engineering level either as\nfar as I can see.\n\nDetailed comments:\n- Page 1: ...provides tooling for analysing the provable requirements...\n\tseems like a complicated way of saying something that could be said simple\n\n- Page 2: Without loss of generality, ...\n\tYou are assuming a parametric model. Not sure what generality this phrase\n\trefers to.\n\n- Page 2: A formalization as $\\theta^\\star = argmin(\\theta, L(\\theta, \\varphi))$\n\tis inaccurate in the sense that it does not acknowledge the existing of\n\tmultiple optima.\n\tI would recommend not to use the $argmin$ operator here, since $argmin$ for\n\tsomething like a neural network would for example either return a global\n\toptimum (which no optimizer used in practice finds, and is not ment here)\n\tor would take on a set value with multiple local minima for example.\n\n\tIn the same context, the authors should mention the issues about uniqueness\n\tof optima (we are not even really finding optima when training neural networks),\n\timplicit functions / implicit differentiation\n\n\tIn the context of using stochastic optimizers one should also at least\n\tmention something about the differentiability of outputs of such optimizers\n\tand how they potentially depend on randomness of mini-batches\n\t(what if different randomness is used with the same or a perturbed\n\thyper parameter?)\n\n- Page 3: You mention the potential statefulness of the optimizer.\n\tWhy not explicitly carry it in the math notation?\n\tProbably things would get cluttered but saying it should be covered within\n\t$\\varphi$ does not seem reasonable to me.\n\n- Page 3: While this may seem like a fairly trivial formalization...\n\tYes, but also nesting this in an outer loop is fairly trivial in the sense\n\tthat it is a well known approach.\n\n- Page 4: there exist continuous hyperparam...\n\tIf they are not continuous then they should not even occur in this\n\tformalization so saying there exist... does not make much sense to me here\n\n\t$\\alpha \\subseteq \\varphi^\\text{opt}$ implies that $\\varphi^\\text{opt}$ is\n\ta set from notation although we are treating it as a vector everywhere else\n\n- Page 4: All of section 2.4 seems somewhat trivial to me, but I guess that is\n\thighly subjective.\n\n- Page 5: in the definition of stop operator perhaps use $:\\Leftrightarrow$\n\n- Page 5: Perhaps explicitly mention how your approach differs from a reverse\n\tmode differentiation of training or if it does not differ, say this.\n\n- Page 14: When talking about _S_GD (instead of just GD) perhaps mention\n\tsomething about non-existence of mini-batch randomness / being deterministic\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "r1lteyFy9B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work presented a general formulation of a wide class of existing meta-learning approaches, and proved the requirements that must be satisfied for such approaches to be possible.\n\nHalf of the work is focused on describing the unnamedlib library, which extends PyTorch to enable the easy\nand natural implementation of such meta-learning approaches.\n\nThe early sections are interesting, especially section 2, which gives some great insights to the existing inner loop pattern in meta-learning. However, from section 3, the paper has turned to examples and related works, where I was hoping the author would give more detailed analysis of the pattern. My concern is the authors have spent too much space on the unnamedlib library. So http://www.jmlr.org/mloss/ might be a more suitable place for publication.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "BJgg7RijtS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose the general formulation of recent meta-learning methods and propose a good library to use.\n\nPros:\n1. The general formulation of recent meta-learning methods is reasonable.\n2. The proposed library is easy to use.\n\nCons:\n\nThe paper lacks technical novelty. I understand the goal of this paper is to build a library. However, the paper only describes a general formulation for recent meta-learning methods (e.g., MAML) and implement the formulation. It is better to clarify and some key engineering challenges and do the corresponding experiments.\n\nIn addition, in the experiment parts, the authors only compare the results with MAML++. It will be more convincing if the authors can analyze other popular meta-learning methods (e.g.. Prototypical network [1], meta-LSTM [2]). \n\nAnother suggestion is that the authors can give some examples to connect current meta-learning models with the proposed general formulation. For example, the meaning of \\phi_i^opt, \\phi_i^loss in MAML, Prototype, Reptile, etc.\n\nIt is better to explain the meaning of different colors in Figure 3.\n\n[1] Snell, Jake, Kevin Swersky, and Richard Zemel. \"Prototypical networks for few-shot learning.\" Advances in Neural Information Processing Systems. 2017.\n[2] Ravi, Sachin, and Hugo Larochelle. \"Optimization as a model for few-shot learning.\" ICLR (2016).\n\n\n\nDecision after rebuttal: I have read the authors' responses. Like review 2, I also think the \"generalization\" is overclaimed, it only provides a general formulation. Thus, I finally decide to keep my score.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers agree that the technical innovations presented in this paper are not great enough to justify acceptance.  The authors correctly point out to the reviewers that the ICLR CFP states that the topics of "implementation issues, parallelization, software platforms, hardware” are acceptable.  I would point out that most papers in these spaces describe *technical innovations* that enable improvements in "parallelization, software platforms, hardware" rather than implementations of these improvements.   However, it is certainly true that a software package is an acceptable (although less common) basis for a publication, provided is it sufficiently unique and impactful.  After pointing this out to the reviewers and collecting opinions, the reviewers do not feel the combined technical and software contributions of this paper are enough to justify acceptance. 
 
Decision: Reject