Title: GENERALIZED INNER LOOP META-LEARNING

Abstract: Many (but not all) approaches self-qualifying as "meta-learning" in deep learning and reinforcement learning fit a common pattern of approximating the solution to a nested optimization problem. In this paper, we give a formalization of this shared pattern, which we call GIMLI, prove its general requirements, and derive a general-purpose algorithm for implementing similar approaches. Based on this analysis and algorithm, we describe a library of our design, unnamedlib, which we share with the community to assist and enable future research into these kinds of meta-learning approaches. We end the paper by showcasing the practical applications of this framework and library through illustrative experiments and ablation studies which they facilitate.

1. INTRODUCTION:

Although it is by no means a new subfield of machine learning research (see e.g. Schmidhuber, 1987; Bengio, 2000; Hochreiter et al., 2001) , there has recently been a surge of interest in metalearning (e.g. Maclaurin et al., 2015; Andrychowicz et al., 2016; Finn et al., 2017) . This is due to the methods meta-learning provides, amongst other things, for producing models that perform well beyond the confines of a single task, outside the constraints of a static dataset, or simply with greater data efficiency or sample complexity. Due to the wealth of options in what could be considered "meta-" to a learning problem, the term itself may have been used with some degree of underspecification. However, it turns out that many meta-learning approaches, in particular in the recent literature, follow the pattern of optimizing the "meta-parameters" of the training process by nesting one or more inner loops in an outer training loop. Such nesting enables training a model for several steps, evaluating it, calculating or approximating the gradients of that evaluation with respect to the meta-parameters, and subsequently updating these meta-parameters. This paper makes three contributions. First, we propose a formalization of this general process, which we call Generalized Inner Loop Meta-Learning (GIMLI), and show that it subsumes several recent approaches. The proposed formalism allows us to describe the meta-optimization process in general terms and analyse its requirements. Second, we derive a general algorithm that supports the implementation of various kinds of meta-learning fitting within the GIMLI framework and its requirements. Third, based on this analysis and algorithm, we describe a lightweight PyTorch library that enables the straightforward implementation of any meta-learning approach that fits within the GIMLI framework in canonical PyTorch, such that existing codebases require minimal changes, supporting third party module implementations and a variety of optimizers. Through a set of indicative experiments, we showcase the sort of research directions that are facilitated by our formalization and the corresponding library.

The overarching aim of this paper is not-emphatically-to purport some notion of ownership or precedence in any sense over existing efforts by virtue of having proposed a unifying formulation. Rather, in pointing out similarities under such unification, it provides theoretical and practical tools for facilitating further research in this exciting domain.

2. GENERALIZED INNER LOOP META-LEARNING:

Whereby "meta-learning" is taken to mean the process of "learning to learn", we can describe it as a nested optimization problem according to which an outer loop optimizes meta-variables controlling the optimization of model parameters within an inner loop. The aim of the outer loop should be to improve the meta-variables such that the inner loop produces models which are more suitable according to some criterion. In this section, we will formalize this process, which we call Generalized Inner Loop Meta-Learning (GIMLI). We will define our terms in Section 2.1, give a formal description of the inner and outer loop's respective optimization problems in Section 2.2 and 2.3, followed by a description and proof of the requirements under which the process can be run in Section 2.4. Finally, in Section 2.5 we describe an algorithm which permits an efficient and exact implementation of GIMLI, agnostic to the model and optimizers used.

2.1. DEFINITIONS:

Let us assume we wish to train a model parameterized by θ. Let ϕ describe some collection of possibly unrelated "meta-parameters" describing some aspect of the process by which we train our model. These could be, for example the learning rate or some other real-valued hyperparameters, task loss weights in multi-task learning, or perhaps the initialization of the model weights at the beginning of training. For disambiguation, we will refer to the subset of ϕ describing optimization parameters by ϕ opt , and those meta-parameters which parameterize aspects of the training loss calculation at every step (e.g. the loss itself, the task mixture, regularization terms, etc) by ϕ loss , i.e. ϕ = ϕ opt ∪ ϕ loss . We will give several concrete examples in Section 3.

Let L train be a function of θ and ϕ corresponding to the overall training objective we seek to minimize directly or indirectly (e.g. via an upper bound, a Monte Carlo estimate of an expectation, etc). Other elements of the objective such as the dataset, and hyperparameters or other aspects of training not covered by ϕ, are assumed to be implicit and held fixed in L train for notational simplicity. With this, the process by which we train a model to obtain test-time parameters θ * can be formalized as shown in Equation 11 .

EQUATION

We assume that this search will be performed using an iterative gradient-based method such as some form of gradient descent as formalized in Section 2.2, and thus that the derivative ∇ θ L train (θ, ϕ) exists, as an obvious requirement.

Furthermore, let L val be a function exclusively of θ (except the specific case where ϕ describes some aspect of the model, e.g. initialization, as is done in MAML (Finn et al., 2017) ) describing some objective we wish to measure the post-training performance of the model against. This could be a validation set from the same task the model was trained on, from a collection of other tasks, or some other measure of the quality of the model parameterized by θ * . Typically, the aforementioned iterative process is not run to convergence, but rather to the point where an intermediate set of model parameters is considered satisfactory according to some criterion (e.g. best model under L val for a fixed budget of training steps, or after no improvement is seen for a certain number of training steps, etc.), and these intermediate parameters will serve as θ * .

2.2. TRAINING:

The process by which we typically approximate θ * against L train decomposes into a sequence of updates of θ. At timestep t, we compute θ t+1 from θ t by first computing a step-specific loss train t (θ t , ϕ loss ). Note that this loss may itself also be a function of step-specific factors, such as the training data used for that timestep, which we leave implicit here for notational simplicity. Following this, we typically compute or approximate (e.g. as in reinforcement learning) the gradients ∇ θt train t (θ t , ϕ loss ) of this loss with regard to θ t , by using algorithms such as backpropagation (Rumelhart et al., 1985) . We then typically use an optimizer to "apply" these gradients to the parameters θ t to obtain updated parameters θ t+1 . As such, optimization processes such as e.g. Adagrad (Duchi et al., 2011 ) may be stateful, in that they exploit gradient history to produce adaptive "local" learning rates. Because some aspects of the optimization process may be covered by our choice of ϕ, we denote this optimization step at time-step t by opt t as shown in Equation 2, where timestep-specific attributes of the optimizer are left implicit by our use of the step subscript.

θ t+1 = opt t (θ t , ϕ opt , G t ) where G t = ∇ θt train t (θ t , ϕ loss ) (2)

For example, where we might use SGD as an optimizer, with the learning-rate being a meta-variable ϕ opt , we could instantiate equation 2 as follows:

opt t (θ t , ϕ opt , G t ) := θ t -ϕ opt • G t where G t = ∇ θt train t (θ t , ϕ loss )

The estimation of θ * from Equation 1 using T + 1 training updates according to Equation 2 yields a double recurrence in both θ t and G t (as it is a function of θ t , as outlined in Equation 3).

θ * ≈ opt T (θ T , ϕ opt , G T ) = opt T (P T , ϕ opt , ∇ θ T train T (P T , ϕ loss ))

where

EQUATION

From this we see that the optimization process used to train a variety of model types, with a variety of optimization methods, can be described as a function yielding test-time model parameters θ * potentially as a function of parameter history θ 1 , . . . , θ T and of meta-parameters ϕ (if any exist). While this may seem like a fairly trivial formalization of a ubiquitous training process, we will see in Section 2.4 that if a few key requirements are met, this process can be nested as the inner loop within an outer loop containing-amongst other things-a meta-training process. From this process, described in Section 2.3, we estimate values of ϕ which improve our training process against some external metric.

2.3. META-TRAINING:

We now describe the outer loop optimization problem which wraps around the inner loop described in Section 2.2. Through this process, we seek a value ϕ * which ensures that the training process L train (θ, ϕ * ) produces parameters θ * which perform best against some metric L val (θ * ) which we care about. The formalization and decomposition of this "meta-training process" into nested optimization problems is shown in Equation 4.

EQUATION

In this section, we introduce a formalization of an iterative process allowing us to approximate this nested optimization process. Furthermore, we describe a general iterative algorithm by which the process in Equation 4 can be approximated by gradient-based methods while jointly estimating θ * according to the process in Equation 1.

An iterative process by which we can estimate ϕ * given Equation 4 following the sort of decomposition of the training process described Equation 1 into the training process described in Equations 2-3 is described below. Essentially, an estimate of θ * is obtained following T + 1 steps training as outlined in Equation 3, which is then evaluated against L val . The gradient of this evaluation, ∇ ϕ L val (θ * ) is then obtained through backpropagation, and used to update ϕ. Using τ as time-step counter to symbolize the time-scale being different from that used in the "inner loop", we formalize this update in Equation 5 where metaopt τ denotes the optimization process used to update ϕ τ using M τ at that time-step.

ϕ τ +1 = metaopt τ (ϕ τ , M τ ) where M τ = ∇ ϕτ L val (θ * ) (5) = metaopt τ (ϕ τ , ∇ ϕτ L val (opt T (P T , ϕ opt , G T )))

where

P T = opt T -1 (P T -1 , ϕ opt , ∇ θ T -1 train T -1 (P T -1 , ϕ loss )) . . .

where P 1 = opt 0 (θ 0 , ϕ opt , G 0 )

2.4. KEY REQUIREMENTS:

For gradient-based meta-learning as formalized in Section 2.3-in particular through the process formalized in Equation 5-to be possible, a few key requirements must be met. We enumerate them here, and then discuss the conditions under which they are met, either analytically or through choice of model, loss function, or optimizer. I. L val is a differentiable function of its input, i.e. the derivative ∇ θ * L val (θ * ) exists.

II. The optimization process opt in Equation 2 is a differentiable2 function of θ and G3 . III. Either or both of the following conditions hold:

(a) there exist continuous optimization hyperparameters (e.g. a learning rate α) covered by ϕ opt (e.g. α ⊆ ϕ opt ) and opt in Equation 2 is a differentiable function of ϕ opt , or (b) the gradient G t for one or more time-steps in Equations 1-3 is a function of ϕ loss (i.e. the derivative ∇ ϕ loss train (θ, ϕ loss ) exists).

In the remainder of this section, we show that based on the assumptions outlined in Section 2.1, namely the existence of gradients on the validation objective L val with regard to model parameters θ and of gradients on the training objective L train with regard to both model parameters θ and metaparameters ϕ, there exist gradients on the validation objective L val with regard to ϕ. We will then, in the next section, demonstrate how to implement this process by specifying an update algorithm for meta-variables.

Implementing an iterative process as described in Equation 5will exploit the chain rule for partial derivatives in order to run backpropagation. The structure of the recurrence means we need to ensure that ∇ θt L val (θ * ) exists for t ∈ {0, . . . , T } in order to compute, for all such θ t , gradient paths

(∇ θt L val (θ * )) • (∇ ϕ θ t ).

We can prove this exists in virtue of the chain rule for partial derivatives and the requirements above:

1. By the chain rule,

∇ θt L val (θ * ) = (∇ θ * L val (θ * )) • (∇ θt θ * ) exists if ∇ θ * L val (θ * ) ex-

ists (and it does, by Requirement I) and

∇ θt θ * = ∇ θt θ T +1 exists. 2. Idem, ∇ θt θ T +1 = (∇ θt+1 θ T +1 ) • (∇ θt θ t+1 ) exists by recursion over t if for all i ∈ [1, T ],

∇ θi θ i+1 exists, which is precisely what Requirement II guarantees.

Therefore ∇ θt L val (θ * ) exists for t ∈ {0, . . . , T }, leaving us to demonstrate that ∇ ϕ θ t is defined for all relevant values of t as a consequence of requirements I-III. We separately consider the case of ϕ loss and ϕ opt as defined in Section 2.1:

1. For ∇ ϕ opt θ t , the gradients trivially exist as a consequence of Requirement IIIa.

2. For ∇ ϕ loss θ t , by the chain rule,

∇ ϕ loss θ t = (∇ Gt-1 θ t ) • (∇ ϕ loss G t-1

). From Requirement II, it follows that ∇ Gt-1 θ t exists, and from Requirement IIIb, it follows that ∇ ϕ loss G t-1 exists, therefore so does ∇ ϕ loss θ t .

Putting these both together, and having covered the union of ϕ opt and ϕ loss by exhaustion, we have shown that the gradients ∇ ϕ L val (θ * ) can be obtained by composition over the gradient paths

(∇ θt L val (θ * ))(∇ ϕ θ t ) for all t ∈ [1, T ].

In Section 2.5 we show how to implement the exact and efficient calculation of ∇ ϕ L val (θ * ). To complete this section, we indicate the conditions under which requirements I-III hold in practice.

Requirement I is simply a function of the choice of evaluation metric used to evaluate the model after training as part of L val . If this is not a differentiable function of θ * , e.g. BLEU (Papineni et al., 2002) in machine translation, then a proxy metric can be selected for meta-training (e.g. negative log-likelihood of held out data), or gradient estimation methods such as REINFORCE (Williams, 1992) can be used.

Requirement II is a function of the choice of optimizer, but is satisfied for most popular optimizers. We directly prove that this requirement holds for SGD (Robbins & Monro, 1951) and for ADA-GRAD (Duchi et al., 2011) in Appendix A, and prove it by construction for a wider class of common optimizers in the implementation and tests of the software library described in Section 4.

Requirement IIIa is a function of the choice of hyperparameters and optimizer, but is satisfied for at least the learning rate in most popular optimizers. Requirement IIIb is a function of the choice of loss function train (or class thereof), in that ∇ θt train t (θ t , ϕ loss ) needs to exist and be a differentiable function of ϕ. Usually, this requirement is held where ϕ is a multiplicative modifier of θ t . For algorithms such as Model Agnostic Meta-Learning (Finn et al., 2017, MAML) , this requirement is equivalent to saying that the Hessian of the loss function with regard to the parameters exists.

2.5. THE GIMLI UPDATE ALGORITHM:

In Algorithm 1, we present the algorithm which permits the computation of updates to metavariables through the nested iterative optimization process laid out above. To clearly disentangle different gradient paths, we employ the mathematical fiction that is the "stop-gradient" operator, which is defined as an operator which maintains the truth of the following expression:

- stop(x) : ∀f f (x) stop(x) = f (x) ∧ ∇ x f (x) stop(x)

= 0

As will be shown below, this will allow us to decompose the computation of updates to the metavariables through an arbitrarily complex training process, agnostic to the models and optimizers used (subject to the requirements of Section 2.4 being satisfied), into a series of local updates passing gradient with regard to the loss back through the inner loop steps. This is akin to backpropagation through time (BPTT; Rumelhart et al., 1985) , a method which has been adapted to other nested optimization processes with various constraints or restrictions (Andrychowicz et al., 2016; Franceschi et al., 2017; 2018) .

Algorithm 1 The GIMLI update loop:

Require: Current model parameters θt, meta-parameters ϕτ Require: Number of meta-parameter updates I, length of unrolled inner loop J 1:

for i = 0 to I -1 do 2: Segment meta-parameters ϕ opt i , ϕ loss i ← split(ϕτ+i) 3:

Copy model state θ 0 ← θt, optimizer state opt 0 ← opt t 4:

for j = 0 to J -1 do 5:

Compute inner gradient Gj ← ∇ θ j train t+j (θ j , ϕ loss i

) and retain gradient graph state 6: Update inner model θ j+1 ← opt j (θ j , ϕ opt i , Gj) 7:

end for 8:

Initialize accumulators:

A opt i ← zerosLike(ϕ opt i ); A loss i ← zerosLike(ϕ loss i ) 9: Compute BJ ← ∇ θ J L val (θ J ) 10: for j = J -1 to 0 do 11: Compute optimizer-gradient derivative O j ← ∇G j opt j (θ j , ϕ opt i , G j ) 12: Update A opt i ← A opt i + B j +1 • (∇ ϕ opt i opt j ( θ j stop(ϕ opt i ) , ϕ opt i , G j stop(ϕ opt i

)

))

13: Update A loss i ← A loss i + B j +1 • O j • (∇ ϕ loss i ∇ θ j train t+j ( θ j stop(ϕ loss i ) , ϕ loss i )) 14: Compute B j ← B j +1 • ((∇ θ j opt j (θ j , ϕ opt i , G j stop(θ j ) )) + O j • (∇ 2 θ j train t+j (θ j , ϕ loss i )))

15: end for 16:

Update meta-parameters ϕτ+i+1 ← metaopt(ϕτ+i, join(A opt i , A loss i

)) 17: end for 18: return Updated meta-parameters ϕτ+I Each iteration through the loop defined by lines 1-17 does one gradient-based update of metaparameters ϕ using the optimizer employed in line 16. Each such iteration, we first (line 3) copy the model and optimizer state (generally updated through some outer loop within which this update loop sits). We then (lines 4-7) compute a series of J updates on a copy of the model, preserving the intermediate gradient computations G 0 , . . . , G J-1 , intermediate model parameters θ 0 , . . . , θ J (sometimes confusingly referred to as "fast weights", following Hinton & Plaut (1987) , within the meta-learning literature), and all associated activations. These will be reused in the second stage (lines 10-15) to backpropagate higher-order gradients of the meta-loss, computed on line 9 through the optimization process that was run in lines 4-7. In particular, in lines 12 and 13, local (i.e. time step-specific) gradient calculations compute part of the gradient of ∇ ϕ opt i L val (θ J ) and ∇ ϕ loss i L val (θ J ), which is stored in accumulators which contain the exact respective gradients by the end of loop. What permits this efficient local computation is the dynamic programming calculation of the partial derivative B j = ∇ θ j as function of only B j +1 and timestep-specific gradients, implementing a second-order variant of BPTT through reverse-mode differentiation.

3. EXAMPLES AND RELATED WORK:

In this section, we highlight some instances of meta-learning which are instances of GIMLI, before discussing related approaches involving support for nested optimization, with applications to similar problems. The aim is not to provide a comprehensive literature review, which space would not permit. Rather, in pointing out similarity under our GIMLI formulation, we aim to showcase that rich and diverse research has been done using this class of approaches, where yet more progress indubitably remains to be made. This is, we believe, the strongest motivation for the development of libraries such as the one we present in Section 4 to support the implementation of algorithms that fall under the general algorithm derived in Section 2.

3.1. EXAMPLES:

Many of the papers referenced below contain excellent and thorough reviews of the literature most related to the type of meta-learning they approach. In the interest of brevity, we will not attempt such a review here, but rather focus on giving examples of a few forms of meta-learning that fit the GIMLI framework (and thus are supported by the library presented in Section 4), and briefly explain why.

One popular meta-learning problem is that of learning to optimize hyperparameters through gradient-based methods (Bengio, 2000; Maclaurin et al., 2015; Luketina et al., 2016; Franceschi et al., 2017) , as an alternative to grid/random search (Bergstra & Bengio, 2012) or Bayesian Optimization (Mockus et al., 1978; Pelikan et al., 1999; Bergstra et al., 2011; Snoek et al., 2012) . Here, select continuous-valued hyperparameters are meta-optimized against a meta-objective, subject to the differentiability of the optimization step, and, where relevant, the loss function. This corresponds to Requirements II and IIIa of Section 2.4 being met, i.e. GIMLI being run with select optimizer hyperparameters as part of ϕ opt . To give a simple concrete example, in the approaches of Bengio (2000) and Maclaurin et al. (2015) , the only meta-variable is the learning rate α, i.e. ϕ = ϕ opt = α.

A related problem is that of learning the optimizer wholesale as a parametric model (Hochreiter et al., 2001; Andrychowicz et al., 2016; Duan et al., 2016) , typically based on recurrent architectures. Again, here the optimizer's own parameters are the optimizer hyperparameters, and constitute the entirety of ϕ opt as used within GIMLI. Requirements II and IIIa are trivially met through the precondition that such optimizers models have parameters with regard to which their output (and losses that are a function thereof) is differentiable. As a concrete example, in the work of Andrychowicz et al. (2016) , an RNN with parameters φ is meta-learned, and models the updates made to parameters during training. In our formalism, this would correspond to setting as meta-variable the parameters of this update network, i.e. ϕ opt = φ.

More recently, meta-learning approaches such as MAML (Finn et al., 2017; Antoniou et al., 2018) and its variants/extensions have sought to use higher-order gradients to meta-learn model/policy initializations in few-shot learning settings. In GIMLI, this corresponds to setting θ 0 = ϕ loss , which then is not an explicit function of train in Equation 3, but rather is implicitly its argument through updates to the inner model over the unrolled optimization. All requirements in Section 2.4 must be satisfied (save IIIa, with Requirement IIIb further entailing that train be defined such that the second derivative of the function with regard to θ exists (i.e. is non-zero).

Finally, recent work by Chebotar et al. (2019) has introduced the ML 3 framework for learning unconstrained loss functions as parametric models, through exploiting second-order gradients of a meta-loss with regard to the parameters of the inner loss. This corresponds, in GIMLI, to learning a parametric model of the loss parameterized by ϕ loss .

3.2. RELATED WORK:

In a sense, many if not all of the approaches discussed in Section 3.1 qualify as "related work", but here we will briefly discuss approaches to the general problem of formalizing and supporting implementations of problems that fit within the nested optimization specified by GIMLI.

The first is work by Franceschi et al. (2018) which describes how several meta-learning and hyperparameter optimization approaches can be cast as a bi-level optimization process, akin to our own formalization in 2.3. This fascinating and relevant work is highly complementary to the formalization and discussion presented in our paper. Whereas we focus on the requirements according to which gradient-based solutions to approaches based on nested optimization problems can be found in order to drive the development of a library which permits such approaches to be easily and scalably implemented, their work focuses on analysis of the conditions under which exact gradient-based solutions to bi-level optimization processes can be approximated, and what convergence guarantees exist for such guarantees. In this sense, this is more relevant to those who wish to analyze and extend alternatives to first-order approximations of algorithms such as MAML, e.g. see work of Nichol & Schulman (2018) or Rajeswaran et al. (2019) .

On the software front, the library learn2learn (Arnold et al., 2019) addresses similar problems to that which we will present in Section 4. This library focuses primarily on providing implementations of existing meta-learning algorithms and their training loops that can be extended with new models. In contrast, the library we present in Section 4 is "closer to the metal", aiming to support the development of new meta-learning algorithms fitting the GIMLI definitions with as little resort to non-canonical PyTorch as possible. A recent parallel effort, Torchmeta (Deleu et al., 2019 ) also provides a library aiming to assist the implementation of meta-learning algorithms, supplying useful data-loaders for meta-training. However, unlike our approach described in 4, it requires re-implementation of models using their functional/stateless building blocks, and for users to reimplement the optimizers in a differentiable manner.

4. THE unnamedlib LIBRARY:

In this section, we provide a high-level description of the design and capabilities of unnamedlib,4 a PyTorch (Paszke et al., 2017 ) library aimed at enabling implementations of GIMLI with as little reliance on non-vanilla PyTorch as possible. In this section, we first discuss the obstacles that would prevent us from implementing this in popular deep learning frameworks, how we overcame these in PyTorch to implement GIMLI. Additional features, helper functions, and other considerations when using/extending the library are provided in its documentation.

4.1. OBSTACLES:

Many deep learning frameworks offer the technical functionality required to implement GIMLI, namely the ability to take gradients of gradients. However, there are two aspects of how we implement and train parametric models in such frameworks which inhibit our ability to flexibly implement Algorithm 1.

The first obstacle is that models are typically implemented statefully (e.g. torch.nn in PyTorch, keras.layers in Keras (Chollet et al., 2015), etc.) , meaning that the model's parameters are encapsulated in the model implementation, and are implicitly relied upon during the forward pass. Therefore while such models can be considered as functions theoretically, they are not pure functions practically, as their output is not uniquely determined by their explicit input, and equivalently the parameters for a particular forward pass typically cannot be trivially overridden or supplied at call time. This prevents us from tracking and backpropagating over the successive values of the model parameters θ within the inner loop described by Equation 3, through an implicit or explicit graph.

The second issue is that, as discussed at the end of Section 2.4 and in Appendix A, even though the operations used within popular optimizers are mathematically differentiable functions of the parameters, gradients, and select hyperparameters, these operations are not tracked in various framework's implicit or explicit graph/gradient tape implementations when an optimization step is run. This is with good reason: updating model parameters in-place is memory efficient, as typically there is no need to keep references to the previous version of parameters. Ignoring the gradient dependency formed by allowing backpropagation through an optimization step essentially makes it safe to release memory allocated to historical parameters and intermediate model states once an update has been completed. Together, these obstacles essentially prevent us from practically satisfying Requirement II of Section 2.4.

4.2. MAKING STATEFUL MODULES STATELESS:

As we wish to track and backpropagate through intermediate states of parameters during the inner loop, we keep a record of such states which can be referenced during the backward pass stage of the outer loop in Algorithm 1. The typical way this is done in implementations of meta-learning algorithms such as MAML is to rewrite a "stateless" version of the inner loop's model, permitting the use, in each invocation of the model's forward pass, of weights which are otherwise tracked on the gradient graph/tape. While this addresses the issue, it is an onerous and limiting solution, as exploring new models within such algorithms invariably requires their reimplementation in a stateless style. This typically prevents the researcher from experimenting with third-party codebases, complicated models, or those which requiring loading pre-trained weights, without addressing a significant and unwelcome engineering challenge.

A more generic solution, permitting the use of existing stateful modules (including with pre-loaded activations), agnostic to the complexity or origin of the code which defines them, is to modify the run-time instance of the model's parent class to render them effectively function, a technique often referred to as "monkey-patching". The high-level function unnamedlib.monkeypatch() does this by taking as argument a torch.nn.Module instance and the structure of its nested sub-modules. As it traverses this structure, it clones the parent classes of submodule instances, leaving their functionality intact save for that of the forward method which implements the forward pass. Here, it replaces the call to the forward method with one which first replaces the stateful parameters of the submodule with ones provided as additional arguments to the patched forward, before calling the original class's bound forward method, which will now used the parameters provided at call time. This method is generic and derived from first-principles analysis of the torch.nn.Module implementation, ensuring that any first or third-party implementation of parametric models which are subclasses of torch.nn.Module and do not abuse the parent class at runtime will be supported by this recursive patching process.

4.3. MAKING OPTIMIZERS DIFFERENTIABLE:

Again, as part of our need to make the optimization process differentiable in order to satisfy Requirement II of Section 2.4, the typical solution is to write a version of SGD which does not modify parameters in-place, but treated as a differentiable function of the input parameters and hyperparameters akin to any other module in the training process. While this, again, is often considered a satisfactory solution in the meta-learning literature due to its simplicity, it too is limiting. Not only does the inability to experiment with other inner loop optimizers prevent research into the applicability of meta-learning algorithms to other optimization processes, the restriction to SGD also means that existing state-of-the-art methods used in practical domains cannot be extended using meta-learning methods such as those described in Section 3, lest they perform competitively when trained with SGD.

Here, while less generic, the solution provided by the high-level function unnamedlib.get diff optim() is to render a PyTorch optimizer instance differentiable by mapping its parent class to a differentiable reimplementation of the instance's parent class. The reimplementation is typically a copy of the optimizer's step logic, with in-place operations being replaced with gradient-tracking ones (a process which is syntactically simple to execute in PyTorch).

To this, we add wrapper code which copies the optimizer's state, and allows safe branching off of it, to permit "unrolling" of the optimization process within an inner loop (cf. the recurrence from Equation 3) without modifying the initial state of the optimizer (e.g. to permit several such unrolls, or to preserve state if inner loop optimizer is used elsewhere in the outer loop). Most of the optimizers in torch.optim are covered by this method. Here too, a runtime modification of the parent optimizer class could possibly be employed as was done for torch.nn.Modules, but this would involve modifying Python objects at a far finer level of granularity. We find that supporting a wide and varied class of optimizers is a sufficient compromise to enable further research. 5

5. EXPERIMENTS:

In this section, we briefly present some results of experiments using select existing methods from Section 3.1, to show case how unnamedlib can be used to simply implement ablation studies and searches over model architectures, optimizers, and other aspects of an experiment. This could, naturally, be done without appeal to the library. However, in such cases, changing the model architecture or optimizer requires re-implementing the model functionally or optimization step differentiably. Here such changes require writing no new code (excluding the line where the model is defined).

5.1. META-LEARNING LEARNING RATES WITH HIGHER GRANULARITY:

As the first application for unnamedlib, we set up a simple meta-learning task where we metaoptimize the learning rate. Employing a handcrafted annealing schedule for learning rates has been the de facto approach to improving a learning algorithm. While scheduled annealing can help to boost the final performance, it usually requires significant hand-tuning of the schedule. In contrast, we adjust learning rates automatically using meta-learning, which unnamedlib enables for arbitary models and optimizers.

We take an image classification model DenseNet-BC(k=12) (Huang et al., 2016) , that provides competitive state-of-the-art results on CIFAR10, and modify its training procedure by replacing the multi-step annealing schedule of learning rate with a meta-optimizer. Specifically, we treat learning rate for each inner optimizer's parameter group as a separate meta-parameter that we will metalearn. Doing so allows individual model parameters to have finer learning rates, that are adjusted automatically. The GIMLI algorithm provides for a clean implementation of such training procedure. 

Learning rates:

Figure 1 : Comparison of meta-learned learning rates against fixed and multi-step annealed for training DenseNet-BC(k=12) on CIFAR10. We observe convergence near state-of-the-art with better sample complexity that using a hand-designed annealing schedule.

We first train two baseline variants of DenseNet-BC(k=12) using setup from (Huang et al., 2016) . In the first variant we keep the learning rate fixed to 0.1 for all 300 epochs, while the second configuration takes advantage of a manually designed multi-step annealing schedule, which drops learning rate by 10 after 150 and 225 epochs. For the meta-learning variant, we split the training set into two disjoint pieces, one for training and another for validation, in proportion of 99 : 1. We then use per parameter group learning rates (299 in total) for meta-optimization, initializing each learning rate to 0.1. We perform one meta-update step after each epoch, where we unroll inner optimizer for 15 steps using batches from the training set, and compute meta-test error on the validation set over 10 batches from the validation set. We use batch size of 16 for meta-update, rather than 64 as in the base training loop. We use Adam (Kingma & Ba, 2014) with default parameters as a choice for meta-optimizer. We average results over 3 random seeds. Figure 1 demonstrates that our method is able reach the state-of-the-art performance faster.

5.2. ABLATING MAML'S MODEL ARCHITECTURE AND INNER OPTIMIZER:

The unnamedlib library enables the exploration of new MAML-like models and inner-loop optimizers, which historically has required non-trivial implementations of the fast-weights for the model parameters and inner optimizers as done in Antoniou et al. (2018) ; Deleu et al. (2019) . These ablations can be important for squeezing the last few bits of accuracy on well-established tasks and baselines that are already near-optimal as shown in Chen et al. (2019) , and is even more important for developing new approaches and tasks that deal with different kinds of data and require adaptation to be done over non-standard operations.

To illustrate how easy unnamedlib makes these ablations, in this section we take a closer look at different model architecture and optimizer choices for the MAML++ approach (Antoniou et al., 2018) . MAML++ uses a VGG network with a SGD inner optimizer for the the Omniglot (Lake et al., 2015) and Mini-Imagenet (Vinyals et al., 2016; Ravi & Larochelle, 2016) tasks. We start with the official MAML++ code and evaluation procedure and use unnamedlib to ablate across VGG, ResNet, and DenseNet models and SGD and Adam optimizers. We provide more details about these in Appendix B. We denote the combination of model and inner optimizer choice with <model>+<opt>. One finding of standalone interest is that we have kept most of the features from MAML++, except we significantly increase the base VGG+SGD performance by using batch normalization in training mode everywhere as in (Finn et al., 2017) instead of using per-timestep statistics and parameters as MAML++ proposes. In theory, this enables more inner optimization steps to be rolled out at test time, which otherwise is not possible with MAML++ because of the pertimestep information, for simplicity in this paper we have not explored this and keep every algorithm, optimizer, and mode to five inner optimization steps. When using Adam as the inner optimizer, we initialize the first and second moment statistics to the statistics from the outer optimizer, which is also Adam, and learn per-parameter-group learning rates and rolling average β coefficients. Our results in Table 1 show that we are able to push the accuracy of MAML++ slightly up with this ablation. We note that this pushes the performance of MAML++ closer to that of state-of-the-art methods such as LEO (Rusu et al., 2018) . Appendix B shows our full experimental results, and we note that in some cases Adam slightly outperforms SGD for a particular model.

6. CONCLUSION:

To summarize, we have presented GIMLI, a general formulation of a wide class of existing and potential meta-learning approaches, listed and proved the requirements that must be satisfied for such approaches to be possible, and specified a general algorithmic formulation of such approaches. We've described a lightweight library, unnamedlib, which extends PyTorch to enable the easy and natural implementation of such meta-learning approaches at scale. Finally we've demonstrated some of its potential applications. We hope to have made the case not only for the use of the mathematical and software tools we present here, but have also provided suitable encouragement for other researchers to use them and explore the boundaries of what can be done within this broad class of meta-learning approaches. 

∇ ϕ θ t+1 = ∇ ϕ Adagrad(θ t , G t ) = ∇ ϕ [θ t - η t i=1 G 2 i G t ] = ∇ ϕ θ t - η∇ ϕ G t t i=1 G 2 i - G t ∇ ϕ η t i=1 G 2 i + ηG t t i=1 G i ∇ ϕ G i t i=1 G 2 i 3 2

where η is the global learning rate, and 

∇ ϕ η is defined iff η ⊆ ϕ else G t ∇ ϕ η t i=1 G 2 i = 0 ∇ θt θ t+1 = ∇ θt Adagrad(θ t , G t ) = ∇ θt [θ t - η t i=1 G 2 i G t ] = ∇ θt θ t -

:

Throughout this paper, we will write arg min x in functional form as follows to facilitate formalizing nested optimization, with x a variable and E an expression (presumably a function of x):

For stateful optimizers, we assume that, where the state is itself a function of previous values of the network parameters (e.g. moving averages of parameter weights), it is included in the computation of gradients of the meta-loss with regard to those parameters.

This requirement is typically only satisfied by deterministic optimizers. However, for stochastic optimizers, if there exists a reparameterization trick (seeKingma & Welling, 2013) for the distribution from which gradients or other components of the update are sampled, the requirement may also be satisfied.

We will publicly release the library described in this section in the coming weeks under the Apache license. References to the library name have been anonymized throughout the paper to preserve the double-blind reviewing process.

We also provide documentation as to how to write differentiable third party optimizers and supply helper functions such as unnamedlib.register optim() to register them for use with the library at runtime.

Figures and Tables:

Results from ablating MAML++ architecture and inner optimizers. "Our base" results are from our VGG model trained with SGD in our MAML++ variant, and "our best" results show the best test accuracy found, with the best model/optimizer combination shown in the text below.

Table content: <table><tr><td/><td/><td colspan="2">omniglot test accuracy</td><td/><td colspan="2">miniImageNet test accuracy</td></tr><tr><td/><td colspan="2">5 Way</td><td>20 Way</td><td/><td colspan="2">5 Way</td></tr><tr><td>Approach</td><td>1 Shot</td><td>5 Shot</td><td>1 Shot</td><td>5 Shot</td><td>1 Shot</td><td>5 Shot</td></tr><tr><td colspan="6">MAML++ (Antoniou et al., 2018) 99.53 ± 0.26% 99.93 ± 0.09% 97.65 ± 0.05% 99.33 ± 0.03% 52.15 ± 0.26%</td><td>68.32 ± 0.44</td></tr><tr><td colspan="7">MAML++ (Our base) 99.62 ± 0.08% 99.86 ± 0.02% 97.21 ± 0.11% 99.13 ± 0.13% 56.33 ± 0.27% 75.13 ± 0.67%</td></tr><tr><td colspan="7">MAML++ (Our best) 99.91 ± 0.05% 99.87 ± 0.03% 99.00 ± 0.33% 99.76 ± 0.01% 56.33 ± 0.27% 76.73 ± 0.52%</td></tr><tr><td/><td>resnet-4+SGD</td><td colspan="3">resnet-4+SGD resnet-12+SGD resnet-8+SGD</td><td>vgg+SGD</td><td>resnet-8+SGD</td></tr></table>

Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. In Advances in neural information processing systems, pp. 2951-2959, 2012. Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems, pp. 3630-3638, 2016. Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-256, 1992.∇ ϕ θ t+1 = ∇ ϕ SGD(θ t , G t ) = ∇ ϕ [θ t -αG t ] = ∇ ϕ θ t -G t ∇ ϕ α -α∇ ϕ G twhere α is the learning rate, and∇ ϕ α is defined iff α ⊆ ϕ else G t ∇ ϕ α = 0 ∇ θt θ t+1 = ∇ θt SGD(θ t , G t ) = ∇ θt [θ t -αG t ] = ∇ θt θ t -α∇ θt G t = 1 -α∇ 2

Table content: <table><tr><td>A OPTIMIZER DIFFERENTIABILITY</td><td/><td/></tr><tr><td>A.1 SGD</td><td/><td/></tr><tr><td>θt</td><td>train t</td><td>(θ t , ϕ)</td></tr><tr><td>A.2 ADAGRAD</td><td/><td/></tr></table>

Full MAML++ model and inner optimizer sweep search results.

Table content: <table><tr><td/><td/><td colspan="2">η∇ θt G t t i=1 G 2 i</td><td colspan="2">+</td><td colspan="2">ηG 2 t ∇ θt G t t i i=1 G 2</td><td>3 2</td></tr><tr><td>= 1 -</td><td>η∇ 2 θt</td><td>train t</td><td colspan="2">(θ t , ϕ)</td><td>+</td><td>ηG 2 t ∇ 2 θt</td><td colspan="2">train t</td><td>(θ t , ϕ)</td></tr><tr><td/><td/><td colspan="2">t i=1 G 2 i</td><td/><td/><td colspan="3">t i=1 G 2 i</td></tr></table>

Peer review (iclr2020):
Summary: The authors present a PyTorch based framework for performing second-order reverse mode autodiff for meta-learning. First, the authors present a formalization of a general prototypical meta-learning setting. They then provide an algorithm that solves this problem via gradient based optimization. Finally, perhaps the main contribution is a specific PyTorch implementation of said algorithm. The type of meta-learning setting the authors consider is one where a gradient based inner loop optimizer finds by performing a finite number of steps. The inner loop optimizer is parameterized through that consists of two parts and . The parameters are somehow part of the loss used for training in the inner loop. Example: Regularization paramter. The parameters do not occur in the loss but in the optimizer step. Example: Learning rate. Example of an inner loop step: where are the parameters of a neural network, is the training loss, is the regularizer, is the learning rate, is the regularization parameter. In this example we would have . The authos assume , the output of the inner loop after steps to be differentiable wrt . Furthermore, the meta-learning loss is assumed to be differentiable wrt so that a gradient of the meta-learning loss wrt to can be computed. The authors also assume the meta-learning loss to be sufficient smooth in such that a gradient based optimization can even be used for meta learning to a local optimum. The authors explicitly write down the reverse mode auto differentiation of the inner loop and show how to, in that way, compute the gradient of the meta-learning loss wrt to . The reverse (adjoint) mode auto differentiation of the above example inner loop step is the following step (iterated over in reverse down from to : where accumulates the gradient of wrt and accumulates the gradient of wrt . The authors give some implementation details specific to some frameworks necessary for implementing such *gradient of an inner loop*. The authors present experiments where they show how to meta-learn learning rates with their framework. They also how their framework can be used to quickly implement a MAML type meta-learning optimizer ablation study comparing various combinations of architecture, optimizer and inner loop steps etc.. Recommendation: I propose to reject the paper. In my eyes the only contribution is the implementation of a meta-learner in PyTorch based on well known methods. The provided unifying formalization is theoretically inaccurate (see below) and to me come across as merely a motivation for their framework (but no value added compared to existing literature). There is no new insight provided on the software engineering level either as far as I can see. Detailed comments: - Page 1: ...provides tooling for analysing the provable requirements... seems like a complicated way of saying something that could be said simple - Page 2: Without loss of generality, ... You are assuming a parametric model. Not sure what generality this phrase refers to. - Page 2: A formalization as is inaccurate in the sense that it does not acknowledge the existing of multiple optima. I would recommend not to use the operator here, since for something like a neural network would for example either return a global optimum (which no optimizer used in practice finds, and is not ment here) or would take on a set value with multiple local minima for example. In the same context, the authors should mention the issues about uniqueness of optima (we are not even really finding optima when training neural networks), implicit functions / implicit differentiation In the context of using stochastic optimizers one should also at least mention something about the differentiability of outputs of such optimizers and how they potentially depend on randomness of mini-batches (what if different randomness is used with the same or a perturbed hyper parameter?) - Page 3: You mention the potential statefulness of the optimizer. Why not explicitly carry it in the math notation? Probably things would get cluttered but saying it should be covered within does not seem reasonable to me. - Page 3: While this may seem like a fairly trivial formalization... Yes, but also nesting this in an outer loop is fairly trivial in the sense that it is a well known approach. - Page 4: there exist continuous hyperparam... If they are not continuous then they should not even occur in this formalization so saying there exist... does not make much sense to me here implies that is a set from notation although we are treating it as a vector everywhere else - Page 4: All of section 2.4 seems somewhat trivial to me, but I guess that is highly subjective. - Page 5: in the definition of stop operator perhaps use - Page 5: Perhaps explicitly mention how your approach differs from a reverse mode differentiation of training or if it does not differ, say this. - Page 14: When talking about _S_GD (instead of just GD) perhaps mention something about non-existence of mini-batch randomness / being deterministic