Bootstrapped Meta-Learning

1 INTRODUCTION . In a standard machine learning problem , a learner or agent learns a task by iteratively adjusting its parameters under a given update rule , such as Stochastic Gradient Descent ( SGD ) . Typically , the learner ’ s update rule must be tuned manually . In contrast , humans learn seamlessly by relying on previous experiences to inform their learning processes ( Spelke & Kinzler , 2007 ) . For a ( machine ) learner to have the same capability , it must be able to learn its update rule ( or such inductive biases ) . Meta-learning is one approach that learns ( parts of ) an update rule by applying it for some number of steps and then evaluating the resulting performance ( Schmidhuber , 1987 ; Hinton & Plaut , 1987 ; Bengio et al. , 1991 ) . For instance , a well-studied and often successful approach is to tune parameters of a gradient-based update , either online during training on a single task ( Bengio , 2000 ; Maclaurin et al. , 2015 ; Xu et al. , 2018 ; Zahavy et al. , 2020 ) , or meta-learned over a distribution of tasks ( Finn et al. , 2017 ; Rusu et al. , 2019 ; Flennerhag et al. , 2020 ; Jerfel et al. , 2019 ; Denevi et al. , 2019 ) . More generally , the update rule can be an arbitrary parameterised function ( Hochreiter et al. , 2001 ; Andrychowicz et al. , 2016 ; Kirsch et al. , 2019 ; Oh et al. , 2020 ) , or the function itself can be meta-learned jointly with its parameters ( Alet et al. , 2020 ; Real et al. , 2020 ) . Meta-learning is challenging because to evaluate an update rule , it must first be applied . This often leads to high computational costs . As a result most works optimise performance after K applications of the update rule and assume that this yields improved performance for the remainder of the learner ’ s lifetime ( Bengio et al. , 1991 ; Maclaurin et al. , 2015 ; Metz et al. , 2019 ) . When this assumption fails , meta-learning suffers from a short-horizon bias ( Wu et al. , 2018 ; Metz et al. , 2019 ) . Similarly , optimizing the learner ’ s performance after K updates can fail to account for the process of learning , causing another form of myopia ( Flennerhag et al. , 2019 ; Stadie et al. , 2018 ; Chen et al. , 2016 ; Cao et al. , 2019 ) . Challenges in meta-optimisation have been observed to cause degraded lifetime performance ( Lv et al. , 2017 ; Wichrowska et al. , 2017 ) , collapsed exploration ( Stadie et al. , 2018 ; Chen et al. , 2016 ) , biased learner updates ( Stadie et al. , 2018 ; Zheng et al. , 2018 ) , and poor generalisation performance ( Wu et al. , 2018 ; Yin et al. , 2020 ; Triantafillou et al. , 2020 ) . We argue that defining the meta-learner ’ s objective directly in terms of the learner ’ s objective—i.e . the performance after K update steps—creates two bottlenecks in meta-optimisation . The first bottleneck is curvature : the meta-objective is constrained to the same type of geometry as the learner ; the second is myopia : the meta-objective is fundamentally limited to evaluating performance within the K-step horizon , but ignores future learning dynamics . Our goal is to design an algorithm that removes these . The algorithm relies on two main ideas . First , to mitigate myopia , we introduce the notion of bootstrapping a target from the meta-learner itself or from some other update rule , what we refer to as a meta-bootstrap . This allows us to infuse information about learning dynamics into the objective . Second , to control curvature , we formulate the meta-objective in terms of minimising distance ( or divergence ) to the bootstrapped target . This gives us control over the optimisation landscape . The key insight is that the meta-learner can learn to learn from itself by trying to match future desired updates with fewer steps . This leads to a bootstrapping effect , where improvements beget improvements . We present a detailed formulation in Section 3 ; on a high level , as in previous works , we first unroll the meta-learned update rule for K steps to obtain the learner ’ s new parameters . Whereas standard meta-objectives optimise the update rule with respect to ( w.r.t . ) the learner ’ s performance under the new parameters , our proposed algorithm constructs the meta-objective in two steps : 1 . It bootstraps a target from the learner ’ s new parameters . In this paper , we generate targets by continuing to update the learner ’ s parameters—either under the meta-learned update rule or another update rule—for some number of steps . 2 . The learner ’ s new parameters—which are a function of the meta-learner ’ s parameters—and the target are projected onto a matching space . A simple example is Euclidean parameter space . To control curvature , we may choose a different ( pseudo- ) metric space . For instance , a common choice under probabilistic models is the Kullback-Leibler ( KL ) divergence . The meta-learner is optimised by minimising distance to the bootstrapped target . We focus on gradient-based optimisation , but other optimisation routines are equally applicable . By optimising meta-parameters in a well-behaved space , we can drastically reduce ill-conditioning and other phenomena that disrupt meta-optimisation . In particular , this form of Bootstrapped Meta-Gradient ( BMG ) enables us to infuse information about future learning dynamics without increasing the number of update steps to backpropagate through . In effect , the meta-learner becomes its own teacher . Because BMG optimises the meta-learner to reach a desired future target state faster , it embodies a form of acceleration akin to Nesterov Momentum ( Nesterov , 1983 ) and Raphson-Newton updates . We show that BMG can guarantee performance improvements ( Theorem 1 ) and that this guarantee can be stronger than under standard meta-gradients ( Corollary 1 ) . Empirically , we find that BMG provides substantial performance improvements over standard meta-gradients in various settings . We obtain a new state-of-the-art result for model-free agents on Atari ( Section 5.2 ) and improve upon MAML ( Finn et al. , 2017 ) in the few-shot setting ( Section 6 ) . Finally , we demonstrate how BMG enables new forms of meta-learning , exemplified by meta-learning ε-greedy exploration ( Section 5.1 ) . 2 RELATED WORK . Bootstrapping as used here stems from temporal difference ( TD ) algorithms in reinforcement learning ( RL ) ( Sutton , 1988 ) . In these algorithms , an agent learns a value function by using its own future predictions as targets . Bootstrapping has recently been introduced in the self-supervised setting ( Guo et al. , 2020 ; Grill et al. , 2020 ) . In this paper , we introduce the idea of bootstrapping in the context of meta-learning , where a meta-learner learns about an update rule by generating future targets from it . Our approach to target matching is related to methods in multi-task meta-learning ( Flennerhag et al. , 2019 ; Nichol et al. , 2018 ) that meta-learn an initialisation for SGD by minimising the Euclidean distance to task-optimal parameters . BMG generalise this concept by allowing for arbitrary metaparameters , matching functions , and target bootstraps . It is further related the more general concept of self-referential meta-learning ( Schmidhuber , 1987 ; 1993 ) , where the meta-learned update rule is used to optimise its own meta-objective . Target matching under KL divergences results in a form of distillation ( Hinton et al. , 2015 ) , where an online network ( student ) is encouraged to match a target network ( teacher ) . In a typical setup , the target is either a fixed ( set of ) expert ( s ) ( Hinton et al. , 2015 ; Rusu et al. , 2015 ) or a moving aggregation of current experts ( Teh et al. , 2017 ; Grill et al. , 2020 ) , whereas BMG bootstraps a target by following an update rule . Finally , BMG is loosely inspired by trust-region methods that introduce a distance function to regularize gradient updates ( Pascanu & Bengio , 2014 ; Schulman et al. , 2015 ; Tomar et al. , 2020 ; Hessel et al. , 2021 ) . 3 BOOTSTRAPPED META-GRADIENTS We begin in the single-task setting and turn to multi-task meta-learning in Section 6 . The learner ’ s problem is to minimize a stochastic objective f ( x ) : = E [ ` ( x ; ζ ) ] over a data distribution p ( ζ ) , where ζ denotes a source of data and x ∈ X ⊂ Rnx denotes the learner ’ s parameters . In RL , f is typically the ( negative ) expected value of a policy πx ; in supervised learning , f may be the expected negative loglikelihood under a probabilistic model πx . We provide precise formulations in Sections 5 and 6 . The meta-learner ’ s problem is to learn an update rule ϕ : X ×H×W → X that updates the learner ’ s parameters by x ( 1 ) = x+ϕ ( x , h , w ) given x ∈ X , a learning state h ∈ H , and meta-parameters w ∈ W ⊂ Rnw of the update rule . We make no assumptions on the update rule other than differentiability in w. As such , ϕ can be a recurrent neural network ( Hochreiter et al. , 2001 ; Wang et al. , 2016 ; Andrychowicz et al. , 2016 ) or gradient descent ( Bengio , 2000 ; Maclaurin et al. , 2015 ; Finn et al. , 2017 ) . The learning state h contains any other data required to compute the update ; in a black-box setting h contains an observation and the recurrent state of the network ; for gradient-based updates , h contains the ( estimated ) gradient of f at x along with any auxiliary information ; for instance , SGD is given by x ( 1 ) = x−α∇xf ( x ) with h = ∇xf ( x ) , w = α ∈ R+ . The standard meta-gradient ( MG ) optimises meta-parameters w by taking K steps under ϕ and evaluating the resulting learner parameter vector under f . With a slight abuse of notation , let x ( K ) ( w ) denote the learner ’ s parameters after K applications of ϕ starting from some ( x , h , w ) , where ( x , h ) evolve according to ϕ and the underlying data distribution . The MG update is defined by w′ = w−β∇wf ( x ( K ) ( w ) ) , β ∈ R+ . ( 1 ) Extensions involve averaging the performance over all iterates x ( 1 ) , . . . , x ( K ) ( Andrychowicz et al. , 2016 ; Chen et al. , 2016 ; Antoniou et al. , 2019 ) or using validation data in the meta-objective ( Bengio et al. , 1991 ; Maclaurin et al. , 2015 ; Finn et al. , 2017 ; Xu et al. , 2018 ) . We observe two bottlenecks in the meta-objective in Eq . 1 . First , the meta-objective is subject to the same curvature as the learner . Thus if f is ill-conditioned , so will the meta-objective be . Second , the meta-objective is only able to evaluate the meta-learner on dynamics up to the Kth step , but ignores effects of future updates . To tackle myopia , we introduce a Target Bootstrap ( TB ) ξ : X 7→ X that maps the meta-learner ’ s output x ( K ) into a bootstrapped target x̃ = ξ ( x ( K ) ) . We focus on TBs that unroll ϕ a further L− 1 steps before taking final gradient step on f , with targets of the form x̃ = x ( K+L−1 ) −α∇f ( x ( K+L−1 ) ) . This TB encourages the meta-learner to reach future states on its trajectory faster while nudging the trajectory in a descent direction . Crucially , regardless of the bootstrapping strategy , we do not backpropagate through the target . Akin to temporal difference learning in RL ( Sutton , 1988 ) , the target is a fixed goal that the meta-learner should try to produce within the K-step budget . Finally , to improve the meta-optimisation landscape , we introduce a matching function µ : X ×X → R+ that measures the ( dis ) similarity between the meta-learner ’ s output , x ( K ) ( w ) , and the target , x̃ , in a matching space defined by µ ( see Figure 1 ) . Taken together , the BMG update is defined by w̃ = w−β∇w µ ( x̃ , x ( K ) ( w ) ) , β ∈ R+ , ( 2 ) where the gradient is with respect to the second argument of µ . Thus , BMG describes a family of algorithms based on the choice of matching function µ and TB ξ . In particular , MG is a special case of BMG under matching function µ ( x̃ , x ( K ) ) = ‖x̃−x ( K ) ‖22 and TB ξ ( x ( K ) ) = x ( K ) − 12∇xf ( x ( K ) ) , since the bootstrapped meta-gradient reduces to the standard meta-gradient : ∇w ∥∥∥x̃− x ( K ) ( w ) ∥∥∥2 2 = −2D ( x̃− x ( K ) ) = D∇xf ( x ( K ) ) = ∇wf ( x ( K ) ( w ) ) , ( 3 ) where D denotes the ( transposed ) Jacobian of x ( K ) ( w ) . For other matching functions and target strategies , BMG produces different meta-updates compared to MG. We discuss these choices below . Matching Function Of primary concern to us are models that output a probabilistic distribution , πx . A common pseudo-metric over a space of probability distributions is the Kullback-Leibler ( KL ) divergence . For instance , Natural Gradients ( Amari , 1998 ) point in the direction of steepest descent under the KL-divergence , often approximated through a KL-regularization term ( Pascanu & Bengio , 2014 ) . KL-divergences also arise naturally in RL algorithms ( Kakade , 2001 ; Schulman et al. , 2015 ; 2017 ; Abdolmaleki et al. , 2018 ) . Hence , a natural starting point is to consider KL-divergences between the target and the iterate , e.g . µ ( x̃ , x ( K ) ) = KL ( πx̃ ‖ πx ( K ) ) . In actor-critic algorithms ( Sutton et al. , 1999 ) , the policy defines only part of the agent—the value function defines the other . Thus , we also consider a composite matching function over both policy and value function . Target Bootstrap We analyze conditions under which BMG guarantees performance improvements in Section 4 and find that the target should co-align with the gradient direction . Thus , in this paper we focus on gradient-based TBs and find that they perform well empirically . As with matching functions , this is a small subset of all possible choices ; we leave the exploration of other choices for future work .

Summary: This paper broadly considers meta-learning, a.k.a. bilevel optimization, across single-task, multi-task, supervised learning, and reinforcement learning settings. The authors aim to resolve two issues with the standard outer-loop gradient-based optimization of the meta-parameters (assuming a differentiable inner-loop): first, since the meta-learning objective is typically computed from learner parameters after applying up to $K$ inner-loop updates, the meta-optimization is myopic in that it does not optimize for further inner-loop improvement after $K$ steps; second, since the functional form of the learner's objective $f$ is used to drive the outer-loop updates, the meta-learning objective inherits the curvature of $f$. The main algorithmic contribution consists of a family of meta-learning objectives called bootstrapped meta-learning, in which meta-parameters are optimized to bring post-inner-loop learner parameters $x^{(K)}$ closer to a bootstrap target (which are also learner parameters) computed from $x^{(K)}$. The authors show that bootstrapped meta-learning generalizes the "direct" (my terminology) gradient-based meta-parameter optimization used in many previous works, recovering it when using specific choices for the bootstrap computation function and learner parameter matching function. With certain strong assumptions, the authors theoretically motivate the use of gradient-based bootstrap target functions for bootstrapped meta-learning in terms of optimization progress. The authors make several experimental contributions: they use bootstrapped meta-learning to achieve state-of-the-art model-free performance on Atari-57, demonstrate the viability of bootstrapped meta-learning in few-shot image classification on miniImageNet, and show that the more flexible, general form of bootstrapped meta-learning can enable meta-learning parameters that do not appear in the computation graph for the task objective, e.g. meta-learning the exploration rate of the behavior policy in $\epsilon$-greedy $Q$-learning.