Bootstrapped Meta-Learning

1 INTRODUCTION . In a standard machine learning problem , a learner or agent learns a task by iteratively adjusting its parameters under a given update rule , such as Stochastic Gradient Descent ( SGD ) . Typically , the learner ’ s update rule must be tuned manually . In contrast , humans learn seamlessly by relying on previous experiences to inform their learning processes ( Spelke & Kinzler , 2007 ) . For a ( machine ) learner to have the same capability , it must be able to learn its update rule ( or such inductive biases ) . Meta-learning is one approach that learns ( parts of ) an update rule by applying it for some number of steps and then evaluating the resulting performance ( Schmidhuber , 1987 ; Hinton & Plaut , 1987 ; Bengio et al. , 1991 ) . For instance , a well-studied and often successful approach is to tune parameters of a gradient-based update , either online during training on a single task ( Bengio , 2000 ; Maclaurin et al. , 2015 ; Xu et al. , 2018 ; Zahavy et al. , 2020 ) , or meta-learned over a distribution of tasks ( Finn et al. , 2017 ; Rusu et al. , 2019 ; Flennerhag et al. , 2020 ; Jerfel et al. , 2019 ; Denevi et al. , 2019 ) . More generally , the update rule can be an arbitrary parameterised function ( Hochreiter et al. , 2001 ; Andrychowicz et al. , 2016 ; Kirsch et al. , 2019 ; Oh et al. , 2020 ) , or the function itself can be meta-learned jointly with its parameters ( Alet et al. , 2020 ; Real et al. , 2020 ) . Meta-learning is challenging because to evaluate an update rule , it must first be applied . This often leads to high computational costs . As a result most works optimise performance after K applications of the update rule and assume that this yields improved performance for the remainder of the learner ’ s lifetime ( Bengio et al. , 1991 ; Maclaurin et al. , 2015 ; Metz et al. , 2019 ) . When this assumption fails , meta-learning suffers from a short-horizon bias ( Wu et al. , 2018 ; Metz et al. , 2019 ) . Similarly , optimizing the learner ’ s performance after K updates can fail to account for the process of learning , causing another form of myopia ( Flennerhag et al. , 2019 ; Stadie et al. , 2018 ; Chen et al. , 2016 ; Cao et al. , 2019 ) . Challenges in meta-optimisation have been observed to cause degraded lifetime performance ( Lv et al. , 2017 ; Wichrowska et al. , 2017 ) , collapsed exploration ( Stadie et al. , 2018 ; Chen et al. , 2016 ) , biased learner updates ( Stadie et al. , 2018 ; Zheng et al. , 2018 ) , and poor generalisation performance ( Wu et al. , 2018 ; Yin et al. , 2020 ; Triantafillou et al. , 2020 ) . We argue that defining the meta-learner ’ s objective directly in terms of the learner ’ s objective—i.e . the performance after K update steps—creates two bottlenecks in meta-optimisation . The first bottleneck is curvature : the meta-objective is constrained to the same type of geometry as the learner ; the second is myopia : the meta-objective is fundamentally limited to evaluating performance within the K-step horizon , but ignores future learning dynamics . Our goal is to design an algorithm that removes these . The algorithm relies on two main ideas . First , to mitigate myopia , we introduce the notion of bootstrapping a target from the meta-learner itself or from some other update rule , what we refer to as a meta-bootstrap . This allows us to infuse information about learning dynamics into the objective . Second , to control curvature , we formulate the meta-objective in terms of minimising distance ( or divergence ) to the bootstrapped target . This gives us control over the optimisation landscape . The key insight is that the meta-learner can learn to learn from itself by trying to match future desired updates with fewer steps . This leads to a bootstrapping effect , where improvements beget improvements . We present a detailed formulation in Section 3 ; on a high level , as in previous works , we first unroll the meta-learned update rule for K steps to obtain the learner ’ s new parameters . Whereas standard meta-objectives optimise the update rule with respect to ( w.r.t . ) the learner ’ s performance under the new parameters , our proposed algorithm constructs the meta-objective in two steps : 1 . It bootstraps a target from the learner ’ s new parameters . In this paper , we generate targets by continuing to update the learner ’ s parameters—either under the meta-learned update rule or another update rule—for some number of steps . 2 . The learner ’ s new parameters—which are a function of the meta-learner ’ s parameters—and the target are projected onto a matching space . A simple example is Euclidean parameter space . To control curvature , we may choose a different ( pseudo- ) metric space . For instance , a common choice under probabilistic models is the Kullback-Leibler ( KL ) divergence . The meta-learner is optimised by minimising distance to the bootstrapped target . We focus on gradient-based optimisation , but other optimisation routines are equally applicable . By optimising meta-parameters in a well-behaved space , we can drastically reduce ill-conditioning and other phenomena that disrupt meta-optimisation . In particular , this form of Bootstrapped Meta-Gradient ( BMG ) enables us to infuse information about future learning dynamics without increasing the number of update steps to backpropagate through . In effect , the meta-learner becomes its own teacher . Because BMG optimises the meta-learner to reach a desired future target state faster , it embodies a form of acceleration akin to Nesterov Momentum ( Nesterov , 1983 ) and Raphson-Newton updates . We show that BMG can guarantee performance improvements ( Theorem 1 ) and that this guarantee can be stronger than under standard meta-gradients ( Corollary 1 ) . Empirically , we find that BMG provides substantial performance improvements over standard meta-gradients in various settings . We obtain a new state-of-the-art result for model-free agents on Atari ( Section 5.2 ) and improve upon MAML ( Finn et al. , 2017 ) in the few-shot setting ( Section 6 ) . Finally , we demonstrate how BMG enables new forms of meta-learning , exemplified by meta-learning ε-greedy exploration ( Section 5.1 ) . 2 RELATED WORK . Bootstrapping as used here stems from temporal difference ( TD ) algorithms in reinforcement learning ( RL ) ( Sutton , 1988 ) . In these algorithms , an agent learns a value function by using its own future predictions as targets . Bootstrapping has recently been introduced in the self-supervised setting ( Guo et al. , 2020 ; Grill et al. , 2020 ) . In this paper , we introduce the idea of bootstrapping in the context of meta-learning , where a meta-learner learns about an update rule by generating future targets from it . Our approach to target matching is related to methods in multi-task meta-learning ( Flennerhag et al. , 2019 ; Nichol et al. , 2018 ) that meta-learn an initialisation for SGD by minimising the Euclidean distance to task-optimal parameters . BMG generalise this concept by allowing for arbitrary metaparameters , matching functions , and target bootstraps . It is further related the more general concept of self-referential meta-learning ( Schmidhuber , 1987 ; 1993 ) , where the meta-learned update rule is used to optimise its own meta-objective . Target matching under KL divergences results in a form of distillation ( Hinton et al. , 2015 ) , where an online network ( student ) is encouraged to match a target network ( teacher ) . In a typical setup , the target is either a fixed ( set of ) expert ( s ) ( Hinton et al. , 2015 ; Rusu et al. , 2015 ) or a moving aggregation of current experts ( Teh et al. , 2017 ; Grill et al. , 2020 ) , whereas BMG bootstraps a target by following an update rule . Finally , BMG is loosely inspired by trust-region methods that introduce a distance function to regularize gradient updates ( Pascanu & Bengio , 2014 ; Schulman et al. , 2015 ; Tomar et al. , 2020 ; Hessel et al. , 2021 ) . 3 BOOTSTRAPPED META-GRADIENTS We begin in the single-task setting and turn to multi-task meta-learning in Section 6 . The learner ’ s problem is to minimize a stochastic objective f ( x ) : = E [ ` ( x ; ζ ) ] over a data distribution p ( ζ ) , where ζ denotes a source of data and x ∈ X ⊂ Rnx denotes the learner ’ s parameters . In RL , f is typically the ( negative ) expected value of a policy πx ; in supervised learning , f may be the expected negative loglikelihood under a probabilistic model πx . We provide precise formulations in Sections 5 and 6 . The meta-learner ’ s problem is to learn an update rule ϕ : X ×H×W → X that updates the learner ’ s parameters by x ( 1 ) = x+ϕ ( x , h , w ) given x ∈ X , a learning state h ∈ H , and meta-parameters w ∈ W ⊂ Rnw of the update rule . We make no assumptions on the update rule other than differentiability in w. As such , ϕ can be a recurrent neural network ( Hochreiter et al. , 2001 ; Wang et al. , 2016 ; Andrychowicz et al. , 2016 ) or gradient descent ( Bengio , 2000 ; Maclaurin et al. , 2015 ; Finn et al. , 2017 ) . The learning state h contains any other data required to compute the update ; in a black-box setting h contains an observation and the recurrent state of the network ; for gradient-based updates , h contains the ( estimated ) gradient of f at x along with any auxiliary information ; for instance , SGD is given by x ( 1 ) = x−α∇xf ( x ) with h = ∇xf ( x ) , w = α ∈ R+ . The standard meta-gradient ( MG ) optimises meta-parameters w by taking K steps under ϕ and evaluating the resulting learner parameter vector under f . With a slight abuse of notation , let x ( K ) ( w ) denote the learner ’ s parameters after K applications of ϕ starting from some ( x , h , w ) , where ( x , h ) evolve according to ϕ and the underlying data distribution . The MG update is defined by w′ = w−β∇wf ( x ( K ) ( w ) ) , β ∈ R+ . ( 1 ) Extensions involve averaging the performance over all iterates x ( 1 ) , . . . , x ( K ) ( Andrychowicz et al. , 2016 ; Chen et al. , 2016 ; Antoniou et al. , 2019 ) or using validation data in the meta-objective ( Bengio et al. , 1991 ; Maclaurin et al. , 2015 ; Finn et al. , 2017 ; Xu et al. , 2018 ) . We observe two bottlenecks in the meta-objective in Eq . 1 . First , the meta-objective is subject to the same curvature as the learner . Thus if f is ill-conditioned , so will the meta-objective be . Second , the meta-objective is only able to evaluate the meta-learner on dynamics up to the Kth step , but ignores effects of future updates . To tackle myopia , we introduce a Target Bootstrap ( TB ) ξ : X 7→ X that maps the meta-learner ’ s output x ( K ) into a bootstrapped target x̃ = ξ ( x ( K ) ) . We focus on TBs that unroll ϕ a further L− 1 steps before taking final gradient step on f , with targets of the form x̃ = x ( K+L−1 ) −α∇f ( x ( K+L−1 ) ) . This TB encourages the meta-learner to reach future states on its trajectory faster while nudging the trajectory in a descent direction . Crucially , regardless of the bootstrapping strategy , we do not backpropagate through the target . Akin to temporal difference learning in RL ( Sutton , 1988 ) , the target is a fixed goal that the meta-learner should try to produce within the K-step budget . Finally , to improve the meta-optimisation landscape , we introduce a matching function µ : X ×X → R+ that measures the ( dis ) similarity between the meta-learner ’ s output , x ( K ) ( w ) , and the target , x̃ , in a matching space defined by µ ( see Figure 1 ) . Taken together , the BMG update is defined by w̃ = w−β∇w µ ( x̃ , x ( K ) ( w ) ) , β ∈ R+ , ( 2 ) where the gradient is with respect to the second argument of µ . Thus , BMG describes a family of algorithms based on the choice of matching function µ and TB ξ . In particular , MG is a special case of BMG under matching function µ ( x̃ , x ( K ) ) = ‖x̃−x ( K ) ‖22 and TB ξ ( x ( K ) ) = x ( K ) − 12∇xf ( x ( K ) ) , since the bootstrapped meta-gradient reduces to the standard meta-gradient : ∇w ∥∥∥x̃− x ( K ) ( w ) ∥∥∥2 2 = −2D ( x̃− x ( K ) ) = D∇xf ( x ( K ) ) = ∇wf ( x ( K ) ( w ) ) , ( 3 ) where D denotes the ( transposed ) Jacobian of x ( K ) ( w ) . For other matching functions and target strategies , BMG produces different meta-updates compared to MG. We discuss these choices below . Matching Function Of primary concern to us are models that output a probabilistic distribution , πx . A common pseudo-metric over a space of probability distributions is the Kullback-Leibler ( KL ) divergence . For instance , Natural Gradients ( Amari , 1998 ) point in the direction of steepest descent under the KL-divergence , often approximated through a KL-regularization term ( Pascanu & Bengio , 2014 ) . KL-divergences also arise naturally in RL algorithms ( Kakade , 2001 ; Schulman et al. , 2015 ; 2017 ; Abdolmaleki et al. , 2018 ) . Hence , a natural starting point is to consider KL-divergences between the target and the iterate , e.g . µ ( x̃ , x ( K ) ) = KL ( πx̃ ‖ πx ( K ) ) . In actor-critic algorithms ( Sutton et al. , 1999 ) , the policy defines only part of the agent—the value function defines the other . Thus , we also consider a composite matching function over both policy and value function . Target Bootstrap We analyze conditions under which BMG guarantees performance improvements in Section 4 and find that the target should co-align with the gradient direction . Thus , in this paper we focus on gradient-based TBs and find that they perform well empirically . As with matching functions , this is a small subset of all possible choices ; we leave the exploration of other choices for future work .

Summary: The paper proposes *Bootstrapped Meta-Learning,* a new meta-learning algorithm for hyperparameter optimization. Drawing inspiration from temporal difference learning techniques in reinforcement learning, the meta-learner is asked to predict the result of additional unrolled steps of the optimization process, by minimizing a carefully selected distance to a target generated during training. This allows for longer meta-learning optimization horizons, without the need for differentiation through longer optimization trajectories. The method is tested for hyperparameter optimization for reinforcement learning, including learning the exploration hyperparameter for a behaviour policy, and in multi-task meta-learning.