Bootstrapped Meta-Learning
Authors: Sebastian Flennerhag
Yannick Schroecker
Tom Zahavy
Hado van Hasselt
David Silver
Satinder Singh
Venue: ICLR 2022 Oral (2022)
Keywords: meta-learning
meta-gradients
meta-reinforcement learning

Abstract: Meta-learning empowers artificial intelligence to increase its efficiency by learning how to learn. Unlocking this potential involves overcoming a challenging meta-optimisation problem. We propose an algorithm that tackles this problem by letting the meta-learner teach itself. The algorithm first bootstraps a target from the meta-learner, then optimises the meta-learner by minimising the distance to that target under a chosen (pseudo-)metric. Focusing on meta-learning with gradients, we establish conditions that guarantee performance improvements and show that metric can be used to control meta-optimisation. Meanwhile, the bootstrapping mechanism can extend the effective meta-learning horizon without requiring backpropagation through all updates. We achieve a new state-of-the art for model-free agents on the Atari ALE benchmark and demonstrate that it yields both performance and efficiency gains in multi-task meta-learning. Finally, we explore how bootstrapping opens up new possibilities and find that it can meta-learn efficient exploration in an epsilon-greedy Q-learning agent - without backpropagating through the update rule.

Published as a conference paper at ICLR 2022
BOOTSTRAPPED META-LEARNING
Sebastian Flennerhag
Yannick Schroecker DeepMind
Tom Zahavy DeepMind
DeepMind flennerhaa@@oogge..com
Hado van Hasselt DeepMind
David Silver DeepMind
Satinder Singh DeepMind
ABSTRACT
Meta-learning empowers artificial intelligence to increase its efficiency by learning how learn. Unlocking this potential involves overcoming a challenging meta- optimisation problem. We propose an algorithm that tackles this problem by letting the meta-learner teach itself. The algorithm first bootstraps target from the meta- learner, then optimises the meta-learner by minimising the distance to that target under a chosen (pseudo-) metrii... Focusing on meta-learning with gradients, we establish conditions that guarantee performance improvements and show that the metric can control mntaaoptimisstion. Meanwhile, the bootstrapping mechanism extend the effective meta-leariing horizon without requiring backpropagation through all updates. We achieve a new state-of-th art for model-free agents on the Atari ALE benchmark and demonstrate that it yields both performance and efficiency gains in multi-task meta-learning. Finally, we explore how bootstrapping opens up new possibilities and find that it can meta-learn efficient exploration in e-greedy Q-lcarning agent-withou backpropagating through the update rule.
INTRODUCTION
In standard machine learning problem, learner or agent learns a task by iteratively adjusting its parameters under a given update rule, such Stochastic Gradient Descent (SGD). Typically, the learner's update rule must be tuned manually. In contrast, humans learn seamlessly by relying on previous experiences to inform their learning processes (Spelke & Kinzler, 2007).
For (machine) learner to have the same capability, it must be able to learn its update rule (or such inductive biases). Meta-learning is one approach that learns (parts of) an update rule by applying it for some number of steps and then evaluating the resulting performance (Schmidhuber, 1987; Hinton & Plaut. 1987; Bengio et al., 1991). For instance, well-studied and often successful approach is to tune parameters
gradient- based update, either online during training on a single task (Bengio, 2000; Maclaurin al., 2015; Xu al., 2018: Zahavy al., 2020), or meta-learned over distribution of tasks (Finn et al., 2017; Rusu et al., 2019; Flennerhag et al., 2020; Jerfel et al., 2019; Denevi et al., 2019). More generally, the update rule can be an arbitrary parameterised function (Hochreiter et al., 2001; Andrychowicz et al., 2016; Kirsch et al., 2019; Oh et al., 2020), or the function itself can be meta-learned jointly with its parameters (Alet et al., 2020; Real et al., 2020).
Meta-learning is challenging because to evaluate an update rule, it must first be applied. This often leads high computational costs. As result most works optimise performance after K applications of the update rule and assume that this yields improved performance for the remainder of the learner's 's lifetime (Bengio al., 1991; Maclaurin et al., 2015; Metz et al., 2019). When this assumption fails, meta-learning suffers from short-horizon bias ( (Wu et al., 2018; Metz et al., 2019). Similarly, optimizing the learner's performance after K updates can fail to account for the process of learning, causing another form of myopia (Flennerhag al., 2019; Stadie et al., 2018; 8; Chen et al., 2016; Cao et al., 2019). Challenges in meta-optimisation have been observed to cause degraded lifetime performance (Lv et al., 2017; Wichrowska et al., 2017), collapsed exploration (Stadie et al., 2018; Chen et al., 2016), biased learner updates (Stadie et al., 2018; Zheng et al., 2018), and poor generalisation performance (Wu et al., 2018; Yin al., 2020; Triantafillor et al., 2020).

---

Published as a conference paper at ICLR 2022
We argue that defining the meta-learner's objective directly in terms of the learner's 's objective-ia -.e. the performance after K update steps -crettes two bottlenecks in meta-optimisation. The first bottleneck is curvature: the meta-objective is constrained to the same type of geometry as the learner; the second is myopia: the meta-objective is fundamentally limited to evaluating performance within the K-step horizon, but ignores future learning dynamics. Our goal is design an algorithm that removes these.
The algorithm relies on two main ideas. First, to mitigate myopia, we introduce the notion of bootstrapping a target from the meta-learner itself, a meta-bootstrup. that infuses information about learning dynamics in the objective. Second, to control curvature, we formulate the meta-objective in terms of minimising distance (or divergence) to the bootstrapped target, thereby controlling the meta-loss landscape. In this way, the meta-learner learns from its future self. This leads to a bootstrapping effect where improvements beget further improvements. We present a detailed formulation in Section 3; on a high level, as in previous works, we first unroll the meta-learned update rule for K steps to obtain the learner's new parameters. Whereas standard meta-objectives optimise the update rule with respect to (w.r.t)) the learner's performance under the new parameters, our proposed algorithm constructs the meta-objective in two steps:
1. It bootstraps a largel from the learner's 's new parameters. In this paper, we generate targets by continuing to update the learner's ' parameters- either under the meta-learned update rule another update rule- for some number of steps.
2. The learner's s new parameters-wwich are a function of the meta-learner's parameters- and the target are projected onto matching space. A simple example is Euclidean parameter space. To control curvature, we may choose a different (pseudo-)metric space. For instance, a common choice under probabilistic models is the Kullback-Leibler (KL) divergence.
The meta-learner is optimised by minimising distance to the bootstrapped target. We focus on gradient-basee based optimisation, but other optimisation routines are equally applicable. By optimising mneta-paametter in a well-behaved space, we can drastically reduce ill-conditioning and other phenomena that disrupt meta-optimisation. In particular, this form of Bootstrapped Gradenta (BMG) enables us to infuse information about future learning dynamics without increasing the number of update steps to backpropagate through. In effect, the meta-learner becomes its own teacher. We show that BMG can guarantee performance improvements (Theorem 1) and that this guarantee can be stronger than under standard meta- reea-ggaddeens (Corollary 1). Empirically, we find that BMG provides substantial performance improvements over standard meaa-ngad in various settings. We obtain a new tate--f-thee-ee result for model-free agents on Atari (Section 5.2) and improve upon MAML (Finn al., 2017) in the few-shot setting (Section 6). Finally, we demonstrate how BMG enables new forms of meta-learning, exemplified by meta-learning e-greedy exploration (Section 5.1).
RELATED WORK
Bootstrapping used here stems from temporal difference (TD) algorithms in reinforcement learning (RL) (Sutton, 1988). In these algorithms, an agent learns a value function by using its own future predictions targets. Bootstrapping has recently been introduced in the self-supervised setting (Guo et al., 2020; Grill et al., 2020). In this paper, we introduce the idea of bootstrapping in the context of meta-learning, where a meta-learner learns about an update rule by generating future targets from
Our approach to target matching is related to methods in multi-task meta-learning (Flennerhag et al., 2019; Nichol et al., 2018) that meta-learn an initialisation for SGD by minimising the Euclidean distance to task-optimal parameters. BMG generalise this concept by allowing for arbitrary meta- parameters, matching functions, and target bootstraps. It is further related the more general concept of seff-referential meta-leamnig (Schmidhuber, 1987; 1993), where the meta-learned update rule is used to optimise its own meta-objective.
Target matching under KL. divergences results in form of distillation (Hinton et al., 2015), where an online network (student) is encouraged to match a target network (teacher). In a typical setup, the target is either fixed (set of) expert(s) (Hinton et al., 2015; Rusu et al., 2015) or moving aggregation of current experts (Teh et al., 2017: Grill et al., 2020), whereas BMG bootstraps a target by following an update rule. Finally, BMG is loosely inspired by trust-region methods that introduce a distance function to regularize gradient updates (Pascanu& & Bengio, 2014; Schulman et al., 2015; Tomar et al., 2020; Hessel et al., 2021).
2

---

Published as a conference paper at ICLR 2022
3 BOOTSTRAPPED META-GRADIENT
We begin in the single-task setting and turn to multi-task meta-lcarning in Section 6. The learner's problem is to minimize a stochastic objective f(x) == E[((x: {)] over a data distri- bution p((). where Ç denotes a source data and E X' 1R= denotes the learner's pa- rameters, In RL, I is typically the (negative) expected value of a policy πxi in supervised learning, f may be the expected negative log- likelihood under probabilistic model Tx. We provide precise formulations in Sections 5 and 6.
(x, fx))
The meta-leamne's problem to learn an up- date rule y X' H x W + X' that updates the learner's s parameters by x (1) x + (xx h, w) given x E X, learning state h H. and peta-parameters
V x())
E W C TM= of the update rule. We make no assumptions on the update rule other than differentiability in w. As such, sp can be a recurrent neural network (Hochre- iter et al., 2001; Wang et al., 2016; Andrychow- icz et al., 2016) or gradient descent (Bengio, 2000; Maclaurin et al., 2015; Finn et al., 2017). The learning state contains any other data re- quired to compute the update: in black-box setting h contains an observation and the re- current state of the network; for braseent-based updates. h contains the (estimated) gradient of at x along with any auxiliary information; for instance, SGD is given by x(1) = vV (x() with h = (x), w = a E Rt.
(s, Txx (s))
x (K)
Figure 1: Bootstrapped Meta-Gradients.
The standard meta-gradient (MG) optimises pertamaasssssess w by taking K steps under ip and evaluating the resulting learner parameter vector under f. With a slight abuse of notation, let xc(x)()) denote the learner's parameters after K applications of şo starting from some (x, h, w), where (x, h) evolve according yp and the underlying data distribution. The MG update is defined by
w' w - (x()((w))
EER+.
Extensions involve averaging the performance over all iterates x(.,..., (.....((K)) (K) (Andrychowicz et al., 2016: Chen et al., 2016; Antoniou al., 2019) or using validation data in the meta-objective (Bengio et al., 1991; Maclaurin et al., 2015; Finn et al., 2017; Xu et al., 2018). We observe two bottlenecks in the meta-objective in Eq. 1. First, the meta-objective subject to the same curvature as the learner. Thus if f is ill- cllcioniii,,,,, so will the meta-objective be. Second, the meta-objective is only able to evaluate the meta-learner on dynamics up to the Kthh step, but ignores effects of future updates.
To tackle myopia, we introduce Target Bootstrap (TB) { X + X' that maps the meta-learner's out- put (K) into a bootstrapped target & = (x)))). We focus on TBs that unroll op further L -- steps before taking final gradient step on f, with targets of the form & = (K+L-1) -o V(f(((((LL))) This TB encourages the meta-learner to reach future states on its trajectory faster while nudging the trajectory in a descent direction. Crucially, regardless of the bootstrapping strategy, we do not backpropagate through the target. Akin temporal difference learning RL (Sutton, 1988), the target is a fixed goal that the leta-learner should try to produce within the K K-step budget.
Finally, to improve the meta-optimisation landscape, we introduce matching function μ : X' x X R2 that measures the (dis)similarity between the meta-learner's output, (w)), and the target, x, in matching space defined by μ (see Figure 1). Taken together, the BMG update is defined by
W w -3 VV 11 (x, (x,x(0 (w))
B E IR,,
(2)
3

---

Published as a conference paper at ICLR 2022
where the gradient is with respect to the second argument of μ. Thus, BMG describes family of algorithms based on the choice of matching function μl and TB &. In particular, MG is a special case of BMG under matching function p(x, x = || - (()) I13 and TB {(x(K)) = x(k) - - - (x)(x()))
since the bootstrapped meta-gradien reduces to the standard veta-gradient:
V x - x () (w) || = 2D - x
(x) = DV&f (x(*)) = Var (x))
(3)
where D denotes the (transposed) Jacobian of x (x((w)) For other matching functions and target strategies, BMG produces different meta-updates compared to MG. We discuss these choices below.
Matching Function Of primary concern to are models that output probabilistic distribution, xx. A common pseudo-metric over a space of probability distributions is the Kullback-Leibler (KL) divergence. For instance, Natural Gradients (Amari, 1998) point in the direction of steepest descent under the Kl--divergence, often approximated through a KL-regglarization term (Pascanu & Bengio, 2014). Kl--divergences also arise naturally in RL algorithms (Kakade, 2001; Schulman al., 2015; 2017; Abdolmaleki et al.. 2018). Hence, a natural starting point is to consider KL-divergences between the target and the iterate, e.g. μ((x, x x)) = KL (T& || Tx(K)). ). In actor-critic algorithms (Sutton et al., 1999), the policy defines only part of the agent- the value function defines the other.
Thus, we also consider a composite matching function over both policy and value function.
Target Bootstrap We analyze conditions under which BMG guarantees performance improvements in Section 4 and find that the target should co-align with the gradient direction. Thus, in this paper we focus on pradient-based TBs and find that they perform well empirically. As with matching functions, this is a small subset of all possible choices; we leave the exploration of other choices for future work.
4 PERFORMANCE GUARANTEES
In this analysis, we restrict attention to the noise-less setting (true expectations). In this setting, we ask three questions: (1) what local performance guarantees are provided by MG? (2) What performance guarantees can BMG provide? (3) How do these guarantees relate to each other? To answer these questions, we analyse how the performance around (xx() changes by updating w either under standard meta- seeta-gradient (Eq. 1) or bootstrapped meta- meta-graddents (Eq. 2).
First, consider improvements under the MG update. In online optimisation, the MG update can achieve strong convergence guarantees if the problem is well-behaved (van Erven & Koolen, 2016), with similar guarantees in the multi-task setting (Balcan et al., 2019; Khodak et al., 2019; Denevi et al., 2019). A central component of these results is that the MG update guarantees local improvement in the objective. Lemma below presents this result in our setting, with the following notation:
let || = V (u, A uu) for any square real matrix A. Let GT D'D E 1a=x*=, with = law (w)) E Rn. Note that (x (K) (w))
DV=,(x(()).
Lemma 1 (MG Descent). Let w' be given by Eq. 1. For B sufficiently small, f f(x()(w))) f(x(K( = V (x())x( + +(82) <.
We defer all proofs to Appendix A. Lemma relates the gains obtained under standard meta-gradients to the local gradient norm of the objective. Because the meta-objective is given by f. the MG update is not freee-free (c.f. Schraudolph, 1999), nor invariant to re-parameterisation. If is highly nin-lrnear, the meta- getaiggrd can vary widely, preventing efficient performance improvement. Next, we turn to BMG, where we assume μ is differentiable and convex, with 0 being its minimum.
Theorem (BMG Descent). Let be given by Eq. for some TB &. The BMG update satisfies
(n(x,x(* (K) -a g) 1(xx x(xx x()) + o(3(a + B)). For (a, 3) sufficiently small, there exists infinitely many & for which f( ()) -f(( (x(xx In particular, {(x( = (K) -OGT g yields improvements
{( (0x(0 - f( (x( = 3
f {(x - f (xx()(w)) = 11xxx (1) + +3((a +)) 0.
a
This is not an optimal rate; there exists infinitely many TBs that yield greater improvements.
4

---

Published as a conference paper at ICLR 2022
Theorem 1 portrays the inherent trade-off in BMG; targets should align with the local direction of steepest descent, but provide as much learning signal as possible. Importantly, this theorem also establishes that directly controls for curvature improvements
expressed in terms of j.. While the TB E (x) x(k) -aG" g yields performance improvements that are proportional to the meta-loss itself, larger improvements are possible by choosing a TB that carries greater learning signal (by increasing yu(8, x()))). To demonstrate that BMG can guarantee larger improvements to the update rule than MG, we consider the TB 5G with the (squared) Euclidean norm. Let IIV (x(K)) le/l|""V (x(x)) ll2 denote the gradient norm ratio.
= 11.113 and x = (x(x(()). Let be given by Eq. 1 and w be given by Eq. 2. For B sufficiently small, f( [x (w)) S S( (w))) strictly if GGT + G and G2 V=](((k)) = O.
Corollary 1. Let p=
Discussion Our analysis focuses on an arbitrary (albeit noiseless) objective f and establishes that BMG can guarantee improved performance under a variety of TBs. We further show that BMG can yield larger local improvements than MG. To identify optimal TBs, further assumptions are required on I and Jr, but given these Theorem 1 can serve as a starting point for more specialised analysis. Empirically, we find that taking L steps on the meta-learned update with an final gradient step on the objective performs well. Theorem 1 exposes trade-off for targets that are "far" away. Empirically, we observe clear benefits from bootstraps that unroll the meta-leamer for several steps before taking gradient step on f: exploring other forms of bootstraps is exciting area for future research.
5 REINFORCEMENT LEARNING
We consider typical reinforcement learning problem, modelled as an MDP . = (S,A. 7)- Given an initial state Sy E S, at each time step E N, the agent takes an action a. ~ Tx (a ss) from a policy S x A + [0, 1] parameterised by x. The agent obtains a reward r++1 R(s,, ar, 81+1) based on the transition 8 ~ P( St, ar). The action-value of the agent's policy given a state So and action as is given by Qx 5o. ag) E o 7'T141 So, ag, Tx] under discount rate E [0, 1). The corresponding value of policy Tx is given by Vx == aag ss) [,)](Qx [-)]
The agent's problem is to learn policy that maximises the value given an expectation over So, defined either by an initial state distribution in the episodic setting (e.g. Atari, Section 5.2) or the stationary state-vin visitaiistttiion distribution under the policy in the non-episodic setting (Section 5.1). Central to RL is the notion of such that [V[] Z [. A common policy-improvenen step is arg maxx I. (awan) (Qx a)].
which takes a current policy Tx and constructs a new policy Tx
Most works in meta-RI. rely on actor-critic algorithms (Sutton et al., 1999). These treat the above popiey-impoovement step as an optimisation problem and estimate a policy-gradien (Williams & Peng, 1991; Sutton et al., 1999) to optimise x. To estimate Vx, these introduce critic u, that is jointly trained with the policy. The policy is optimised under the current estimate of its value function, while the critic is tracking the value function by minimizing Temporal- -Dffeere-e (TD) error. Given a rollout т = (So, ag: 71, S1,....... .r....,,TT, rr,sr)) the objective is given by f(x, z) epa (x) + EEN (x) + ETD (ZD) ePG: EEN, ETD R.,, where
LEN(X) ΣΣ
ax st) log (x st).
1 Σ (a(m)) ve(si)) (TD(Z) 2
LET aEA
LET
(4)
(ro(x) = Σ
Pe log n x st) (c! Uz (s(())
LET
where Pr denotes an importance weight and (! denotes an n-step bootstrap target. Its form depends on the algorithm; in Section 5.1, we generate rollouts from Rx (on-policy), in which case Pt and ((()) = - (n-1) 7'rets+1 + 7 (8t+n) VI, where z denotes fixed (non-differentiabll) parameters.
In the off pol-py setting (Section 5.2), p corrects for sampling bias and c(") is similarly adjusted.
5

---

Published as a conference paper at ICLR 2022
Bootstrapped Meta-gradients
Meta-gradients
Retarn after 10M steps (Millions)
Row./Step
1.5
0.15
0.05
1.0
0.5
Eutropp Rate
0.0
K
19.0
0.0
K+I
64
Roster
Couter
Env Steps Into Cycle
Figure 2: Non-stationary grid-world (Section 5.1). Left: Comparison of total returns under an actor-critic agent over 50 seeds. Right: Learned antropy-regularizate schedules. The figure depicts the average regularization weight (e) over task-cycles at 6M steps in the environment.
5.1 A NON-STATIONARY AND NON-EPISODIC GRID WORLD
We begin with a tabular grid-worid world with two items to collect. Once item is collected, it is randomly re-spawned. One item yields reward +1 and the other a reward of -1. The reward flipped every 100,000 steps. To succeed, memory-less agent must efficiently re-explore the environment. We study an on-policy actor-critic agent with 6PG éTp As baseline, we tune a fixed entropy-rate weight = CEN- We compare against agents that meta-leam online. For MG, we use the actor-critic loss as meta-objective (e fixed), as per Eq. 1. The setup is described full in Appendix B.1
Row./Step
0.15
0.05
0.4
0.2
0.1
BMG Our primary focus is on the effect of bootstrapping. Be- cause this setup is fully online, we can generate targets using the most recent L-1 parameter updates and a final agent param- eter update using с = 0. Hence, the computational complexity of BMG is constant in L under this implementation (see Appendix B.2). We define the matching function as the KL-divergence between (K) and the target, / (()),,((K((())))) (w)) = KT. (лx xx()))
Eav Stegs into Cycle
Figure 3: BMG s-greedy explo- ration under a ((A)-agent.
Figure presents our main findings. Both MG and BMG learn adaptive entropy-rate schedules that outperform the baseline. However, MG fails if 0 in the meta-objective, as it becomes overly greedy (Figure 9). MG shows no clear benefit of longer meta-learning horizons, indicating that myopia stems from the objective itself. In contrast, BMG exhibits greater adaptive capacity and is able to utilise greater meta-lcarning horizons. Too short horizons induce myopia, whereas too long prevent efficient adaptation. For given horizon, increasing K is uniformly beneficial. Finally, find that BMG outperforms MG for a given horizon without backpropagating through all updates. For instance, for K = 8, BMG outperforms MG with K and 7. Our ablation studies (Appendix B.2) show that increasing the target bootstrap length counters myopia; however, using the meta-learned update rule for all L steps can derail meta-optimization.
Next, we consider a new form of meta-learning: learning e-greedy exploration in Q(A)-agent (precise formulation in Appendix B.3). While the parameter has similar effect entropy- regularization, 5 is parameter applied in the behaviour-policy while acting. As it does not feature in the loss function, it not readily optimized by existing meta- preaiggrddeet approaches. contrast, BMG can be implemented by matching the policy derived from a target action-value function, precisely as the actor-critic case. An implication is that BMG can meta-learn without backpropagating through the update rule. Significantly, this opens up meta-learning (parts of) the behaviour policy, which is hard to achieve the MG setup as the behaviour policy is not used in the update rule. Figure 3 shows that meta- leeanlee e-greedy exploration in this environment significantly outperforms the best fixed found by paper-prrreeter tuning. As the actor-critic case, we find that BMG responds positively to longer meta-learning horizons (larger L);; see Appendix B.3, Figure 12 for detailed results.

---

Published as a conference paper at ICLR 2022
JIMI
BMG V. BMG KI. V BMG KI.
Relative perfrmmnce
STACK*
431%, 192]
STACK
Median buman
10-
1-
absolute 0.5
50M
100M
150M
200M
Learbing trames
Figure 4: Human- normalized score across the 57 games Atari ALE. Left: per-game difference in score between BMG and our implementation of STACX* at 200M frames, Right: Median scores over learning compared to published baselines. Shading depict standard deviation across 3 seeds.
5.2 ATARI
High-performing RL agents tend rely on distributed learning systems to improve data efficiency (Kapturowski et al., 2018; Espeholt et al., 2018). This presents serious challenges for meta-learning as the policy gradient becomes noisy and volatile due to off-policy estimation (Xu et al., 2018; Zahavy et al., 2020). Theorem suggests that BMG can be particularly effective in this setting under the appropriate distance function. To test these predictions, we adapt the Self-Tuning Actor-Critic (STACX; Zahavy et al., 2020) to meta-learn under BMG on the 57 environments in the Atari Arcade Learning Environment (ALE; Bellemare al., 2013).
Protocol We follow the original IMPALA setup (Espeholt et al., 2018), but we do not downsample or gray-scale inputs. Following the literature, we train for 200 million frames and evaluate agent performance by median Human Normalized Score (HNS) across 3 seeds (Espeholt et al., 2018; Xu et al., 2018; Zahavy et al., 2020).
STACX The IMPALA actor-critic agent runs multiple actors asynchronously to generate experience for a centralized learner. The learner uses truncated importance sampling to correct for off-policy data in the actor-critic update, which adjusts P and V in Eq. 4. The STACX agent (Zahavy et al., 2020) is state-of-the-ar meta-RL agent. It builds on IMPALA in two ways: (1) it introduces auxiliary tasks in the form of additional objectives that differ only in their hyper-parameters; (2) it meta-learns the thyper-parameter
of each loss function (main and auxiliary). Meta-parrmeeters are given by w = (r", GGG GEN EiD A1 a²) f where À and a are prprrmprraetters of the importance weighting mechanism and n 2 denotes the number of auxiliary tasks. STACX uses the IMPALA objective as the meta-objective with K 1. See Appendix C for complete description.
BMG We conduct ceteris- pariibs-paibus comparisons that only alter the meta-objective: agent parameter updates are identical to those in STACX. When L = 1, the target takes gradient step on the original IMPALA loss, and hence the only difference the form of the meta-objective; they both use the same data and gradient information. For T. > 1, the first T, 1 steps bootstrap from the meta-learned update rule itself. To avoid overfitting, each of the L - 1 steps use separate replay data; this extra data is not used anywhere else. To understand matching functions,
test policy matching and value matching. Policy matching is defined by #xx,, (k) (w)) = KL (π& || Ty(n); we also test symmetric KL-divergence (KL-S)). Value matching is defined by μ(Z, (1) (w)) = [B[ (2.a)) )]].
Figure 4 presents our main comparison. BMG with I. and maatchgg (KL) obtains median HNS of -500%. compared to -350% for STACX. Recall that for L = 1, BMG uses the same data to compute agent parameter update, target update, and matching loss; hence this is an apples-to-apples comparison. Using both policy matching and value matching (with 0.25 weight on the latter) further improves the score to -520% and outperforms STACX across almost all 57 games, with few minor exceptions (left panel, Figure 4). These results are obtained without tuning paraeeeasseeters for BMG. Finally, extending the meta-learning horizon by setting = 4 and adjusting gradient clipping from 3 to .2 obtains score of -610%.
7

---

Published as a conference paper at ICLR 2022
0.40
K &
40
0.35
KL-S
5.5
Eblcode Betum (10000
0.30
5.0
Deasivv
0.25
-00
0.20
4.5
0.15
0.10
0.05
50b
RMS
RMS
RMS
RMS
KL
KL &
0.00
Normalized mean spisodo retum
-2
2
Figure 5: Ablations on Atari. Left: human normalized score decomposition of w.r.t. optimizer (SGD, RMS), matching function (L2, KL, KL & V), and bootstrap steps (L). BMG with (SGD, L2, -= 1) is equivalent to STACX. Center: episode return on Ms Pacman for different L. Right: distribution of episode returns over all 57 games, normalized per-game by mean and standard deviation. All results are reported between 190-200M frames over 3 independent seeds.
In Figure 5, we turn ablations. In the left-panel, we deconstruct BMG into STACX (i.e., MG) and compare performances. We find that roughly 45% of the performance gains comes from curvature correction (given by using RMSProp the target bootstrap). The matching function can further control curvature to obtain performance improvements, accounting for roughly 25%. Finally, increasing L, thereby reducing myopia, accounts for about 30% Comparing the cosine similarity between consequtive meta-gradients, we find that BMG improves upon STACX by two orders magnitude. Detailed ablations in Appendix C.I.
performance improvement.
The center panel of Figure 5 provides deep-dive in the effect of increasing the meta-learning horizon (L > 1) in Ms Pacman. Performance uniformly increasing in L, providing further support that BMG increase the effective meta-horizon without increasing the number of update steps to backpropagate through. A more in-depth analysis Appendix C.3 reveals that K more sensitive to curvature and the quality of data. However, bootstrapping only from the meta-leamnrr for all L steps can lead to degeneracy (Appendix C.2, Figure 14). In terms of replay (Appendix C.2), while standard MG degrades with more replay, BMG benefits from more replay in the target bootstrap.
The right panel of Figure studies the effect of the matching function. Overall, joint policy and value matching exhibits best performance. In contrast to recent work (Tomar al., 2020; Hessel al., 2021), we do not find that reversing the KL-direction is beneficial. Using only value-matching results worse performance, as it does not optimise for efficient policy improvements. Finally, we conduct detailed analysis of scalability in Appendix C.4. While BMG is 20% slower for K 1, = 1 due to target bootstrap, 200% faster when MG uses K and BMG uses K 1, 3.
MULTI-TASK FEW-SHOT LEARNING
Multi-task meta-learning introduces expectation over task objectives. BMG is applied by comput- ing task-specific bootstrap targets, with the meta- gradient being the expectation over task-specific matching losses. For general multi-task task formulation, classification paradigm. Let fo X' - IR. denote the negative log-likeliln bigllikeldhood loss on some data 'D A task is defined pair of datasets (Dr. D',), where DD is training set and D' is validation set. In the M M-shot-N-wa setting, each task has N classes and DD contains M observations per class.
Appendix D; here we focus the few-shot
The goal of this experiment is study how the BMG objective behaves in the multi-task setting. For this purpose, we focus on the canonical MAML setup (Finn et al., 2017), which meta-learns an initialisation (0) for SGD that is shared across task distribution p(T). Adaptation is defined by x/ 16 +a V.D (xf* (+-)), with a E R, fixed. The meta-objective the validation loss in expectation the task distribution: Eifn (x (K) (w))]. Several works have extended this setup by altering the update rule (ç) (Lee & Choi, 2018; 88; Zintgraf et al., 2019; Park & Oliva, 2019; Flennerhag al., 2020). As our focus is the meta-objective, focus comparisons with MAMI..
8

---

Published as a conference paper at ICLR 2022
65
BMG MG
25
1
BMG
Meta-training time (h)
1
MG
Mete-esst accuracy
2.0
Meta-tost acouracy
66.0
64-
15
64.5
63
10-
BMG
63.5
MG
100
200
300
62
63
64
65
10
Meta-train batches (thousands)
accuracy (%)
Figure 6: Minilmagenet 5-way-3-shot meta-test performance. Left: performance as a function of meta-training batches. Center: performance as a function of wall-clock time. Right: best reported performance under each K. Error bars depict standard deviation across seeds.
BMG For each task, target ž, bootstrapped by taking L. SGD steps from x/K) using validation data. The BMG objective is the expected distance, Blu(X,,
x² xx {)))), The KL-divergence as matching function has an interesting connection to MG. The target x, can be seen as an "expert" on task so that BMG is a form of distillation (Hinton et al., 2015). The log-likelihood loss used by MG is also KL divergence, but w.r.t. "cold" expert that places all mass on the true label. Raising the temperature in the target can allow BMG to transfer more information (Hinton & Plaut, 1987).
Setup We use the Minilmagenet benchmark (Vinyals et al., 2016) and study two forms efficiency: for data efficiency, we compare meta-test performance as function of the number of meta-training batches; for computational efficiency, we compare meta-test performance as a function of training time. To reflect what each method would achieve for given computational budget, meta-test performance for the byper-parameter configuration with best meta- meta- validation performance. For MG, we tune the meta-learning rate 3 {10-3 10-4), K E {1,5, 10), and options first-order approximations (FOMAML; Finn et al., 2017) or (ANIL; Raghu et al., 2020)). For BMG, we tune 3 (10-5 10-4), K (1,5). well L 5, 10), and the direction of the KL.
report
The left panel of Figure 6 presents results on data efficiency. For few meta-updates, MG and BMG are on par. For 50 000 meta-updates and beyond, BMG achieves strictly superior performance, with the performance delta increasing over meta-updates. The central panel presents results on computational efficiency; we plot the time required reach given meta-test performance. This describes the relationship between performance and computational complexity. We find BMG exhibits better scaling properties, reaching the best performance of MG approximately half the time. Finally, in the right panel, we study the effect of varying K. BMG achieves higher performance for both K = 1 and 5. We allow MG to also use conduct analysis of the impact BMG has curvature and meta-gradien variance in Appendix D.3. To summarise, we find that BMG significantly improves upon the MG meta-objective, both in terms of data efficiency, computational efficiency, and final performance.
10, but this did not yield any significant gains. We
7 CONCLUSION
In this paper, have put forth the notion that efficient meta-learning does not require the meta- objective to be expressed directly in terms of the learner's 's objective. Instead, we present an alternative approach that relies on having the meta-leamnr match desired target. Here, we bootstrap from the meta-learned update rule itself to produce future targets. While using the meta-learned update rule as the bootstrap allows for open-ended meta-learning process, some grounding is necessary. As an instance of this approach, we study bootstrapped meta-gradients, which can guarantee performance improvements under appropriate choices of targets and matching functions that can be larger than those of standard meta- geada-gadssetss. Empirically, we observe substantial improvements on Atari and achieve a new ofte-offthh-art while obtaining significant efficiency gains in a multi-task meta- learning setting. We explore new possibilities afforded by the target- mratet-matchingg nature of the algorithm and demonstrate that it can learn to explore in an e-greedy Q-learning agent.

---

Published as a conference paper at ICLR 2022
REFERENCES
Abbas Abdolmalcki, Jost Tobias Springenberg. Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum Posteriori Policy Optimisation. International Conference on Learning Representations, 2018.
Ferran Alet, Martin F. Schneider, Tomas Lozano-Perez, and Leslie Pack Kaelbling, Meta- Meta--earnnng Curiosity Algorithms. International Conference on Learning Representations, 2020.
Shun-Ichi Amari. Natural Gradient Works Efficiently in Learning. Neural computation, 10(2): 251-276, 1998.
Marein Andrychowiez, Misha Denil, Sergio Gómez, Matthew Hoffman, David Pfau, Tom Schaul, and Nando de Freitas. Learning to Learn by Gradient Descent by Gradient Descent. In Advances in Neural Information Processing Systems, 2016.
Antreas Antoniou, Harrison Edwards, and Amos J. Storkey. How to Train Your MAML. In International Conference Learning Representations, 2019.
Poria-Florina Balcan, Mikhail Khodak, and Ameet Talwalkar. Provable Guarantees for Gradient- Based Meta-Learning. In International Conference on Machine Learning, 2019.
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The Arcade Learning Environment: An Evaluation Platform for General Agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
Yoshua Bengio. Gradient-Based Optimization of Hyperparameters. Neural computation, 12(8): 1889-1900, 2000.
Yoshua Bengio, Samy Bengio, and Jocelyn Cloutier. Learning Synaptic Learning Rule. Université de Montréal, Département d d'informatique et de recherche opérationnelle, 1991.
Cao, T Chen, Z Wang, and Y Shen. Learning to Optimize in Swarms. Advances in Neural Information Processing Systems, 2019.
Yutian Chen, Matthew W. Hoffman, Sergio Gomez Colmenarejo, Misha Denil, Timothy P. Lillicrap, and Nando de Freitas. Learning to learn for Global Optimization of Black Box Functions. In Advances in Neural Information Processing Systems, 2016.
Giulia Denevi, Dimitris Stamos, Carlo Ciliberto, and Massimiliano Pontil. Online-Within-Onil Meta-Learning. In Advances in Neural Information Processing Systems, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: Large-Scale Hierarchical Image Database. In Computer Vision and Pattern Recognition, 2009.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable Distributed Deep-RL with Importance Weighted Actor-Leamer Architectures. In International Conference Machine Learning, 2018
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. International Conference on Machine Learning, 2017.
Sebastian Flennerhag, Pablo G. Moreno, Neil D. Lawrence, and Andreas Damianou. Transferring Knowledge across Learning Processes. In International Conference Learning Representations, 2019.
Sebastian Flennerhag, Andrei A. Rusu, Razvan Pascanu, Francesco Visin, Hujun Yin, and Raia Hadsell. Meta-Learning with Warped Gradient Descent. In International Conference on Learning Representations, 2020.
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas L. Griffiths. Recasting Gradient-Based Meta-Leaming as Hierarchical Bayes. In International Conference on Learning Representations, 2018
10

---

Published as a conference paper at ICLR 2022
Jean-Bastir Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning. In Advances in Neural Information Processing Systems, 2020.
Zhaohan Daniel Guo, Bernardo Avila Pires, Bilal Piot, jean-Bastien Grill, Florent Altché, Rémi Munos, and Mohammad Gheshlaghi Azar. Bootstrap Latent- Predictive Representations for Multi- task Reinforcement Learning. In International Conference Machine Learning, 2020.
Matteo Hessel, Ivo Danihelka, Fabio Viola, Arthur Guez, Simon Schmitt, Laurent Sifre, Theo- phane Weber, David Silver, and Hado van Hasselt. Muesli: Combining Improvements in Policy Optimization. arXiv preprint arXiv:2104.06/553 2021.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the Knowledge in Neural Network. arXiv 2015,
preprint arXiv:1503.022
Geoffrey E. Hinton and David C. Plaut. Using Fast Weights to Deblur Old Memories. In Cognitive Science Society, 1987.
Sepp Hochreiter, A. Steven Younger, and Peter R. Conwell. Learning To Learn Using Gradient Descent. In International Conference on Artificial Neural Networks, 2001.
Ghassen Jerfel, Erin Grant, Tom Griffiths, and Katherine Heller. Reconciling Meta- Deta-Learning and Continual Learning with Online Mixtures of Tasks. In Advances in Neural Information Processing Systems, 2019.
Norman P Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, and AI Borchers. In-Datacenter Performance Analysis of Tensor Processing Unit. In International Symposium on Computer Architecture, 2017.
Sham M Kakade. A Natural Policy Gradient. In Advances in Neural Information Processing Systems, 2001.
Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. Recurrent Experience Replay in Distributed Reinforcement Learning. In International Conference Learning Representations, 2018.
Mikhail Khodak, Maria-Florina F Balcan, and Ameet S Talwalkar. Adaptive Gradient-Band Based Meta- Learning Methods. Advances in Neural Information Processing Systems, 2019.
Louis Kirsch, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Improving Generalization in Meta Reinforcement Learning Using Learned Objectives. arXiv preprint arXiv: 19tp.0490009 2019.
Yoonho Lee and Seungjin Choi. Gradient- Gradeeen-BBased Meta-Learning with Learned Layerwise Metric and
Subspace. In International Conference on Machine Learning. 2018.
Kaifeng Lv. Shunhua Jiang, and Jian Li. Learning Gradient Descent: Better Generalization and Longer Horizons. In International Conference on Machine Learning, 2017.
Marlos C. Machado, Mare G. Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents. Journal of Artificial Intelligence Research, 61:523 ::22255-2,, 2018.
Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient-Based Hyperparameter Opti- mization Through Reversible Learning. In International conference on machine learning, pp. 2113-2122. PMLR, 2015.
Luke Metz, Niru Maheswaranathan, Jeremy Nixon, Daniel Freeman, and Jascha Sohl-Dickstein. Understanding and Correcting Pathologies in the Training of Leamned Optimizers. In International Conference Machine Learning, 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing Atari with Deep Reinforcement Learning. arXiv preprint 1312v=1312.5002. 2013.
11

---

Published as a conference paper at ICLR 2022
Alex Nichol, Joshua Achiam, and John Schulman. On First-Order Meta-Learning Algorithms. arXiv preprint ArXiv:/803.02999 2018.
Junhyuk Oh, Matteo Hessel, Wojciech M Czarnecki, Zhongwen Xu, Hado P van Hasselt, Satinder Singh, and David Silver. Discovering Reinforcement Learning Algorithms. In Advances in Neural Information Processing Systems, volume 33, 2020.
Eunbyung Park and Junier B Oliva. Meta-Curvature. In Advances in Neural Information Processing Systems, 2019.
Razvan Pascanu and Yoshua Bengio. Revisiting Natural Gradient for Deep Networks. In International Conference on Learning Representations, 2014.
Jing Peng and Ronald J. Williams. Incremental Multi-Step Q-Learning. In International Conference on Machine Learning, 1994.
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML. In International Conference on Learning Representations, 2020.
Sachin Ravi and Hugo Larochelle. Optimization as a Model for Few-Shot Learning. International Conference on Learning Representations, 2017.
Esteban Real, Chen Liang, David R. So, and Quoe V. Le. AutoML-Zero: Evolving Machine Learning Algorithms From Scratch. In International Conference on Machine Learning, 2020.
Andrei Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirk- patrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy Distillation. arXiv preprint arXiv:1511.06295 2015.
Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osin- dero, and Raia Hadsell. Meta-Learning with Latent Embedding Optimization. In International Conference on Learning Representations, 2019.
Jürgen Schmidhuber. Evolutionary Principles in Self-Referential Learning. PhD thesis, Technische Universität München, 1987.
Jürgen Schmidhuber. A 'self-referential" weight matrix. In International Conference on Artificial Neural Networks, pp. 446-450. Springer, 1993.
Simon Schmitt, Matteo Hessel, and Karen Simonyan. Off-Policy Actor-Critic with Shared Experience Replay. In International Conference on Machine Learning, 2020.
Nicol N. Schraudolph. Local Gain Adaptation in Stochastic Gradient Descent. In International Conference on Artificial Neural Networks, 1999.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust Region Policy Optimization. In International Conference on Machine Learning. 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms. arXiv preprint
2017.
Elizabeth S. Spelke and Katherine D Kinzler. Core Knowledge. Developmental science, (0(1):89-96, 2007.
Bradly C. Stadie, Ge Yang, Rein Houthooft, Xi Chen, Yan Duan, Yuhuai Wu, Pieter Abbeel, and Ilya Sutskever. Some Considerations on Learning to Explore via Meta-Reiifforeement Learning. In Advances in Neural Information Processing Systems, 2018.
Richard S. Sutton. Learning to Predict by the Methods of Temporal Differences. Machine learning. 3(1):9-44 1988.
Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy Gradient Methods for Reinforcement Learning with Function Approximation. In Advances in Neural Information Processing Systems, volume 99, 1999.
12

---

Published as a conference paper at ICLR 2022
Yee Whye Teh, Victor Bapst. Wojciech M Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust Multitask Reinforcement Learning. In Advances in Neural Information Processing Systems, 2017.
Manan Tomar, Lior Shani, Yonathan Efroni, and Mohammad Ghavamzadeh. Mirror Descent Policy Optimization. arXiv preprini arXiv:2005.090 09814, 2020.
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, and Pierre-Antoine Manzagol. Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples. International Conference on Learning Representations, 2020.
Tim van Erven and Wouter M Koolen. MetaGrad: Multiple Learning Rates in Online Learning. In Advances in Neural Information Processing Systems, 2016.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap. Koray Kavukcuoglu, and Daan Wierstra. Match- ing Networks for One Shot Learning. In Advances in Neural Information Processing Systems, 2016.
Jane X. Wang, 7eb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel 7. Leibo, Rémi Munos, Charles Blundell, Dharshan Kumaran, and Matthew Botvinick. Learning to Reinforcement Learn. In Annual Meeting of the Cognitive Science Society, 2016.
Olga Wichrowska, Niru Maheswaranathan, Matthew W. Hoffman, Sergio Gómez Colmenarejo, Misha Denil, Nando de Freitas, and Jascha Sohl-Dickstein. Learned Optimizers that Scale and Generalize. In International Conference on Machine Learning, 2017.
Ronald J Williams and Jing Peng. Function Optimization using Connectionist Reinforcement Learning Algorithms. Connection Science, 3(3):411-268 1991.
Yuhuai Wu, Mengye Ren, Renjie Liao, and Roger B. Grosse. Understanding Short-Horizon Bias in
Stochastic DettaOptimizzaiin. In International Conference Learning Representations, 2018.
Zhongwen Xu, Hado P. van Hasselt, and David Silver. Advances in Neural Information Processing Systems, 2018.
Reinforcement Learning. In
Mingzhang Yin, George Tucker, Mingyuan Zhou, Sergey Levine, and Chelsea Finn. Meta-Learning without Memorization. In International Conference on Learning Representations, 2020.
Tom Zahavy, Zhongwen Xu, Vivek Veeriah, Matteo Hessel, Junhyuk Oh, Hado P van Hasselt, David Silver, and Satinder Singh. A Self-Tuning Actor-Critic Algorithm. Advances in Neural Information Processing Systems, 33, 2020.
Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On Learning Intrinsic Rewards for Policy Gradient Methods. Advances in Neural Information Processing Systems, 2018
Luisa Zintgraf, Kyriacos Shiarli, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast Context Adaptation via Meta-Leaming, In International Conference on Machine Learning, 2019.
13

---

Published as a conference paper at ICLR 2022
Bootstrapped Meta-Learning: Appendix
CONTENTS
Appendix A: proofs accompanying Section 4.
Appendix B: non-stationary Grid-World (Section 5.1).
Appendix C: ALE Atari (Section 5.2).
Appendix D: Multi-task meta-lcarning, Few-Shot Learning on Minilmagenet (Section 6).
A
PROOFS
This section provides complete proof' for the results in Section 4. Throughout, we assume that (x(") (0) w) is given and write x = x x()), h = h() We assume that h evolves according to some ,x())) into new learner state h (*), including any sampling of data (c.f. Section 3). Recall that we restrict attention to the noiseless
process that maps history (²)) = (x( h0) ((k-)) h (*-1),
x())
setting, and hence updates are considered in expectation. We define the map x(K) (w)
(w) by
(1) = x + p x (0)
h (0) w)
x(2) = xx( + + x (1) h w)
(k) = x (K-1) (x1xx(11), (k-1)),
The derivative 9 x(K)(w) differentiates through each step this process (Hochreiter et al., 2001).
3w As previously stated, we assume f is Lipschitz and that x is Lipschitz w.rt. w. We are now in position to prove results from the main text. We re-state them for convenience.
Lemma 1 (MG Descent). Let w' be given by Eq. For sufficiently small, f( f(x()(ww))
((x(() (w) = - (x() + 0(82) <0.
(w)). D g. Under Eq. 1, we find w' = w -BDg. By first-order Taylor Series Expansion of f around (x, h w') with respect
Proof. Define = V fx (x)))((((w),, The meta meta-gradient at (x, h, w) is given by vx (w)))
f(x(1((w))) = ((x(() + (Dg. w' w) + +0 (1))r)))))))) = (w)) 3(D g, Dg) + o(32) g (l)
= -f(x((((w)) - BB l + + 1l
with || ll 2 by virtue of positive semi-definiteness of G. Hence, for 32 small the residual vanishes and the conclusion follows.
Theorem (BMG Descent). Let v be given by Eq. for some TB &. The BMG update satisfies
1( (() - [(x(10
(1(x,x o. g) - -((x x x(K)) + o(3(a + B)).
For (a. sufficiently small, there exists infinitely many f which f( f(x (k) - xc (w)) O. In particular, {(x(K)) = x(K) -OGT g yields improvements
[( (K)))((W) (Kx())
_xx( o(3(a + B)) <
a
This is not an optimal rate; there exists infinitely many TBs that yield greater improvements.
14

---

Published as a conference paper at ICLR 2022
Proof. The bootstrapped meta- vretaggradient at (x, h, w) is given by
Visli x<k((w)) Du, where 11 V=ua(i, 2)) 12-x(k))
Under Eq. 2, we find w = w-BDu. u. Define g = vxf(x(H)). By first-order Taylor Series Expansion of f around (x, h, w) with respect w:
f(x(k((w) = =((x!(((w) + (Dg, w - -w) + o
=((x!K) (w) - B Du) + 0 Du
= =f(x! - 3(u, G" g) + u ((
(5)
To bound the inner product, expand μ(x; .) around point x K) + d. where d R", w.r.t x(K);
p x (K) d) ا x(x) + (u, d) o(1 d 112).
Thus, choose d -a g. for some a E 2 and rearrange get
3 (m(x,x(() -aG g) x(xxx((1) o(a ((
(3(u, GT g)
Substitute into Eq. 5 to obtain
f(x!") f(x(() 3
[ ( ( -aG" g) u(x x x(K))
(6)
+ + l6 + 1/ 13
Thus, the BMG update comes out as the difference between to distances. The first distance is distortion terms that measures how well the target aligns the tangent vector GG" g, which the direction steepest descent in the immediate vicinity of (K) (c.f. Lemma 1). The second term measures learning; greater distance carry more signal for meta-learning. The two combined captures the inherent trade-off in BMG; moving the target further away increases distortions from curvature, but may also increase the learning signal. Finally, the residual captures distortions due to curvature.
Existence. To show that there always exists a target that guarantees a descent direction, choose x x() -aG" g. This eliminates the first distance in Eq. 6 as the target perfectly aligned the direction of steepest descent and we obtain
- ( (w)) = -xxx (K) + o(3(a B)).
a
The residual vanishes exponentially fast a and go Hence, there is some (a. (3) R such that for any (a, B) E a0, x (0,3). f(x (x(()) (w)) - f (x(xx(()) (w)) O. For any such choice of (a, $). by virtue of differentiability in there exists some neighborhood N around (K) -aG for which any & E N satisfy f( (())(W) - (x(x)(()) <0.
Efficiency. We are to show that, given (a. 3), the set of optimal targets does not include x x(K) -aG" g. To show this, it is sufficient to demonstrate that show that this is not a local minimum of the right hand-side in Eq. 6. Indeed,
[(xx, x (K) -a g) ((x)x(()) + (a + 8 (llll(r))
2
x=x(x) -GG
3 Va u(x, x (k)
+ o 0.
(x)()) GG
where 3² the gradient of the residual (I u depends on x) w.r.t. x = ( -aG g. To
complete the proof, let denote the above gradient. Construct alternative target x' nu for some E R.. By standard gradient descent argument, there is some ij such that any E (0. yields alternate target x that improves over &
15

---

Published as a conference paper at ICLR 2022
We now prove that, controlling for scale, BMG can yield larger performance gains than MG. Recall that && (xx(x)) = x( -aG Vf
(K)) v Consider SG: with r: (xV V ((x(x))
Corollary 1. Lel = || 1 and & = Eb (x(()). Let w' be given by Eq. and w be given by Eq. 2. For 3 sufficiently small, f( (w)) S f f (xx()(())) with strict inequality if GG" +G
Proof. Letg = (=)(x))) By Lemma 1, f ff('))((x(()))))))) -3 g (00)) From Theorem with μ = I1- o( ( - ff(x (w)) = -r( g. G" +0((a)+++
For 3 sufficiently small, the inner products dominate and we have
S( x (<) (w)) - 1( (x)((()))) R 8 (r(G" g, G" g) (G" g g.g)
To determine the sign of the expression in parenthesis, consider the problem
(GTH g. v) s.t. || v ll2 I.
max VER"I
Form the Lagrangian L(v. A) = (GT
g. v) X(1 v ll2 - 1). Solve for first-order conditions:
v || Gº
G
=0
If A = O, then we must have v ||20, which clearly is not an optimal solution. Complementary slack-
ness then implies || v" ll2 = 1, which gives À x= || v ||2|| g ll2 and hence v G g /l|G" g ll22- By virtue of being the maximiser, v attains higher function value than any other v with || v 22 S in particular v = g //lg l12. Evaluating the objective at these two points gives
g/ll
(GT g, G g)
(GT g.g) g l2
G( g, G" g) 2 (G" g. g)
||G g
where we use that r r= 1= /l// ||G g 22 by definition. Thus (x)( S f(x( (x()) with strict inequality if GGT * G and G" g
B
NON-STATIONARY NON-EPISODIC REINFORCEMENT LEARNING
B. SETUP
This experiment is designed to provide controlled setting to delineate the dif- ferences between standard eeta-ggadients and bootstrapped meta-gradients. The environment is 5 x 5 grid world with two objects; a blue and red square (Figure 7). Thus, we refer to this environment as the two-colors domain. At each step, the agent (green) can take action to move either up, down, left, or right and observes the position of each square and itself. If the agent reaches a coloured square, it obtains a reward of either +1 -1 while the colour is randomly moved to an unoc- cupied location. Every 100 000 steps, the reward for each object flips. For all other transitions, the agent obtains reward of -0.04. Observations are constructed by concatenating one-hot encodings of the each z- and y-coordinate of the two colours and the agent's position, with a total dimension of 2 x 5 30 (two coordinates for each of three objects, with each one-hot vector being 5-dimensional).
Figure 7: Two-colors Grid-world.
The agent's goal is to collect either blue squared by navigating the green square.
The two-colors domain is designed such that the central component determining how well memory-less agent adapts is its exploration. Our agents can only regulate exploration through policy entropy. Thus, to converge on optimal task behaviour, the agent must reduce policy entropy. Once the task switches, the agent encounters what is effectively novel task (due to it being memory-less). To rapidly adapt the agent must first increase entropy in the policy to cover the state-space. Once the agent observe rewarding behaviour, it must then reduce entropy to converge on task-optimal behaviour.
red
All experiments run the CPU of a single machine. The agent interacts with the environment and update its parameters synchronously in a single stream of experience. A step is thus comprised of the following operations, in order: (1) given observation, agent takes action, (2) if applicable, agent update its parameters, (3) environment transitions based on action and return new observation. The parameter update step is implemented differently depending on the agent, described below.
16

---

Published as a conference paper at ICLR 2022
Algorithm 1 N-step RL actor loop
Require: N
P Rollout length. Policy parameters. Environment state. > Initialise rollout.
Require: x "
Require: S
B + (s)
for t 1,2,...... N do
a ~ xx
> Sample action.
cnv(s,
Take step in environment.
6- B--BU(a,T,
D Add to rollout.
end for return s,
Algorithm 2 K step online learning loop
Require: N, K
Rollout length, meta-update length. b Policy, value function, and meta parameters. Environment state.
Require: x E IR= 2 R== W E R"
Require:
for k 1, 2, K do
B Lc
Algorithm 1. Inner update step.
(x, z) + p((x, z). B, w)
end for return x, 2, B
Algorithm 3 Online RL with BMG
Require: N, K,
Rollout length, meta-update length, bootstrap length. > Policy, value function, and meta parameters. >> Environment state.
Require: x E IR" E R" W E R
Require:
t (x.z)
while True do u K) + InnerLoop(u, w,s, N,
b K step inner loop, Algorithm 2.
n (K+L-1) B +- InnerToop(u w,s, N,
T, bootstrap, Algorithm 2. Gradient step on objective É. > BMG outer step. D Continue from most resent parameters.
ú t u (K+L-1) - aV,l(u(ktbb)) B)
w - -W w u u(K)(w))
< u (K+L-1)
end while
B.2 ACTOR-CRITIC EXPERIMENTS
Agent The first agent we evaluate is a simple actor-critic which implements softmax policy (тx) and critic (v²) using separate feed-forward MLPs. Agent parameter updates are done according to the actor-critic loss in Eq. 4 with the on-policy n-step return target. For given parameterisation of the agent, we interact with the environment for N 16 steps, collecting all observations, rewards, and actions into a rollout (Algorithm 1). When the rollout is full, the agent update its parameters under the actor-critic loss with SGD as the optimiser (Algorithm 2). To isolate the effect of meta-learning, all hyper-parameters except the entropy regularization weight (e égN) are fixed (Table I); for each agent, we sweep for the learning rate that yields highest cumulative reward within 10 million step budget. For the non-adaptive baseline, additionally sweep for the best regularization weight.
Meta-learning To meta-learn the entropy regularization weight, we introduce a small MLP with meta-pant patt-parramters that ingests statistic of the learning process the average reward over each of the 10 most recent rollouts-and and predicts the entropy rate c (t) E R4, to use the agent's parameter update of x. To compute meta-updates, for given horizon T K T K + (L 1), we fix w and make T agent parameter updates to obtain sequence (7), (1), (1)) ..... ...,,,,,.. x(T) (T)).
17

---

Published as a conference paper at ICLR 2022
Rewerd (Millama)
betbbttbee Berverc (Milllmme)
Comobette Revere (Milliom
1.5
1.5
X-3
1.5
X-3
K-?
K-?
2.0
K-15
Kmd1
KROt
0.5
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.5
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Steps
Stage
Staps
(a) Fixed entropy-regularivation
(h) Meta-gradients
(c) Meta-gradients + regularization
K+L=2
K+I=1
K-Lm8
K+L-16
K-L=32
Domelattd Roword
1.0
K-15
0.5
0.0
0.2
0.4
0.6 Stops
0.8
1.0 167
0.0
0.2
0.4
0.6 Stops
0.8
1.0
0.0
0.2
0.4
0.6 Staps
0.8
1.0
0.0
0.2
0.4 0.5
0.8
1.0
0.0
0.2
0.4 0.6
0.8
1.0
167
167
Stops
1c7
Staps
167
(d) Bootstrapped mela-gradients
Figure 8: Total rewards on two-colors with actor-critics. Shading: standard deviation over 50 seeds.
is optimised by averaging each policy and entropy loss encountered in the sequence, i.e. the meta-objective is given by $ Σ & (x(*) (w)) + Emetaf (xx where Emeta. {0, 0.1] is fixed dyper-parameter and le implies that the objective computed under Tt.
MG
BMG is optimised by computing the matching loss May (x, x ((K((w)) where X is given by X x -V (fFa (x(()) Emeta (x(()))
())).. That is to say, the TB "unrolls" the meta-leamer for L - steps, starting from (x(k),z*) (k))), and takes final policy-gradics noted). Thus, in this setting, our TB exploits that the first (L - 1) steps have already been taken by the agent during the course of learning (Algorithm 3). Moreover, the final Lth step only differs the entropy regularization weight, and can therefore be implemented without an extra gradient computation. As such, the meta-update under BMG exhibit great computational overhead the MG update. In practice, we observe no significant difference in wall-clock speed for given K.
step (Emeta 0 unless otherwise
Main experiment: detailed results The purpose of main experiment Section 5.1 is (a) test whether larger meta-learning irtizons-ppaticculee
by increasing T.-can mitigate the short-horizon bias, and (b) test whether the agent can learn exploration schedule without explicit domain knowledge in the meta-objective (in the form of entropy regularization), As reported in Section 5.1, we find the answer to affirmative in both cases. To shed further light on these findings, Figure
K=1
1.25
K=3
Entropy
1.00
K=15
Entropy
1..
K=1
K=1,
0.75
0.50
0.25
0.0
0.2
0.4
0.6
0.8
1.0
0.00
0.25
0.50
0.75
1.00
Enviromment Steps
167
Environment Steps
1e7
Figure 9: Range of the entropy of softmax-police over time (2-colors). Each shaded area shows the difference between the entropy 3333 steps after the agent observes new entropy and the entropy after training the seward-functin regularization (left) reduce entropy over time while Bootstrapped meta- grata-ggrdienss (right) maintain entropy with large enough meta-learning horizon. Averaged across 50 seeds.
for 100000 steps. Meta-gradients without explicit entropy-
18

---

Published as a conference paper at ICLR 2022
Batropy- lots welgen
Areu
1.0
1e6
Xtle2 S0
0.8
Total peturnss
K+1mm
Entropy-low
1.0
0.4
0.2
0.2
0.0
0.0
0.2
Environment Staps
0.4
0.6
0.8
107
1.0
0.0
0.2
Pnvtromment Steps
0.4
0.8
0.5
1c7
1.0
Noat
Type target update
Figure 10: Ablations for actor-critic agent with BMG. Each shaded area shows the range of entropy regularization weights generated by the meta-learner. The range computed as the difference between at the beginning and end of each reward-cycle. Left: entropy regularization weight range when K = 1 and L = Center: entropy regularization weight range when K and L 1. Right: For K = 1 effect of increasing L with or without meta-entropy regularization. Result aggregated over 50 seeds.
reports cumulative reward curves for our main experiment in Section 5.1. We note that MG tends to collapse for any K unless the meta-objective is explicitly regularized via meea- To characterise why MG fail for meta = O, Figure portrays the policy entropy range under either MG or BMG. MG is clearly overly myopic by continually shrinking the entropy range, ultimately resulting in non-adaptive policy.
meta-regguarizztion To fully control for the role of reta--egularrization, we conduct further exper- iments by comparing BMG with and without entropy regularization (i.e. émeta) in the Lth target update step. Figure 10 demonstrates that BMG indeed suffers from myopia when L = 1, resulting in a collapse of the en- tropy regularization weight range. However, increasing the meta-learning horizon by setting L = 7 obtains a wide entropy regularization weight range. While adding meta- regularization does expand the range somewhat, the dif- ference in total return is not statistically significant (right panel, Figure 10).
Ablation:
Masiching functonn X+L
KL-5
AU-
XL2
Total Return (Millions)
1.5
1.0
0.3
0.0
0.0
0.2
0.4
0.6
0.8
1.0
Environment Steps
1e7
Figure 11: Total reward two-colors with an actor-critic agent and different matching functions for BMG. Shading: standard deviation over 50 seeds.
Ablation: target bootstrap Our main TB takes L - steps under the meta-learned update rule, i.e. the meta- learned entropy regularization weight schedule, and an Lth policy-gen gradient step without entropy regularization. In this ablation, we very that taking final step under different update rule is indeed critical. Figure 10 shows that, for = and L E (1,7), using the meta-learned update rule for all target update steps leads to positive feedback loop that results in maximal entropy regularization, leading to a catastrophic loss of performance (right panel, Figure 10).
Ablation: matching function Finally, we control for different choices of matching function. Figure 11 contrasts the mode-covering version, KI-11, with the mode-seeking version, KI.-2, well as the symmetric KI. We observe that, in this experiment, this choice is not as significant as in other experiments. However, in Atari, we find the mode-covering version to perform slightly better.
B.3 Q-LEARNING EXPERIMENTS
Agent In this experiment, we test Peng's Q(A) (Peng & Williams, 1994) agent with e-greedy exploration. The agent implements feed-forward MLP to represent Q-function øx that is optimised online. Thus, agent parameter update steps do not use batching but done online (i.e. on each step). To avoid instability, (EMA) over the agent parameter gradient. In this experiment we fix all hyper-parameter of the update rule (Table 1) and instead focuses meta-learned e-greedy exploration.
momentum term that maintains Exponentially Moving Average
19

---

Published as a conference paper at ICLR 2022
Returns (Millions)
Leid Le32
Total Reture (Milboo))
Palicy Marching
"Valae" matchung
Le128
Pland 0.3
0.05
x-03
1.0
0.5
0.4
0.0
0.00
0.25
0.50
0.75
1.00
0.0
0.2
0.4
0.8
0.0
1.0
Environment Steps
167
Dnvtronment
Stops
107
Steps Cycle
Figure 12: Results on two-colors under Q(A) agent with meta-learned e-greedy exploration under BMG. Averaged over 50 seeds.
BMG We implement BMG in similar fashion to the actor-critic case. The meta-learner is represented by smaller MLP e (-) with pertmparsseters w that ingests the last 50 rewards, denoted by and outputs the use on the current time-step. That is to say, given meta parameerssssrss w, the agent's 's policy is defined by
Ew((t) + Sw(t,)
a = arg max (x(Sr, b)
[
Tx St, be, w)
(w(tt)
clse.
Policy-matching This policy can be seen a stochastic policy which takes the Q-maximizing action with probability -- and otherwise picks an action uniformly at random. The level of entropy in this policy is regulated by the meta-learner. We define TB by defining a target policy under da. where x is given by taking L update steps. Since there are no meta-parameters the update rule, all L steps use the same update rule. However, we define the target policy as the greedy policy
1 if a arg max b (s; b)
Txx
Sr)
else.
The resulting BMG update is simple: minimize the KL-divergence n²' ,x,((, x) = KL (Tx Tx) by adjusting the entropy xx through sw. Thus, policy-matching under this target encourages the meta-learner to match greedy policy-improvemmet operation for a further L steps. More specifically, if arg max (s, b) = arg max (x, b), so that the greedy policy improvement matches the target, then the matching loss is minimised by setting e O. If greedy policy improvement does not correspond, so that acting greedily w.r.t. Qx does not match the target, then the matching loss is minimised by increasing entropy, i.e. increasing s. The meta-objective is defined in terms of x as it does not require differentiation through the update-rule.
target 9x that has been trained
'Value' matching A disadvantage of policy matching is that it provides a sparse learning signal: E is increased when the target-policy differs from the current policy and decreased otherwise. The magnitude of the change depends solely the current value of E. alternative matching functions that provide a richer signal. Inspired by value- matthen for actor-critt agents, we construct a form of 'value" matching by taking the expectation over ix under the induced stochastic policy, (s) := EnEA Tx (a 3)4x(s. a). The resulting matching objective is given by
therefore desirable to evaluate
, x) EE (ux ux(s;; t, w))
While the objective is structurally similar to value-matching u does not correspond to well-defined value-function since 9x is not estimate of the action-valle of πx-
Detailed results Figure 12 shows the learned e-schedules for different meta- leatnng horizons: large enough, the agent is able to increase exploration when the task switches and quickly recovers a near-optimal policy for the current cycle. Figure 12 further shows that a richer matching function, in this case in the form of 'value' matching, yield improved performance.
20

---

Published as a conference paper at ICLR 2022
Table 1: Two-colors hyper-parameters
Actor-critic
Inner Learner
Optimiser
SGD
Learning rate
0.1
Batch size
16 (losses are averaged)
0.99
KT [
MLP hidden layers (v, T)
2
MLP feature size (v, m
256
Activation Function
ReLU
Meta-learner
Optimiser
Adam
(Adam)
10-1
31.33
0.9, 0.999
Learning candidates
(3-10-9, 10000,,110311000--00000000 5,004,, 10-1)
MLP hidden layers (c)
MLP feature size (e)
32
Activation Function
ReLU
Output Activation
Sigmoid
Q(1)
Inner Learner
Optimiser
Adam
Learning Rate
3 100
(Adam)
10
31, 32
0.9, 0.999
Gradient EMA
0.9
0.7
0.99
MLP hidden layers (Q)
2
MLP feature size (Q)
256
Activation Function
ReLU
Meta-learner
Learning Rate
10-4
(Adam)
10-4
31.3g
0.9, 0.999
Gradient EMA
0.9
MLP hidden layers (e)
I
MLP feature size (e)
32
Activation Function
ReLU
Output Activation
Sigmoid
21

---

Published as a conference paper at ICLR 2022
C ATARI
Setup Hyper-parametters are reported in Table 2. We follow the original IMPALA (Espeholt et al., 2018) setup, but do not down-sample or gray-scale frames from the environment. Following previous works (Xu et al., 2018; Zahavy et al., 2020), we treat each game level as separate learning problem; the agent randomly initialized the start of each learning run and meta-learning is conducted online during learning on single task, see Algorithm 6. We evaluate final performance between 190-200 million frames. All experiments are conducted with 3 independent runs under different seeds. Each of the 57 levels in the Atari suite is unique environment with distinct visuals and game mechanics. Exploiting this independence, statistical tests of aggregate performance relies on a total sample size per agent of 3 x 57 171.
Agent We use standard feed-forward agent that received a stack of the 4 most recent frames (Mnih et al., 2013) and outputs a softmax action probability along with a value prediction. The agent implemented as a deep neural network; we use the IMPALA network architecture without LSTMs, with larger convolution kernels to compensate for more a complex input space, and with a larger conv-to-liness to-lin projection. We add experience replay (as per (Schmitt et al., 2020)) to allow multiple steps on the target. All agents use the same number of online samples; unless otherwise stated, they also use the same number of replay samples. We ablate the role of replay data in Appendix C.2.
STACX The IMPALA agent introduces specific form of importance sampling in the actor critic update and while STACX largely rely on the same importance sampling mechanism, differs slightly to facilitate the meta-gradient flow. The actor-critic update in STACX is defined by Eq. 4 with the following definitions of and G. Let P 2 ē E RR, be given and let v : S x A + [0, 1] represent the behaviour policy that generated the rollout. Given Tx and i22 define the Leaky V-Trace target by
Th == xx (a.
sc) / v(a 5s)
Pt = ap min [Mc, p} + (1 ap)m
C X (ac min (); 2) + (1 a mz)
8 = Pt (TU2(()+1) + T++1 - U2 (s())
(n-1)
n( =2(ss) +
Σ i=0
II cuts
8e+8
with ap 2 og Note that assumingg 2 and 1-in the on-policy setting this reduces to the
n-step return since 7/2- = 1, 50 P:: G. = 1. The original v-trace target sets Opp O. 1.
STACX defines the main "task" as tuple "0 Wo)), consisting of a policy, critic, and an actor-critic objective (Eq. 4) under Leaky V-trace correction with meta- parameerssssrss Wo. Auxiliary tasks are analogously defined tuples (x', wt, ft. ws)). i 1. All policies and critics share the same feature extractor but differ in separate MLP for each x' and v². The objectives differ in their hyper-parameters, with all hyper-parameters being meta-learned. Auxiliary policies are not used for acting; only the main policy x² interacts with the environment. The objective used to update the agent's parameters is the sum of all tasks (cach task is weighted through EpG, CEN, ETD). The objective used for the MG update is the original IMPALA objective under fixed pperrpprraeeers p (see Oetta-Optiiiiation in Table 2). Updates to agent parameters and meta-parament simultaneously on rollouts т. Concretely, let m denote parameters of the feature extractor, with (x,, 2;) denoting parameters of task i's policy MLP and critic MLP. Let u; := (m, x,, 2;) denote parameters of (T',, v²), with u = (m, Xn, Zo.... Xn, Z,). Let w = (Wa.... Wn) and denote by h auxiliary vectors of the optimiser. Given batch of) rollout(s) T, the STACX update is given by
happen
).
(1)
RMSProp (u, has ...)
8, V. Σ S (u:w,)
1=1
(w()) (1) (1) = Adam (w, has (..))
g.. V ( (u) (w):
(1)
BMG We use the same setup, architecture, and hyper parameters for BMG as for STACX unless otherwise noted; the central difference is the computation of g.. For L 1, we compute the
22

---

Published as a conference paper at ICLR 2022
bootstrapped meta-gradien under μ4, data 7 by
8u Villlr (uo. up (w)) where (ūo. RRMPPrp " (2) V Vufr up (ub :p))
Note that the target BMG does not use additional gradient information or additional data for L = 1. The only extra computation is the element- wise update required to compute ig and the computation of the matching loss. We discuss computational considerations in Appendix C.4. For L, > 1, we take T, step under the meta-learned objective with different replay data in each update. To write this explicitly, let be the rollout data above. Let +()) denote separate sample of only replay data used in the Ith target update step. For L 1, the TB described by the process
the same gradient Vuf(up :P) the outer objective in STACX; hence,
(u (1) ) RMSProp (u!", (1) h (!!))
(
S ( w:)
(1),
(1) (io"), ) = RMSProp (i1 g88
((1)
V.
S (ii (1),
w:)
(uo -) RMSProp (u0 (1( h
(L-11 2((-11)
v. ()) .p). gu
|(L-1)
Targets and corresponding momentum vectors are discarded upon computing the meta- meta-gradient. This TB corresponds to following the meta-learned update rule for L 1 steps, with a final step under the IMPALA objective. We show in Appendix C.3 that this final step is crucial stabilise meta-learning. For pseudo-code, see Algorithm 6.
Matching functions are defined in terms of the rollout t and with targets defined in terms of the main task Ug. Concretely, we define the following objectives:
(1) p [uo ub" (ww) KL
ग Tu. (T)"(())
Tuo
1 (uo. (u"((w) E
2 "no (ww)
1oto (uo, up (w)) = = (uo ( (w)) Aset (ūo, up (w)).
0.25,
ntب (ūo: up (w)) = jio u (1) (w((w)
23

---

Published as a conference paper at ICLR 2022
Algorithm 4 Distributed N-step RL actor loop
Require: N
b Rollout length.
Require: R
Centralised replay server.
Require: d
> Initial state method. Parameter syne method.
Require: c
while True do
B N then
R RUB
Send rollout to replay. Syne parameters from learner. Optional state reset. D Initialise rollout.
x - c()
s d(s)
< (s)
end if
a ~ (8)
> Sample action.
r' as)
Take step in environment. > Add to rollout.
B B BUa,,,
end while
Algorithm 5 K Ksstep distributed learning loop
Require: B1, B2, ................................ Bx
K N- rollouts. b Policy, value function, and meta parameters.
Require: x E IR. 2 E IR. . E R
for & 1, 1,2, ..... K do
(x, z) 4 (((, z), Bk. w)
b Inner update step.
end for return x,
Algorithm 6 Distributed RL with BMG
Require: N, K, L,, M Rollout length, meta-update length, bootstrap length, parallel actors. & Policy, value function, and meta parameters.
Require: x E
Rt (x,z) Initialise R replay buffer
2 R== R
> Collects N-step trajectories from actors. Run concurrently, Algorithm 4.
Initialise M asynchronous actors
while True do
{B(()) (K) + InnerLoop(u, w, (B(K) }
R
Sample K rollouts from replay.
> K-step inner loop, Algorithm
N= u (K+L-1) +- InnerToop (K), w, (B(0) L-1
L 1-step bootstrap, Algorithm 5. Gradient step on objective é. BMG outer step.
- n (K+L-1) -a Va
B (()-1))
w w -V (K) (w)) (w))
u u k
Optional: continue from K + L - 1 update.
Send parameters x from learner actors.
end while
24

---

Published as a conference paper at ICLR 2022
Table 2: Atari hyper-paramettrs
ALE (Bellemare et al., 2013)
Frame dimensions (H, W. D)
160, 210,
Frame pooling
None
Frame grayscaling
None
Num. stacked frames
4
Num. action repeats
4
Sticky actions (Machado et al., 2018)
False
Reward clipping
[-1,
0 loss of life
True
Max episode length
108 000 frames
Initial noop actions
30
IMPALA Network (Espeholt et al., 2018)
Convolutional layers
4
Channel depths
64,
128, 128, 64
Kernel size
3
Kernel stride
1
Pool size
Pool stride
2
Padding
'SAME'
Residual blocks per layer
2
Conv-to-linear feature size
512
STACX (Zahavy et al., 2020)
Auxiliary tasks
2
MLP hidden layers
2
MLP feature size
256
Max entropy loss value
0.9
Optimisation
Unroll length
Batch size
18
of which from replay
12
of which online data
6
Replay buffer size
10 000
LASER (Schmitt al., 2020) KL-threshold
Optimiser
RMSProp
Initial learning rate
10-4
Learning rate decay interval
200 000 frames
Learning rate decay rate
Linear to 0.99
Momentum decay
Epsilon
10-4
Gradient clipping, max norm
0.3
Meta-Optimisation
7, A, p, č, a
0.995, 1, 1, 1, 1, 0.01, 0.25
<pG- éEN, ETD Optimiser
Adam 10-3
Learning rate
$1,82 82 Epsilon
0.9, 0.999 10-4
Gradient clipping, norm
0.3
25

---

Published as a conference paper at ICLR 2022
C.I BMG DECOMPOSITION
In this section, decompose the BMG agent under- stand where observed gains come from. To do so, we begin by noting that-by virtue of Eq. 3-STAXX
special
6.0
case of BMG under μ(ū up (1) (w)) = l u ( (w)ll
Human sormalized lizedddddd score
5.5
with u u / v..f (u (1) :p). That is say, if the
larget is generated by a pure SGD step and the matching
5.0
function is the squared L2 objective. We will refer this configurations as SGD, L2. From this baseline- STACX-a minimal change to retain the matching func- tion but use RMSProp to generate the target. We refer t this configuration as RMS, L2. From Corollary 1, we should suspect that correcting for curvature should improve performance. While RMSProp not a represen- tation of the metric G in the analysis, it is nevertheless providing some form of curvature correction. The match- ing function can then be used for further corrections.
4.5
4.0
3.5
SGD
RMS L-1
RMS
RMS L-1
RMS
KL V L-4
Figure 13: Atari BMG decomposition. We report human normalized score (me- dian, quantiles, 1 IQR) between 190- 200M frames over all 57 games, with independent runs for each configuration.
Figure 13 shows that changing the target update rule from SGD to RMSProp, thereby correcting for curvature, yields a substantial gain. This supports our main claim that BMG control for curvature and thereby facilitate meta- optimisation. Using the squared Euclidean distance in parameter space (akin to (Nichol al., 2018; Flennerhag al., 2019)) is surprisingly effective. However, it exhibits substantial volatility and is prone to crashing (c.f. Figure 15); changing the matching function to policy KL-divergence stabilizes meta-optimisation. Pure policy-matching leaves the role of the critic-i.e. policy evaluation wvauuation-implici.. Having an accurate value function approximation important to obtain high-quality policy gradients. It is therefore unsurprising that adding value matching provides statistically significant improvement. Finally, also mitigate myopia by extending the meta-learning horizon, our TB by unrolling the meta-learned update rule for L 1 steps. This is roughly important correcting for curvature, in terms of the relative performance gain.
find that BMG can
To further support these findings, estimate the effect BMG has ill-conditioning and meta- gradient variance on three games where both STACX and BMG exhibit stable learning (to avoid confounding factors of non-stationary dynamics): Kangaroo, Star Gunner, and Ms Pacman. While
Table 3: Meta- deta-gradient cosine similarity and variance per-game at 50-150M frames over 3 seeds.
KL
KL & V
L2
STACX
Kangaroo
Cosine similarity Meta- eeta-gradient variance
0.19 (0.02)
0.11 (0.01)
0.001 (1e-4)
0.009 (0.01)
0.05 (0.01)
0.002 (le-4)
2.3e-9 (4e-9)
6.4e-4 (7e-4) 44
Meta-gradient norm variance
49
68
47
Ms Pacman
Cosine similarity
0.11 1 (0.006)
0.03 (0.006)
0.002 (4e-4)
-0.005 (0.01)
Meta- sita-gradient variance Meta-gradient norm variance
90 (12)
0.8 (0.2)
9.6e-7 (2e-8) 4.2
0.9 (0.2)
2.1
7.9
2.1
Star Gunner
Cosine similarity
0.13 (0.008)
0.07 (0.001)
0.003 (5e-4)
0.002 (0.02)
Meta- deta-grradien variance
4.2 (1.1)
1.5 (2.3)
1.9e-7 (3e-7)
0.06 (0.03)
Meta- Meta-gradics gradient norm variance
6.1
6.6
11.7
6.5
26

---

Published as a conference paper at ICLR 2022
the Hessian of the meta-gradien citta-gradiet// is intractable, immediate effect of ill-conditioning is gradient interference, which we can estimate through cosine similarity between consecutive meta- veeta-radients. per-batch basis. Table 3 presents mean statistics between
We estimate meta- -ratding variance 50M and 150M frames, with standard deviation over 3 seeds. BMG achieves a meta-gradient cosine similarity that is generally orders of magnitude larger than that of STACX. It also explicitly demonstrates that using the KL divergence as matching function results in better curvature relative to using the L2 distance. The variance of the meta-gradientt is larger for BMG than for STACX (under KL). This is due to intrinsically different gradient magniudess To make comparisons, we report the gradient norm to gradient variance ratio, which roughly indicates signal to noise. We note that in this metric, BMG tends to be on par with or lower than that of STACX.
C.2 EFFECT OF REPLAY
We find that extending the meta-learning horizon by tak- ing more steps the target leads large performance improvements. To obtain these improvements, we find that
35
L=4
L=1
Episode return (x100))
30
L=4, R
is critical re-sample replay data for each step, as op- posed to re-using the same data for each rollout. Figure 14 demonstrates this for L. on MsPacman. This can be explained by noting that reusing data allows the target to overfit to the current batch. By re-sampling replay data we obtain more faithful simulation of what the meta-learned update rule would produce in L - 1 steps.
L=4, w
25
20
15
5
The amount of replay data is confounding factor in the meta-objective. We stress that the agent parameter update is always the same in any experiment to say, the additional use of replay data only affects the computation of the meta-objective. To control for this additional data in the meta-objective, we consider a subset of games where we see large improvements from T. > 1, We run STACX and BMG with L 1, but increase the amount of replay data used to compute the meta-objective to match the total amount of replay data used in the meta- objective when L This changes the aaline-to-replay ratio from 6: 12 to 6 48 in the meta objective.
50M
100M 150M 200M Learning frames
run. That is
Figure 14: Atari, learning curves on MS Pacman for KL L = 4, R com- putes the Lthe step on only replay data, I, 4, w uses the meta-learned objec- tive for the Ltth step (with Lthh step com- puted on online and replay data, as per default), Shading depicts standard devia- tion across 3 seeds.
Figure 15 shows that the additional replay data is not responsible for the performance improvements we see for L, = 4. In fact, we find that increasing the amount of replay data in the meta-objective exacerbates ooolpy issues and leads to reduced performance. It is striking that BMG can make use of this extra off-policy data. Recall that on the target, and use the original online-to-replay ratio (6 12) in the Lth step. In Figure 14, we test the effect of using only replay for all L steps and find that having online data in the Lth update step is critical. These results indicate that BMG can make effective use of replay by simulating the effect of the meta-learned update rule off-policy data and correct for potential bias using online data.
only off-policy replay data to take the first L steps
C.3 V
Given that increasing I. yields substantial gains in performance, it is interesting to compare against increasing K, the number of agent parameter updates to backpropagate through. For fair comparison, we use identical setup as for L > in the sense that K- steps, while we use the default rollout 7 for the Kth step. Hence, the data characteristics for K are identical to those of L
new replay data for each of the initial
However, an important difference arise because each update step takes K steps on the agent's s parameters. This means advantage as it is able to do more updates to the agent's 's parameters. With that said, these additional K 1 updates replay data only.
the 200 million frames budget, K > has computational
27

---

Published as a conference paper at ICLR 2022
asternids
defender
robetank
down
wizard of
75
50
1.00
75
50
25
100
1005, 12
75
50
25
STACK L.2)
75
50
%
50
100 150
200
50 100
150
200 0
50
100
150
200
50
100
150
200
0
50
100
150 200
50
100
150
200
Figure 15: Atari experience replay ablation. We report episode returns, normalized to be in the range [0, max return] for each game for ease of comparison. Shading depicts standard deviation across 3 seeds. D denotes default BMG configuration for L 1, with L 4 analgously defined. R denotes = 1, but with additional replay in the meta-objective to match the amount of replay used in L = 4.
Figure 16 demonstrates that increasing K is fundamentally different from increasing L. We generally observe loss of performance, again due to interference from replay. This suggests that target bootstrapping allows a fundamentally different way of extending the meta-learning horizon. In particular, these results suggests that meta-bootstin bootstrapping allows us to use relatively poor-quality (as evidence by K > 1) approximations without impairing the agent's s actual parameter update. Finally, there are substantial computational gains from increasing the meta- meta-learning horizon via L rather than K (Figure 17).
long-term consequences of the meta-learned update rule
C.4 COMPUTATIONAL CHARACTERISTICS
IMPALA's distributed setup is implemented on a single machine with 56 CPU cores and 8 TPU (Jouppi al., 2017) cores. 2 TPU cores used to act in 48 environments asynchronously in parallel, sending rollouts to a replay buffer that centralized learner use to update agent parameters and meta- peta-parameters. Gradient computations are distributed along the batch dimension across the remaining 6 TPU cores. All Atari experiments use this setup; training for 200 millions frames takes 24 hours.
Figure 17 describes the computational properties of STACX and BMG as a function of the number of agent parameters and the meta-learing horizon, H. For STACX, the meta-learning horizon is defined by the number of update steps to backpropagate through, K. For BMG, we test one version which holds J, = fixed and varies K, as in for STACX, and one version which holds K ficed and varies L. To control for network size, vary the number of channels in the convolutions of the network. We use a base of channels per layer, x = (16, 32, 32, 16). that we multiply by a factor 1, 2, 4. Thus consider networks with kernel channels Ir (16, 32, 32, 16), 2r (32, 64, 64, 32), and 4x = (64, 128, 128, 64). Our main agent uses a network size (Table 2) equal to 4x. We found that larger networks would not fit into memory when K > 1.
First, consider the effect of increasing K (with L = 1 for BMG). For the small network (1x), BMG is roughly on par with STACX for all values of K considered. However, BMG exhibits poorer scaling
28

---

Published as a conference paper at ICLR 2022
asternids
defender
robetank
down
wizard of
75
K-2
50
mud "iann
1.00
50
25
100
1005, 12
75
25
k
m//
75
50
STACK
0
50
100
150
200
50
100
150
200
0
50 100
150
200
50
100
150
200
0
50
100
150
200
50 100
150
200
Figure 16: Atari K vs L ablation. We report episode returns, normalized to be in the range [0, max return] for each game for ease of comparison. Shading depicts standard deviation across seeds. D denotes default BMG configuration for L 1, with L analogously defined. K denotes L 1, but K = steps on agent parameters.
in network size, owing the additional update step required compute the target bootstrap. For 4r, our main network configuration, we find that BMG is 20% slower in terms of wall-clock time. Further, we find that neither STACX nor BMG can fit the network size memory when K 8.
Second, consider the effect of increasing L with BMG (with K 1). For 1x, we observe no difference in speed for any H. However, increasing I. exhibits dramatic improvement in scaling for H > 2 especially for larger networks. In fact, I. = 4 exhibits factor 2 speed-up compared to STACX for = 4, 4x and is two orders of magnitude faster for H = 8, 2x.
C.5 ADDITIONAL RESULTS
Figure 19 presents per-game results learning curve for main configurations considered in this paper. Table presents mean episode returns per game between 190-200 millions frames for all main
H-2
Seconda per update sep
0.5
0.50
0.55
1.2
BMG, H-X
BMG, Hel
400
0.4
0.45
1.0
STACX,
300
200
0.8-
0.3
0.35
100
0.30
0.6
0.2
0.25
2x
2x
4%
2x
Network
Figure 17: Atari: Computational characteristics as a function of network size (see Appendix C.4) and meta-learning horizon H. When H K, we vary the number of update steps to backpropagate through (with = for BMG). When = L, we vary the number target update steps (with K 1). Measurements taken over the first 20 million learning frames the game Pong.
29

---

Published as a conference paper at ICLR 2022
configurations. Finally, consider two variations of BMG in the L regime (Figure 18); one version (NS) re-computes the agent update after updating meta- partameetssssss a form of trust-region method. The other version (DB) exploits that the target has a taken further update step and uses the target as new agent parameters. While NS is largely on par, interestingly, DB fails completely.
C.6 DATA AND HYPER-PARAMETE SELECTION
We use the ALE Atari environment, publicly avail- able at https: / //gthub.. com/ htt///mmmeeeemarraee Arcade-Learning-Be Enw.ernnert,,,n..mmmmmo licensed under GNU GPL 2.0. Environment hyper-parameters were selected based on prior works (Mnih et al., 2013; Espeholt et al., 2018; Zahavy et al., 2020; Schmitt et al.,
Median human normalized score
BMG KL
-5
BMG w. KI,
BMG W. KL, DB
4-
2020). Network, optimisation and meta-optimisation pyper--arameters are based on the original STACX implementation and tuned for optimal performance. Our median human normalized score matches published results, For BMG, we did not tune these hyper-parameters except for L > 1. In this case, observed that unique replay data the initial yield any benefits. We observed tendency to crash, and thus reduced the gradient clipping ratio from .3 to .2. For BMG configurations that use both policy and value matching, we tuned the weight on value matching by a grid search over (0.25, 0.5, 0.75) Ms Paeman, Zaxxon, Wizard of Wor, and Seaquest, with 0.25 performing best.
0
50M 100M 150M 200M Learning frames
steps was necessary to
Figure 18: Atari BMG, alternative meta- update strategies. NS re-computes the agent-update the meta-update, akin to trust-region method. DB uses the boot- strap target as the next agent parame- ters. Shading depicts standard deviation across 3 seeds.
D MULTI-TASK META-LEARNING
D.1 PROBLEM FORMULATION
Let p(T) denote a given task distribution, where E N indexes a task f". Each task is also associated with distinct learner states h, and task parameters x,, but all task learners use the same meta- meta-leamed update rule defined by meta-parameters w. Hence, the meta-learner's problem is again to learn an update rule, but now in expectation over all learning problems. The MG update (Eq. 1) thus takes the form w' w - V. (x, (K) (w))), where the expectation is with respect to (f", h,, x7) and (k) (w) is the K K-step update on task 7 given (f h,,x)). Since p(T) is independent of w, this xf
(K) (w))]. i.e. the single-task meta eeta-gradieet in Section in update becomes w' w [V ft (x. [W))) expectation over the task distribution.
With that said, the expectation involves integrating over x,). This distribution is defined differently depending on the problem setup. In few-shot learning, x+ and h, are typically shared initialisations (Finn et al., 2017: Nichol et al., 2018; Flennerhag et al., 2019) and f differ in terms of the data (Vinyals et al., 2016). However, it is possible to view the expectation as prior distribution over task parameters (Grant et al., 2018; Flennerhag et al., 2020). In online multi-task learning, this expectation often reduces to expectation current task-learning states (Rusu al., 2015; Denevi et al., 2019).
The BMG update is analogously defined. Given TB ç, define the task-specific target x, given x; (K) by ((x!x)). The BMG meta-loss takes the form w' -3V un F,, (u- 1 (((((())) (w))), where μ4, is defined on data from task r. As with the MG update, as the task distribution is independent w, this simplifies to w' data from 7. Hence, as with MG, the multi-task BMG update is an expectation over the single-task BMG update in Section 3. See Algorithm for detailed description.
- 3 [V Wr (x,, x! (w))], where μ4, is the matching loss defined on task
30

---

Published as a conference paper at ICLR 2022
D.2 FEW-SHOT MINIIMAGENET
Setup Minilmagenet (Vinyals et al., 2016; Ravi & Larochelle, 2017) is sub-sample of the Imagenet dataset (Deng et al., 2009). Specifically, it is subset of 100 classes sampled randomly from the 1000 classes in the ILSVRC-12 training set, with 600 images for each class. We follow the standard protocol (Ravi & Larochelle, 2017) and split classes into non-overlapping meta-training, meta-validation, and meta-tests sets with 64, 16, and 20 classes in each, respectively. The datasset is licenced under the MIT licence and the ILSVRC licence. The dataset can be obtained from https: : / ppapersw thoode. classification tasks are sampled following standard protocol (Vinyals et al., 2016). For each task, M = 5 classes are randomly sampled from the train, validation, or test set, respectively. For each class, K observations without replacement. The task validation set is constructed similarly from a disjoint set of L images per class. We follow the original MAML protocol for meta- veeta-triiiing (Finn et al., 2017), taking K task adaptation steps during muta-training and 10 adaptation steps during meta testing.
com/ htmm/datsee....mm aeetasee/miit / i imagenet-1. M-shot-N-wee
randomly sampled
We study how the data-efficiency and computational efficiency of the BMG meta-objective compares against that of the MG meta-objective. To this end, for data efficiency, we report the meta-test set performance as vary the number of meta- betta-bacches each algorithm is allow for meta-training. As more batthbatchhs mean more meta-tasks, this metric captures how well they leverage additional data. For computational efficiency, we instead report meta-test set performace as a function of total metattraining time. This metric captures computational trade-offs that arise in either method.
For any computational budget in either regime (i.e. meta-batches or T hours of training), we report meta-test set performance across 3 seeds for the hyper- performance (Table 4). This reflects the typical protocol for selecting hyper-panc paper-parametes, and what each method would attain under given budget. For both methods, we sweep over the meta-learning rate B; for shorter training runs, higher meta-learning critical quickly converge. This however lead to sub-optimal performance for larger meta-training budgets, where a smaller meta-learning rate can produce better results. The main determinant for computational cost is the number of steps to backpropagate through, K. For BMG,
with best validation
sweep over K E {1. For MG, sweep over K E {1, 5, 10). We allow K = 10 for MAML to ensure fair comparison, as BMG can extend its effective meta-learning horizon through the target bootstrap; we sweep that the combination of K and L. effectively lets BMG interpolate between different computational trade-off's. Standard MG does not have this property, but several first-order approximations have been proposed: we allow the MG approach to switch from a full meta-gradient to either the FOMAML approximation (Finn et al., 2017) or the ANIL approximation (Raghu al., 2020).
I. {1,5, 10). Note
Model, compute, and shared hyper-parameters We use the standard convolutional model (Vinyals et al., 2016), which is a 4-layer convolutional model followed by a final linear layer. Each convolutional layer is defined by 3 x 3 kernel with 32 channels, strides of 1, with batch normalisation, a ReLU activation and 2 x 2 max- max-pooling. We use the same syper-parameters of optimisation and veta-optimisation Table 4. Each model is trained on a single machine and runs on a V100 NVIDIA GPU.
in the original MAML implementation except specified in
Table 4: Hyper-parameter sweep per computational budget.
MAML
BMG
(0.0001, 0.001) {1,5, 10)
{0.0001, 0.001}
K
{1,5}
L
51,5, 10)
(KL (x || KL (- x))
FOMAML
True, False True, False
ANIL
Total
24
24
31

---

Published as a conference paper at ICLR 2022
Table 6: Effect of BMG on ill-conditioning and meta-gradient variance on 5-way-5-shot Minilmagenet. Estimated meta-gran eetaagrrdieet cosine similarity (0) between consecutive gradients, meta-gradieed variance (V), and meta-gradient norm variance ratio (p). Standard deviation across 5 independent seeds.
MAML V
BMG V
K
L
0
P
0
P
1
1
0.17 (0.01)
0.21 (0.01)
0.02 (0.02)
0.17 (0.01)
0.0002 (5e-6)
0.59 (0.03)
5
0.18 (0.01)
0,001 (le-5)
0.23 (0.01)
10
0.19 (0.01)
0.0003 (2e-5)
0.36 (0.01)
5
1
0.03 (0.01) 0.07 (0.009)
0.08 (0.03)
0.03 (0.005)
0.01 (9e-5)
0.84 (0.03
5
0.04 (0.005)
0.001 (5e-5)
0.46 (0.02)
10
0.05 (0.004)
0.003 (3e-5)
0.18 (0.02)
D.3 ANALYSIS
In terms of data-efficiency, Table 7 reports best hyper-parameters for each data bud- get. For both BMG and MG, we note that small budgets rely on fewer steps to backpropagate through and a higher learning rate. BMG tends to prefer a higher target bootstrap this regime. MG switches to backpropagation through K > I sooner than BMG, roughly around 70 000 meta-updates, while BMG switches around 120 000 meta-updates. This ac- counts for why BMG can achieve higher performance faster, as it can achieve simi- lar performance without backpropagating through more than one update. It is worth noting that as BMG is given larger train- ing budgets, to prevent meta-overfitiing shorter target bootstraps generalize better. We find that other hyper-paranen parameters are not important for overall performance.
Table 5: Meta-training steps per second for MAML
and BMG on 5-way-5-shot Minilmagenet. Standard deviation across seeds in parenthesis.
K
L
H =
K +
MAML
BMG
1
2
14.3 (0.4)
12.4 (0.5)
5
6
6.9 (0.3)
10
11
4.4 (0.1)
5
1
6
4.4 (0.06)
4.2 (0.04)
5
10
3.2 (0.03)
10
15
2.5 (0.01)
10
1
11
2.3 (0.01)
2.2 (0.01)
5
15
1.9 (0.01)
10
20
1.7 (0.01)
15
15
1.4 (0.01)
20
20
1.1 (0.01)
In terms of computational efficiency. Ta- ble 7 reports best seper-parameters each time budget. The pattern here follows a similar trend. MG does btter under a lower learning rate already after hours, whereas BMG switches after about 8 hours. This data highlights the dominant role K plays in determining training time.
We compare wall-clock time per meta-training step for various values of K and L Table 5. In our main configuration, i.c.
5, L 10, BMG achieves a throughput of 2.5 meta-trann tettitriiingg steps per second, compared 4.4 for MAML, making BMG 50% slower. In this setting, BMG has an effective meta- leaanie horizon 15, whereas MAML has a horizon 5. For MAML to achieve an effective horizon of 15, it's throughput would be reduced to 1.4, instead making MAML 56% slower than BMG.
Finally, we conduct similar analysis Atari (Appendix C.1) study the effect BMG has ill-conditioning and meta-gradient variance. We estimate ill-conditioning through cosine similarity between consecutive meta-gradients, and meta-gradient variance a per meta-batch basis, We report mean statistics for the 5-way-5-shor setup between 100 000 and 150 000 meta- meta-gradiem steps, with standard deviation over 5 independent seeds, in Table 6.
Unsurprisingly, MAML and BMG are similar in terms of curvature, as both can have a KL-divergence increases, suggesting that BMG
type of -njea-obeeetive. BMG obtains greater cosine similarity can transfer more information by having a higher temperature in its target. However, BMG exhibits substantially lower meta-gradient variance, and the ratio of meta-gradient norm to variance is an order of magnitude larger.
32

---

Published as a conference paper at ICLR 2022
Algorithm 7 Supervised multi-task meta-learning with BMG
Require: K. L
meta-update length, bootstrap length meta-batch size, inner batch size, meta-training steps. D model and meta parameters.
Require: M, M.N,T T
Require: x E IR" E T
for 1 1.2,..., do
f- ()
Initialise meta-gradient.
for i 1, ..,, ,, do
T P(T)
Sample task. b For MAML, set x = w.
X, f- X
for k = 1, 2, ..... K do
Sr ~ Ptrain (C T)
Sample batch of task training data. Task adaptation.
X, = + (x Sr, w)
end for x(k) xx for 1 = 1, 1,2,.... ,. do
K-step adaptation.
S, ~ Pres( T) SPeies
> Sample batch of task test data. b L step bootstrap.
x, =x, t+p(x, Sr: w) end for S (S T)
final gradient step then x, = -a V. xx((x, (+)
Assign target.
else
x x +x(xx, Sr: w)
end if
g + M( (())(w))
g
end for
w
8 g
BMG outer step.
end for
33

---

Published as a conference paper at ICLR 2022
Table Data-efficiency: mean meta-test accuracy over 3 seeds for best hyper-parameters per data
budget. j = 1 corresponds to KL (x .) and
2 to KL
x).
Step (K)
8
K
L
μ
Acc.
B
K
FOMAML ANIL
Acc. (%)
10
10
1
10
1
61.4
10
False
True
61.7
20
10-3
1
10
1
61.8
10-3
False
False
61.9
30
10-3
1
10
1
62.5
10-3
10
False
True
62.3
40
10-3
5
1
1
63.1
10-3
5
False
False
62.7
50
10-3
5
63.5
10-3
10
False
True
62.9
60
10-3
5
I
63.7
10-3
False
False
63.0
70
10-3
1
2
63.7
10-3
5
False
False
63.0
80
10-3
5
63.7
10 4
False
False
63.1
90
10-3
5
I
63.8
10
5
False
False
63.3
100
10-3
1
2
63.8
10
5
False
False
63,4
110
10-3
2
63.9
10 10 4
False
False
63.6
120
10-4
5
5
1
63.9
10
5
False
False
63.6
130
10
5
10
1
64.0
10 10
False
False
63.6
140
10-4
5
5
1
64.1
10 4
False
False
63.6
150
10
5
5
64.2
10 10 4
10
False
True
63.6
160
10
5
5
1
64.3
10
5
False
False
63.6
170
10
5
5
64.4
10 -
False
False
63.7
180
10-1
5
5
64.5
10-4 1
10
False
False
63.8
190
10
5
10
1
64.6
10
5
False
False
63.9
200
10-3
5
2
64.7
10-4
10
False
False
64.0
210
10
5
1
64.7
10-1
5
False
False
64.1
220
10
5
5
64.7
10
10
False
False
64.2
230
10
5
5
1
64.8
10
False
False
64.2
240
10-4
5
1
2
64.8
10
False
False
64.1
250
10
5
5
I
64.9
10 10
5
False
False
64.1
260
10
5
1
1
64.9
10
False
False
64.0
270
10
5
64.8
10
False
False
63.9
280
10-1
5
1
64.8
10-1
5
False
False
63.8
290
10
10-4
5
1
1
64.7
10-4
False
False
63.8
300
10
5
5
1
64.7
10
False
False
63.8
34

---

Published as a conference paper at ICLR 2022
Table 8: Companational-efficiinnyy mean meta-test accuracy over 3 seeds for best hyper-parameters per time budget. / corresponds KT. (& and y KT. x).
Time (h)
3
L
Acc.
(%)
8
K
FOMAML
ANIL
Acc.
(%)
1
10
1
1
2
63.5
10-3
False
False
63.0
2
10
1
1
2
63.6
10-3
10
False
True
63.0
3
10-3
5
1
63.7
10 10 3
False
False
63.0
4
10-3
5
1
63.8
10
5
False
True
63.1
4
10-3
5
1
63.8
10-4
False
True
63.4
5
10-3
5
63.8
10 10 4
False
False
63.5
6
10
5
10
63.8
10 10
False
False
63.6
7
10
5
63.8
10
False
False
63.6
10 3
5
63.8
10 4
False
False
63.6
9
10
5
63.9
10
False
False
63.6
10
10
5
64.2
10
False
False
63.7
11
10-4
64.3
10
False
False
63.8
12
10 10-4
5
5
64.5
10
False
False
63.9
13
10
5
5
1
64.6
10
False
False
63.9
14
10-4
5
2
64.7
10
False
False
63.8
15
10-4
5
64.8
10-4
False
False
63.4
16
10
5
1
64.8
10-3
10
False
False
63.2
17
10-4
5
64.8
10
10
False
False
63.3
18
10-4
5
10
64.8
10
10
False
False
63.5
19
10
5
5
1
64.8
10 10
10
False
False
63.6
20
10
5
5
64.7
10 10
10
False
Falsc
63.8
21
10
5
10
64.7
10
10
False
False
63.9
21
10 10-4
5
10
64.7
10
10
False
False
63.8
22
10
5
5
64.7
10
10
False
False
63.9
23
10
5
64.7
10
10
False
False
63.8
24
10-4
5
10
1
64.7
35

---

Published as a conference paper at ICLR 2022
alien
amidar
assault
asterix
asteroids
atlantis
bank heist
battle zone
beam rider
berzerk
bowling
boxing
breakout
centipede
chopper command
crazy climber
defender
demon attack
double dunk
enduro
fishing derby
freeway
frostbite
gopher
gravitar
hero
hockey
jamesbond
kangaroo
krull
kung fu mastaontezuma revengams pacman name this game
phoenix
pong
private eye
qbert
riverraid
road runner
robotank
seaquest
skiing
solaris
space invaders
star gunner
surround
tennis
time pilot
tutankham
KL&V, L=4
KL
up down
video pinball
wizard of wor yars gerenee
KL-S
V
L2
STACX
Figure 19: Atari, per-game performance across 3 seeds. Shading depicts standard deviation.
36

---

Published as a conference paper at ICLR 2022
Table 9: Mean per-game performance between 190-200M frames.
KI.
KI. & V
KI. & 1.=4
KL--S
1.2
STACX
V
Alien
45677
44880
58067
35750
50692
31809
7964
Amidar
4800
7099
7528
4974
7691
3719
1896
Assault
20334
29473
33019
21747
28301
19648
4101
Asterix
511550
439475
533385
487367
6798
245617
86053
Asteroids
145337
238320
289689
8585
220366
156096
56577
Atlantis
831920
813772
814780
806698
854441
848007
648988
Bank Heist
571
1325
13
1165
1329
1339
Battle Zone
73323
88407
88350
78941
50453
78359
72787
Beam Rider
37170
51649
57409
41454
67726
62892
74397
Berzerk
21146
2946
1588
2183
240
1523
1069
Bowling
46
50
42
46
50
28
52
Boxing
100
100
86
100
100
100
100
Breakout
742
832
847
774
827
717
16
Centipede
537032
542730
558849
291569
550394
478347
8895
Chopper Command
830772
934863
838090
736012
11274
846788
341350
Crazy Climber
233445
212229
265729
199150
229496
182617
126353
Defender
393457
374012
421894
364053
69193
344453
55152
Demon Attack
132508
133109
133571
132529
133469
130741
129863
Double Dunk
22
23
23
21
23
24
23
Enduro
2349
2349
2350
2360
2365
259
2187
Fishing Derby
41
63
68
41
52
62
59
Freeway
10
30
25
31
30
18
33
Frostbite
8820
3895
3995
5547
13477
2522
1669
Gopher
116010
116037
122459
92185
122790
87094
11920
Gravitar
271
709
748
259
3594
2746
944
Hero
60896
48551
52432
56044
51631
35559
20235
Ice Hockey
20
Jamesbond
22129
25951
30157
25766
18200
26123
23263
Kangaroo
12200
12557
13174
1940
13235
3182
8722
Krull
10750
9768
10510
11156
10502
10480
8899
Kung Fu Master
51038
58732
54354
54559
63632
67823
54584
Montezuma Revenge
0
O
0
0
0
0
Ms Pacman
25926
22876
28279
26267
27564
12647
2759
Name This Game
31203
31863
36838
30912
32344
24616
12583
Phoenix
529404
542998
658082
407520
440821
370270
247854
Pitfall
Pong
21
21
21
21
21
21
21
Private Eye
165
144
130
100
68
Qbert
87214
37135
72320
30047
75197
27264
3901
Riverraid
129515
132751
32300
91267
177127
47671
26418
Road Runner
240377
61710
521596
17002
424588
62191
34773
Robotank
64
66
71
65
61
65
Scaquest
684870
2189
82925
616738
1477
1744
3653
Skiing
-10023
-8988
-9797
-8988
.9893
-00504
13312
Solaris
2120
2182
2188
1858
2194
2326
2202
Space Invaders
35762
54046
40790
11314
49333
34875
15424
Star Gunner
588377
663477
790833
587411
39510
298448
43561
Surround
9
10
3
Tennis
23
24
23
21
19
24
Time Pilot
94746
60918
68626
95854
93466
49932
40127
Tutankham
282
268
280
288
101
205
Up N Down
342121
303741
381780
109392
202715
315588
17252
Venture
0
Video Pinball
230252
479861
399094
505212
485852
441220
77100
Wizard Of Wor
21597
45731
49806
22936
10817
47854
24250
Yars Revenge
77001
286734
408061
32031
398656
113651
77169
Zaxxon
44280
49448
59011
36261
49734
56952
35494
37


Reviews:
[{"review_id": "yAHaPq-8sm7", "reviewer": "Reviewer_u9p2", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality:** The method proposed in the paper is, to the best of my knowledge, novel. The related work section adequately connects the algorithm with existing work in similar directions. \n\n**Significance:** The empirical results show considerable improvement w.r.t. well-performing baselines; moreover, the general idea behind the method could inspire future research.\n\n**Rigour:** Both the theoretical results and the experimental protocols seem sound and solid to me.\n\n**Strengths**\n\n- The method is based on a conceptually compelling and inspiring idea.\n- The empirical results are remarkable. The ablations and additional experiments (also from the Appendix) help in understanding what matters in the practical algorithm, as well as highlighting the important parts of the contribution.\n- In particular, being able to meta-learn hyperparameters for a behaviour policy can open up new avenues for exploration in reinforcement learning.\n- The theoretical results have a clear scope (although not so large) and provide some easy to understand local improvement guarantees.\n- The paper is well-written and generally easy to follow.\n\n**Minor Concerns / Questions**\n\n- I believe the name *matching function* makes the presentation of the method a little bit harder to digest. Since the function is a pseudometric (i.e., the larger it is, the larger the distance from the target), it should really be called with a name that reminds the reader of this nature (e.g., *mismatch function*).\n- I enjoyed the theoretical results, but it is a pity that they only deal with targets of specific forms and, especially, with $L=1$ only. Ideally, theoretical result with a dependency on $L$ would shed some light on the benefits and limitations of longer bootstrapping horizons.\n- Can the authors elaborate on the connection between the way the bootstrapping target is formed in their method and traditional temporal difference learning? In particular, the grounding role of that subtracted gradient \"nudging the trajectory in a descent direction\" is the same as the one of the reward in temporal difference learning; but, while the reward is at the beginning of the trajectory, the grounding is here at the end of the optimization subtrajectory. Is there any mathematical connection beyond the general shared motivation?\n- As briefly touched upon in some passages of the paper, when the underlying function is highly nonlinear, there is the risk that the bootstrapping mechanism can lead the optimization process in worse areas of the landscape. For instance, if the function in Figure 1 had a bump/plateaux where $\\tilde w$ is, the bootstrapping mechanism would cause more troubles than standard meta-gradients. Why is this not happening in practice?\n\n-------\n*After rebuttal*: I am happy with the answer the authors provided and the update to the paper, which will help the readers understand the relationship between the proposed method and TD-learning. Overall, the improvements make me believe that this paper should be highlighted at the conference, to give other researchers working in the field the possibility to get inspired by this new idea. I am thus raising my score to 10.", "summary_of_the_paper": "The paper proposes *Bootstrapped Meta-Learning,* a new meta-learning algorithm for hyperparameter optimization. Drawing inspiration from temporal difference learning techniques in reinforcement learning, the meta-learner is asked to predict the result of additional unrolled steps of the optimization process, by minimizing a carefully selected distance to a target generated during training. This allows for longer meta-learning optimization horizons, without the need for differentiation through longer optimization trajectories. The method is tested for hyperparameter optimization for reinforcement learning, including learning the exploration hyperparameter for a behaviour policy, and in multi-task meta-learning.", "summary_of_the_review": "The algorithmic and empirical contributions of the paper, as well as the theoretical grounding, largely justify in my opinion its acceptance at the conference.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "ZPUeQVh_un8", "reviewer": "Reviewer_SXZx", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\n-  The paper is generally clear and well written.  It is content dense in and certain areas would benefit from more detailed discussions.  However, the appendices are very detailed and provide answers to almost all of the questions that arise during first reading.  In my suggestions below I’ve highlighted a few areas I feel this content would be especially beneficial (and one pare where the main text could be stripped-back as there is always a trade-off with fixed page limits).\n\n- The key ideas of BMG, bootstrapping a target to combat myopic MG updates and using matching functions to improve the meta-learning dynamics, are novel and well motivated.  I strongly believe that these ideas will be of interest to the broader meta-learning community and will likely lead to multiple future research directions.\n\n- The empirical results are strong and thorough.  The toy-model grid world in Sec 5.1 effectively highlights key properties of the proposed algorithm — such as the ability of BMG to exploit longer meta-learning horizons — and extensive testing on Atari environments (Sec 5.2) shows a significant improvement of STACX and leaves little doubt of the efficacy of BMG for self-tuning algorithms.   Moreover, the demonstrated performance improvement of BMG over MAML (sec 6) suggests broad possible applications and further endorses the proposed algorithm.\n\n- BMG also has a couple of nice properties besides raw performance: (i) achieving less myopic updates without having to backpropagate through many update steps of the inner-loop parameters and (ii) allowing for the meta-learning of “behavioural parameters” that are not used in the learning rule (exploration epsilon is used as the example).  Whilst (i) is clearly desirable in the pursuit of computational efficiency, (ii) is very intriguing and opens the door to new applications of meta-learning (indeed, whilst it is mentioned in the abstract and summaries, I believe this point is slightly undersold in the main text and that this experiment could be presented in more detail than the single paragraph at the end of Sec 5.1 it is given — however, given the page limit I can understand the authors predicament).\n\n**Weakenesses**\n\n- My primary concern is not with the content of the paper, but that the it can be quite difficult to intuitively link the numerous experiments back to intuition or interpretation of the results.  I believe this was largely because the exact methodologies are difficult to follow and, indeed, I found it was essential to refer to the appendices repeatedly to fully unpack the experiments and appreciate the results.  Whilst I am sympathetic to space constraints, I believe the authors would be well served to provide more detailed descriptions in the main text or, ideally, an algorithm box.\n\n- I find the implementation and implications of the experiment on multi-task few-shot learning (Sec 6) unclear in the following regards:\n\t- I do not understand the intuition of why a “hot” expert transforms more information than a “cold” expert, and, moreover, why BMG is able to use this to improve performance.  Could the authors clarify these points.\n\t- I feel this section in particular suffers from a lack of formal introduction of the task and methodology.  For example, the adaptation seems to be defined as a single step, whereas the appendix (D.2) notes that 10 are used during meta-testing.  Concretely, I think a formal description of the training procedure for BMG would be appropriate in the main text.\n\n- I do not find the analysis presented in Sec 4 (“Performance Guarantees”) especially insightful.  My reading is that, whilst the MG update presented in Lemma 1 can guarantee local improvement, practical implementations of BMG do not.  Empirically, however, grounding the bootstrapped target with a single final step on the meta-objective is sufficient for good performance.  This conclusion is evident from the experiments themselves, and so I would have no issue if Sec 4 was in the appendix.\n\n**Errata**\n\n- Sec 3, paragraph starting “To tackle myopia…”: the inline equation for $\\tilde{x}$ is missing brackets around the step counters for $x^{(K+L-1)}$.\n- Sec 5.2, second last paragraph: typo - “K is more sensitive curvature and the quality of data”.\n- Sec 6, sub-section BMG: typo - “raising the temperature in the expect allows”.\n- Sec 6, sub-section Setup: typo - “For 50 meta-updates and beyond..”: this should be 50k meta-updates I believe.", "summary_of_the_paper": "The paper presents Bootstrapped meta-gradients (BMG), an extension of typical meta-gradients (MG) for the task tuning meta-parameters that control the learning process (i.e. update step) of a learner.  In general terms, MG applies a (meta-)parameterised update rule to a learner for K steps, and then backpropagates through these updates to update the meta-parameters in the direction that improves the performance of the adapted learner.  The authors identify two limitations to this approach, and propose BMG to address them. (i) MG is myopic, in the sense that it does not account for future learning dynamics beyond these K steps, therefore BMG proposes to bootstrap a target from the K-step parameters (in practice by continuing to optimise w.r.t. parameterised update rule for L-1 steps, and then taking a final step w.r.t. a fixed objective to ground the signal). (ii) MG updates are necessarily restricted to be within the geometry of the parameterised learning process.  In contrast, BMG introduces a matching function to measure the distance between the learners K-step parameters and bootstrapped target in an arbitrary (and hopefully more-suitable space).\n\nAfter framing the problem and BMG, the authors provide a discussion of the necessary conditions for BMG to guarantee performance improvements, though ultimately the presented algorithm is justified empirically using experiments on (i) a toy RL problem, (ii) the Atari RL test suite, (iii) multi-task few-shot adaptation on an image recognition task.  In all settings, BMG provides significant improvement over meta-learning baselines — most impressively achieving a new SOTA on Atari.  Moreover, key features of BMG are highlighted, including the ability to extend the meta-learning horizon without increasing the number of updates steps through which we must backpropagate, and that behavioural parameters outside of the update rule (specifically, epsilon in epsilon-greedy exploration) can also be meta-learned.", "summary_of_the_review": "Overall I believe this paper proposed an insightful and novel approach to addressing the stated limitations of typical MG approaches.  The authors do a good job of motivating the key innovations proposed and, given the significant research interest in MG’s in the past few years, it is reasonable to assume that the methodologies presented will be of broad interest.  Moreover, the experimental results are impressive and clearly demonstrate the improved performance of BMG and analyse where this comes from.  I do believe that the experimental results would be better served with more detailed descriptions of the problem settings and methodologies, however given the overall level of detail presented in the appendices I do not doubt there validity or significance.  Even so, for the stated reasons of novelty, interest and potential impact, I believe the paper is suitable for acceptance in the current form.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "LKbM0s4Gmxg", "reviewer": "Reviewer_N6Le", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- The proposed idea is simple.\n- The writing is clear.\n- To my knowledge, the proposed idea is novel yet concretely linked to many past works by virtue of generalizing them. The authors give the precise form to recover MG from BMG.\n- The proposed idea results in strong improvement of the STACX agent in Atari-57, resulting in a state-of-the-art result (caveat: for model-free agents; the gap to model-based agents remains large).\n- The authors demonstrate that the proposed idea is suitable in few-shot image classification, a popular application of meta-learning for few-shot learning.\n- The authors run informative ablation studies that support the intuition behind the benefit of BMG: resolving curvature and mitigating myopia.\n\nWeaknesses\n- Given that you say BMG is compatible with any update function (so long as it is differentiable in the meta-parameters), it would be nice to have some experiments on learned sequence model update rules (e.g. RNN). All current experiments use update rules with a fixed functional form.\n- I am not putting much weight on section 4 (\"Performance Guarantees\") given the gap between its assumptions and results vs. what is actually implemented, and the restriction to local optimization.\n\nMinor comments\n- p. 3, target bootstrap paragraph, 3rd line: are we missing a learning rate for the expression of the target?\n- p. 5, the actor-critic RL objective and Eq. 4: as-is, we are always minimizing policy entropy; is there a sign error?", "summary_of_the_paper": "This paper broadly considers meta-learning, a.k.a. bilevel optimization, across single-task, multi-task, supervised learning, and reinforcement learning settings. The authors aim to resolve two issues with the standard outer-loop gradient-based optimization of the meta-parameters (assuming a differentiable inner-loop): first, since the meta-learning objective is typically computed from learner parameters after applying up to $K$ inner-loop updates, the meta-optimization is myopic in that it does not optimize for further inner-loop improvement after $K$ steps; second, since the functional form of the learner's objective $f$ is used to drive the outer-loop updates, the meta-learning objective inherits the curvature of $f$. The main algorithmic contribution consists of a family of meta-learning objectives called bootstrapped meta-learning, in which meta-parameters are optimized to bring post-inner-loop learner parameters $x^{(K)}$ closer to a bootstrap target (which are also learner parameters) computed from $x^{(K)}$. The authors show that bootstrapped meta-learning generalizes the \"direct\" (my terminology) gradient-based meta-parameter optimization used in many previous works, recovering it when using specific choices for the bootstrap computation function and learner parameter matching function. With certain strong assumptions, the authors theoretically motivate the use of gradient-based bootstrap target functions for bootstrapped meta-learning in terms of optimization progress. The authors make several experimental contributions: they use bootstrapped meta-learning to achieve state-of-the-art model-free performance on Atari-57, demonstrate the viability of bootstrapped meta-learning in few-shot image classification on miniImageNet, and show that the more flexible, general form of bootstrapped meta-learning can enable meta-learning parameters that do not appear in the computation graph for the task objective, e.g. meta-learning the exploration rate of the behavior policy in $\\epsilon$-greedy $Q$-learning.", "summary_of_the_review": "I believe that the meta-learning community will find this paper interesting. It provides new insights into formulating meta-learning models and gives examples of such insights being applied to obtain empirical gains. The paper is well-written and features exemplary empirical execution.\n\nPOST-REBUTTAL: I have updated my score to 10 and confidence to 5. I think this paper should get an oral, if not best paper. It is the best in my batch and is arguably the best I have read all year. It may be the best paper I have ever peer-reviewed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FxRFF2uVWU", "reviewer": "Reviewer_gG1V", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**‌Originality**\n\nThe paper is original and novel, proposing a new algorithm to overcome two shortcomings of standard meta-optimization algorithms: curvature mismatch and limited evaluation.\n\n**Quality**\n\nThe paper is technically sound, and claims are backed by solid experiments in the reinforcement learning and multi-task meta-learning evaluation setting.\n\n**Clarity**\n\nThe paper is clear, however, the algorithm description in section 3 is very abstract. I think the paper would benefit from a running example and a dedicated section and pseudo-code describing the algorithm and how it can be instantiated in different experimental settings.\n\n**Significance**\n\nThe work is significant and will benefit the reinforcement and meta-learning community by addressing some of the limitation of the current meta-learning algorithms.\n\n**Limitations**\n\n- The theoretical analysis is limited to noiseless 1-step target updates.\n- The experimental evaluation in the multi-task meta-learning setting is limited to only compare with MAML on computer vision applications. \n\n**Questions to Authors**\n\n- Some engineering / handcrafting is still required by the machine learning practitioner to select what \"target\" the meta-learner is going to optimize, as well as the proper \"metric\" for the meta-learner to optimize for. Could the authors comment a bit about what heuristics they used when making these decisions? and whether the automation for this process is possible or not? \n- What would it take to extend the analysis beyond 1-step noiseless target updates?\n- How does the performance of BMG compare to alternative meta-learning algorithms like R2D2, Meta-OPT-net and prototypical networks? Have the authors experimented with other meta-learning benchmarks beyond image classification?\n\n**Minor Typos**\n\n- Abstract: \"show that metric\" -> \"show that a metric\"\n", "summary_of_the_paper": "‌‌The paper presents a new meta-learning algorithm to address two shortcomings of standard meta-optimization algorithms: curvature (the meta-learner's objective is typically constrained to the same type of geometry as the learner), and limited evaluation (the meta-objective is evaluated only with-in a K-step horizon, ignoring future learning dynamics). The proposed algorithm addresses these two issues by minimizing the distance to a bootstrapped target under a chosen metric. Empirically, the new algorithm  achieved a new state-of-the art for model-free agents on the Atari ALE benchmark and yielded gains in multi-task meta-learning. Theoretically, some guarantees on performance improvements are provided.\n", "summary_of_the_review": "‌‌\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses a meta-learning method which involves bilevel optimization. It is claimed that two limitations (myopia of MG and restricted consideration of geometry of search space) that most of existing methods have can be resolved by the MBG with a properly chosen pseudo-metric. The algorithm first bootstraps a target from the meta- learner, then optimizes the meta-learner by minimizing the distance to that target under a chosen pseudo-metric. The authors also establish conditions that guarantee performance improvements and show that metric can be sued to control meta-optimization. All the reviewers agree that the idea is interesting and experiments well support it. Authors did a good job in the rebuttal phase, resolving most of concerns raised by reviewers, leading that two of reviewers raised their score. While the current theoretical results are limited to a simple case where L=1$, the method is attractive for meta-learning community. All reviewers agree to champion this paper. Congratulations on a nice work.
Decision: Accept (Oral)