Title: Optimistic Meta-Gradients
Venue: NeurIPS 2023
Accepted: yes
Keywords: meta-learning

online optimisation

convex optimisation
Abstract: We study the connection between gradient-based meta-learning and convex optimisation. We observe that gradient descent with momentum is a special case of meta-gradients, and building on recent results in optimisation, we prove convergence rates for meta learning in the single task setting. While a meta-learned update rule can yield faster convergence up to constant factor, it is not sufficient for acceleration. Instead, some form of optimism is required. We show that optimism in meta-learning can be captured through the recently proposed Bootstrapped Meta-Gradient (Flennerhag et. al., 2022) method, providing deeper insight into its underlying mechanics.

Optimistic Meta-Gradients

Abstract

We study the connection between gradient-based meta-learning and convex optimisation. We observe that gradient descent with momentum is a special case of meta-gradients, and building on recent results in optimisation, we prove convergence rates for meta-learning in the single task setting. While a meta-learned update rule can yield faster convergence up to constant factor, it is not sufficient for acceleration. Instead, some form of optimism is required. We show that optimism in meta-learning can be captured through the recently proposed Bootstrapped Meta-Gradient [9] method, providing deeper insight into its underlying mechanics.

1 Introduction

In meta-learning, a learner is using a parameterised algorithm to adapt to a given task. The parameters of the algorithm are then metalearned by evaluating the learner's resulting performance [25, 10, 2]. As such, meta-learning features a complex interaction between the learner and the meta-learner. The learner's problem is to minimize the expected loss f of a stochastic objective by adapting its parameters x ∈ R n . The learner has an update rule ϕ at its disposal that generates new parameters x t = x t -1 + ϕ ( x t -1 , w t ) ; we suppress data dependence to simplify notation. A simple example is when ϕ represents gradient descent with w t = η its step size, that is ϕ ( x t -1 , η ) = -η ∇ f ( x t -1 ) [17, 26]; several works have explored meta-learning other aspects of a gradientbased update rule [6, 21, 7, 30, 31, 9, 15, 22]. ϕ need not be limited to a gradient-based update, it can represent some algorithm implemented within a Neural Network [25, 11, 1, 29].

The meta-learner's problem is to optimise the meta-parameters w t to yield effective updates.

0.0

0.5

1.0

1.5

2.0

2.5

3.0

1e6

50

55

60

65

70

75

Top-1 Test Accuracy (%)

SGD

Standard meta-learning

Optimistic meta-learning

In a typical (gradient-based) meta-learning setting, it does so by treating x t as a function of w . Let h t , defined by h t ( w ) = f ( x t -1 + ϕ ( x t -1 , w )) , denote the learner's post-update performance as a function of w . The learner and the meta-learner co-evolve according to

where Dϕ ( x, w ) denotes the Jacobian of ϕ with respect to w . The nested structure between these two updates makes it challenging to analyse meta-learning, in particular it depends heavily on the properties of the Jacobian. In practice, ϕ is highly complex and so Dϕ is almost always intractable. For this reason, the only theoretical results we are aware of specialise to the multi-task setting, where the learner must adapt to a new task f t . Acceleration in this setup is driven by the tasks similarity. That is, if all tasks are sufficiently similar, a meta-learned update can accelerate convergence [14]. However, these results do not yield acceleration in the absence of a task distribution to the best of our knowledge.

This paper provides an alternative view. We study the classical convex optimisation setting of approximating the minimiser min x f ( x ) . We observe that setting the update rule equal to the gradient, i.e. ϕ : ( x, w ) ↦→ w ∇ f ( x ) , recovers gradient descent. Similarly, we show in Section 4 that ϕ can be chosen to recover gradient descent with momentum. This offers another view of meta-learning as a non-linear transformation of classical optimisation. An implication thereof is that task similarity is not necessary condition for improving the rate of convergence via meta-learning. While there is ample empirical evidence to that effect [30, 31, 9, 16], we are only aware of theoretical results in the special case of meta-learned step sizes [17, 26].

Given a function f that is convex with Lipschitz smooth gradients, meta-learning improves the rate of convergence by a multiplicative factor of λ to O ( λ/T ) via the smoothness of the update rule. To achieve accelerated convergence, O (1 /T 2 ) , some form of optimism is required, typically in the form of a prediction of the next gradient. We consider optimism with meta-learning in the convex setting and prove accelerated rates of convergence, O ( λ/T 2 ) . Again, meta-learning affects these bounds by a multiplicative factor. Our main contributions are as follows:

We show that meta-gradients contain gradient descent with momentum (Heavy Ball [23]; Section 4) and Nesterov Acceleration [20] as special cases (Section 5).

We show that gradient-based meta-learning can be understood as a non-linear transformation of an underlying optimisation method (Section 4).

We establish rates of convergence for meta-learning in the convex setting (Section 4).

We show that optimism can be expressed through the recently proposed Bootstrapped MetaGradient method [BMG; 9]. Our analysis provides a first proof of convergence for BMG and highlights the underlying mechanics that enable faster learning with BMG (Section 6).

2 Meta-Learning as Convex Optimisation

Problem definition. This section defines the problem studied in this paper and introduces our notation (see Appendix A, Table 1). Let f : X → R be a proper and convex function. The problem of interest is to approximate the global minimum min x ∈X f ( x ) . We assume a global minimiser exists and is unique, defined by

We assume that X ⊆ R n is a closed, convex and non-empty set. f is differentiable and has Lipschitz smooth gradients with respect to a norm ‖ · ‖ , meaning that there exists L ∈ (0 , ∞ ) such that ‖∇ f ( x ) -∇ f ( y ) ‖ ∗ ≤ L ‖ x -y ‖ for all x, y ∈ X , where ‖ · ‖ ∗ is the dual norm of ‖ · ‖ . We consider the noiseless setting for simplicity; our results carry over to the stochastic setting by replacing the key online-to-batch bound used in our analysis by its stochastic counterpart [13].

Algorithm. Let [ T ] = { 1 , 2 , . . . , T } . We are given weights { α t } T t =1 , each α t > 0 , and an initialisation (¯ x 0 , w 1 ) ∈ X ×W . At each time t ∈ [ T ] , an update rule ϕ : X ×W → X generates the update x t = ϕ (¯ x t -1 , w t ) , where W ⊆ R m is closed, convex, and non-empty. We discuss ϕ momentarily. The algorithm maintains the online average

where x 1: t = ∑ t s =1 α s x s , α 1: t = ∑ t s =1 α s , and ρ t = α t /α 1: t . Our goal is to establish conditions under which { ¯ x t } T t =1 converges to the minimiser x ∗ . While this moving average is not always used in practical applications, it is required for accelerated rates in online-to-batch conversion [27, 3, 13].

Convergence depends on how meta-parameters w t are chosen. The meta-learner faces a sequence of losses h t : W → R defined by the composition h t ( w ) = f ((1 -ρ t )¯ x t -1 + ρ t ϕ (¯ x t -1 , w )) . As such, the meta-learner is facing an online optimization, which we model under Follow-The-RegularizedLeader (FTRL; reviewed in Section 3): given w 0 , each w t is chosen according to

Note that Eq. 3 subsumes the standard meta-gradient; if ‖ · ‖ is the Euclidean norm, an interior solution yields w t +1 = w t -α t β ∇ h t ( w t ) . It is straightforward to extend Eq. 3 to account for meta-updates that use AdaGrad-like [5] acceleration by altering the norms (see [12]).

Update rule. It is not possible to prove convergence outside of the convex setting, since ϕ may reach a local minimum, where local changes to w do not yield better updates in x , yet the x sequence is not converging. Convexity means that each h t must be convex, which requires that ϕ is affine in w (but may vary non-linearly in x ). Wealso assume that ϕ is smooth with respect to ‖·‖ , in the sense that it has bounded norm; for all x ∈ X and all w ∈ W we assume that there exists λ ∈ (0 , ∞ ) for which

Limitations. Our analysis makes relatively strict assumptions. Most meta-learning systems are not affine in w . A notable case where our assumptions to hold is meta-learned step-sizes or preconditioning matrices. For other update rules, our analysis holds up to first-order Taylor approximation error. We carry out experiments in Section 7 to empirically verify our theoretical insights.

3 Preliminaries: Online Convex Optimisation

In this section, we present analytical tools from the optimisation literature that we build upon. In a standard optimisation setting, there is no update rule ϕ ; instead, the iterates x t are generated by a gradient-based algorithm, akin to Eq. 3. Our problem setting reduces to standard optimisation if ϕ is defined by ϕ : ( x, w ) ↦→ w , in which case x t = w t . In this paper, we use batch-to-online conversion as our analytical tool. This strategy treats the iterates x 1 , x 2 , . . . as generated by an online learning algorithm, for which we can obtain a regret bound. This regret bound can then be turned into a convergence rate, detailed momentarily.

Online Optimisation. In online convex optimisation [32], a learner is given a convex decision set U and faces a sequence of convex loss functions { α t f t } T t =1 . At each time t ∈ [ T ] , it must make a prediction u t prior to observing α t f t , after which it incurs a loss α t f t ( u t ) and receives a signal-either α t f t itself or a (sub-)gradient of α t f t ( u t ) . The learner's goal is to minimise regret , R ( T ) := ∑ T t =1 α t ( f t ( u t ) -f t ( u )) , against a comparator u ∈ U . An important property of a convex function f is f ( u ′ ) -f ( u ) ≤ 〈∇ f ( u ′ ) , u ′ -u 〉 . Hence, the regret is largest under linear losses: ∑ T t =1 α t ( f t ( u t ) -f t ( u )) ≤ ∑ T t =1 α t 〈∇ f t ( u t ) , u t -u 〉 . For this reason, it is sufficient to consider regret under linear loss functions. An algorithm has sublinear regret if lim T →∞ R ( T ) /T = 0 .

FTRL & AO-FTRL. The meta-update in Eq. 3 is an instance of Follow-The-Regularised-Leader (FTRL) under linear losses. In Section 6, we show that BMG is an instance of the Adaptive-Optimistic

FTRL (AO-FTRL) [24, 19, 13, 28]. In AO-FTRL, we have a strongly convex regulariser ‖ · ‖ 2 . AOFTRL sets the first prediction u 1 to minimise ‖ · ‖ 2 . Given linear losses { g s } t -1 s =1 and learning rates { β t } T t =1 , each β t > 0 , the algorithm proceeds according to

where each ˜ g t is a 'hint' that enables optimistic learning [24, 19]; setting ˜ g t = 0 recovers the original FTRL algorithm. The goal of a hint is to predict the next loss vector g t ; if the predictions are accurate AO-FTRL can achieve lower regret than its non-optimistic counter-part. Since ‖ · ‖ 2 is strongly convex, FTRL is well defined in the sense that the minimiser exists, is unique, and finite [18]. The regret of FTRL and AO-FTRL against any comparator u ∈ U can be upper-bounded by

Hence, hints that predict g t well can reduce the regret substantially. Without hints, FTRL can guarantee O ( √ T ) regret (for non strongly convex loss functions). However, [4] show that under linear losses, if hints are weakly positively correlated-defined as 〈 g t , ˜ g t 〉 ≥ glyph[epsilon1] ‖ g t ‖ 2 for some glyph[epsilon1] > 0 -then the regret guarantee improves to O (log T ) , even for non strongly-convex loss functions. We believe optimism provides an exciting opportunity for novel forms of meta-learning. Finally, we note that these regret bounds (and hence our analysis) can be extended to stochastic optimisation [19, 12].

Online-to-batch conversion. The main idea behind online to batch conversion is that, for f convex, Jensen's inequality gives f (¯ x T ) -f ( x ∗ ) ≤ ∑ T t =1 α t 〈∇ f ( x t ) , x t -x ∗ 〉 /α 1: T . That is, the suboptimality gap of the average iterate ¯ x T can be bounded by the regret in the sequence x 1 , x 2 , . . . , x T . Applying this bound naively yields O (1 /T ) rate of convergence. In recent work, [3] shows that one achieve tighter bounds by instead querying the gradient at the average iterate, f (¯ x T ) -f ( x ∗ ) ≤ ∑ T t =1 α t 〈∇ f (¯ x t ) , x t -x ∗ 〉 /α 1: T . A tighter bound means a faster rate of convergence. Recently, [13] tightened the analysis further and proved that the sub-optimality gap can be bounded by

were we define R x ( T ) := ∑ T t =1 α t 〈∇ f (¯ x t ) , x t -x ∗ 〉 as the regret of the sequence { x t } T t =1 against the comparator x ∗ . With this machinery in place, we now turn to deriving our main results.

4 Meta-Gradients without Optimism

The main difference between classical optimisation and meta-learning is the introduction of the update rule ϕ . To see how this acts on optimisation, consider two special cases. If the update rule just return the gradient, ϕ = ∇ f , Eq. 3 reduces to gradient descent (with averaging). This inductive bias is fixed and does not change with experience, so acceleration is not possible: the rate of convergence is O (1 / √ T ) [28]. The other extreme is an update rule that only depends on the meta-parameters, ϕ ( x, w ) = w . Here, the meta-learner has ultimate control and selects the next update without constraints. The only relevant inductive bias is contained in w . To see how this inductive bias is formed, suppose ‖ · ‖ = ‖ · ‖ 2 so that Eq. 3 yields w t +1 = w t -α t ρ t β ∇ f (¯ x t ) (assuming an interior solution). Combining this with the moving average in Eq. 2, we may write the learner's iterates as

where each ˜ ρ t = ρ t 1 -ρ t -1 ρ t -1 and ˜ β t = α t ρ t β . Setting β = 1 / (2 L ) and each α t = t yields ˜ ρ t = t -2 t +1 and ˜ β t = t/ (4( t +1) L ) , which recovers Polyak's canonical Heavy-Ball method [23]. Hence, gradient descent with momentum is a special case of meta-learning under the update rule ϕ : ( x, w ) ↦→ w . Because Heavy Ball carries momentum from past updates, it can encode a model of the learning dynamics that leads to faster convergence, on the order O (1 /T ) . The implication of this is that the dynamics of meta-learning are fundamentally momentum-based and thus learns an inductive bias in the same cumulative manner. This similarity carries in our theoretical analysis, which we turn to next.

The central challenge in applying the bound in Eq. 6 to Algorithm 2 is that the iterates x t are generated under the update rule ϕ . Hence, we cannot apply standard regret bounds directly. Instead, observe that

The first term in the final inequality can be understood as the regret under convex losses glyph[lscript] t ( · ) = α t 〈∇ f (¯ x t ) , ϕ (¯ x t -1 , · ) 〉 . Since ϕ is affine, glyph[lscript] t is convex and thus this regret can be upper-bounded by linearising the losses. The linearisation reads 〈 Dϕ (¯ x t -1 , w t ) T ∇ f (¯ x t ) , ·〉 , which is identical the linear losses 〈∇ h t ( w t ) , ·〉 faced by the meta-learner in Eq. 3. In other words, the regret component can be upper-bounded by the of the meta-learner,

The regret of the learner is therefore upper bounded by

The last term captures the difference in comparator capacity and more specifically the amount of regret they can inflict. If the comparator w ∗ has more power than that of x ∗ , it can accumulate a lower total loss, in which case this term will be negative, allowing us to discard it. Intuitively, the comparator x ∗ is non-adaptive. It must make one choice x ∗ and suffer the average loss. In contrast, the comparator w ∗ becomes adaptive under the update rule; it can only choose one w ∗ , but on each round it plays ϕ (¯ x t -1 , w ∗ ) . If ϕ is sufficiently flexible, this gives the comparator w ∗ more power than x ∗ , and hence it can force the meta-learner to suffer greater regret. When this is the case, we say that regret is retained when moving from x ∗ to w ∗ . As long as ϕ is not degenerate, this is typically easy to satisfy by making W sufficiently large.

Definition 1. Given f , { α t } T t =1 , and { x t } T t =1 , an update rule ϕ : X ×W → X preserves regret if there exists a comparator w ∈ W that satisfies

If such w exists, let w ∗ denote the comparator with smallest norm ‖ w ‖ .

Lemma 1. Given f , { α t } T t =1 , and { x t } T t =1 , if ϕ preserves regret, then

Proof: Appendix D. From Eq. 8, it is clear that for ϕ to retain regret, it must admit a parameterisation that correlates negatively with the gradient. In other words, ϕ must be able to behave as a gradient descent algorithm. However, this must not hold on every step, only sufficiently often. For instance, ϕ ( x, · ) affine can be made to satisfy this condition if X and W are chosen appropriately.

Theorem 1. Let ϕ preserve regret and satisfy the assumptions in Section 2. Then

If x ∗ is a global minimiser of f , setting α t = 1 and β = 1 λL yields f (¯ x T ) -f ( x ∗ ) ≤ λL diam( W ) T .

Proof: Appendix D. Compared to Heavy Ball, meta-learning introduces a constant λ that captures the smoothness of the update rule. Hence, while meta-learning does not achieve better scaling in T through ϕ , it can improve upon classical optimisation by a constant factor if λ < 1 .

That meta-learning can improve upon momentum is borne out experimentally. We consider the problem of minimizing an ill-conditioned convex quadratic and compare standard momentum to a version with meta-learned step-size, i.e. ϕ : ( x, w ) ↦→ w glyph[circledot]∇ f ( x ) , where glyph[circledot] is the Hadamard product. We find that introducing a non-linearity ϕ leads to a sizeable improvement in the rate of convergence. See Section 7.1 for further details.

5 Meta-Gradients with Optimism

It is well known that minimizing a smooth convex function admits convergence rates of O (1 /T 2 ) . Our analysis of meta-learning does not achieve these rates. Previous work indicate that we should not expect it to either; to achieve the theoretical lower-limit of O (1 /T 2 ) , some form of optimism (c.f. Section 3) is required. A typical form of optimism is to predict the next gradient. This is how Nesterov Acceleration operates [20], and is the reason for its O (1 /T 2 ) convergence guarantee.

From our perspective, meta-learning is a non-linear transformation of the iterate x . Hence, we should expect optimism to play a similarly crucial role. Formally, optimism comes in the form of hint functions { ˜ g t } T t =1 , each ˜ g t ∈ R m , that are revealed to the meta-learner prior to selecting w t +1 . These hints give rise to Optimistic Meta-Learning (OML) via meta-updates

If the hints are accurate, meta-learning with optimism can achieve an accelerated rate of O ( ˜ λ/T 2 ) , where ˜ λ is a constant that characterises the smoothness of ϕ , akin to λ . Again, we find that metalearning behaves as a non-linear transformation of classical optimism and its rate of convergence is governed by the geometry it induces.

For a complete description, see Algorithm 4. These updates do not correspond to the typical metaupdate in Algorithm 1; however, we show momentarily that they can be interpreted as the targets in the BMG method, summarised in Algorithm 3. Before turning to BMG, we establish that optimistic meta-learning in the convex setting does indeed yield acceleration.

Theorem 2. Let ϕ preserve regret and assume Algorithm 4 satisfy the assumptions in Section 2. Then

Proof; Appendix D. From Theorem 2, it is clear that if ˜ g t is a good predictor of Dϕ (¯ x t -1 , w t ) T ∇ f (¯ x t ) , then the positive term in the summation can be cancelled by the negative term. In a classical optimisation setting, Dϕ = I n , and hence it is easy to see that simply

choosing ˜ g t to be the previous gradient is sufficient to achieve the cancellation [13]. Indeed, this choice gives us Nesterov's Accelerated rate [28]. The upshot of this is that we can specialise Algorithm 4 to capture Nesterov's Accelerated method by choosing ϕ : ( x, w ) ↦→ w -as in the reduction to Heavy Ball-and setting the hints to ˜ g t = ∇ f (¯ x t -1 ) . Hence, while the standard meta-update without optimism contains Heavy Ball as a special case, the optimistic meta-update contains Nesterov Acceleration as a special case.

In the meta-learning setting, Dϕ is not an identity matrix, and hence the best targets for meta-learning are different. Naively choosing ˜ g t = Dϕ (¯ x t -1 , w t ) T ∇ f (¯ x t -1 ) would lead to a similar cancellation, but this is not allowed. At iteration t , we have not computed w t when ˜ g t is chosen, and hence Dϕ (¯ x t -1 , w t ) is not available. The nearest term that is accessible is Dϕ (¯ x t -2 , w t -1 ) .

Corollary 1. Let each ˜ g = Dϕ (¯ x , w ) T ∇ f (¯ x ) . Assume that ϕ satisfies

for all x ′′ , x ′ , x ∈ X and w,w ′ ∈ W , for some ˜ λ > 0 . If each α t = t and β t = t -1 2 t ˜ λL , then f (¯ x T ) -f ( x ∗ ) ≤ 4 ˜ λL diam( W ) T 2 -1 .

These predictions hold empirically in a non-convex setting. We train a 50-layer ResNet using either SGD with a fixed learning rate, or an update rule that adapts a per-parameter learning rate online, ϕ : ( x, w ) ↦→ w glyph[circledot]∇ f ( x ) . We compare the standard meta-learning approach without optimism to optimistic meta-learning. Figure 1 shows that optimism is critical for meta-learning to achieve acceleration, as predicted by theory (experiment details in Appendix C).

6 Bootstrapped Meta-Gradients as a form of Optimism

Given Theorem 2, it is of interest to study practical ways of implementing optimism in meta-learning. We study a recently proposed variant of meta-gradients, Bootstrapped Meta-Gradients (BMG) [8]. Informally, instead of directly minimising the loss f , the meta-objective in BMG is the distance between the meta-learner's output x t and a desired target z t . The target is computed by unrolling the meta-learner for a further number of steps, thus implicitly embodying a form of optimism, before a gradient step is taken: z t = x t + ϕ ( x t , w t ) -∇ f ( x t + ϕ ( x t , w t )) . This encodes optimism via ϕ because it encourages the meta-learner to build up momentum (i.e. to accumulate past updates). To see this formally, we turn to AO-FTRL. First, we provide a more general definition of BMG. Let µ : X → R be a convex distance generating function and define the Bregman Divergence B µ : R n × R n → R by

Given initial condition ( x 0 , w 1 ) , the BMG updates proceed according to

where d t : R n → R is defined by d t ( w ) = B µ z t ( x t -1 + ϕ ( x t -1 , w t )) ; each z t ∈ R n is referred to as a target. See Algorithm 3 for an algorithmic summary. To show how this relates to AO-FTRL, let µ = f . In this case, the BMG update reads w t +1 = w t -β t Dϕ ( x t -1 , w t ) T ( ∇ f ( z t ) -∇ f ( x t )) . We can obtain these updates via our convex framework (i.e. Algorithm 4) by setting ˜ g t +1 = ∇ f ( z t ) . In this case, we have that (Corollary 3, Appendix E) AO-FTRL reduces to

where ξ t = β t α t Dϕ (¯ x t -2 , w t -1 ) T ∇ f ( z t -1 ) denotes an error correction term that removes the previous target. This error correction term is theoretically important for stability [13], as the accumulation of hints can otherwise dominate the true signal. That the original BMG does not feature this error term may explain the instabilities the authors observed when setting too aggressive targets [8]. Since Algorithm 3 does not average its iterates-while Algorithm 4 does-we see that these updates (ignoring ξ t ) are identical up to scalar coefficients (that can be controlled for by scaling each β t and each ˜ g t +1 accordingly). Our next results present formulas for constructing targets in BMG or hints in AO-FTRL so that the two commute.

Theorem 3. Targets in Algorithm 3 and hints in algorithm 4 commute in the following sense. BMG → AO-FTRL. Let BMG targets { z t } T t =1 by given. A sequence of hints { ˜ g } T t =1 can be constructed recursively by

so that interior updates for Algorithm 4 are given by

AO-FTRL → BMG. Conversely, assume a sequence { ˜ y t } T t =1 are given, each ˜ y t ∈ R n . If µ strictly convex, a sequence of BMG targets { z t } T t =1 can be constructed recursively by

so that BMG updates in Eq. 10 are given by

where each ˜ g t +1 is given by

Proof; see Appendix E. As an immediate consequence of this, we can apply our optimistic metagradient analysis (Theorem 2) to BMG to obtain a rate of convergence. This is captured in the following corollary.

Corollary 2. Let each ˜ g t +1 = Dϕ (¯ x t -1 , w t ) T ˜ y t +1 , for some ˜ y t +1 ∈ R n . If each ˜ y t +1 is a better predictor of the next gradient than ∇ f (¯ x t -1 ) , in the sense that

then Algorithm 4 guarantees convergence at a rate O ( ˜ λ/T 2 ) .

In other words, for certain choices of targets, BMG yields accelerated rates of convergence.

7 Experiments

In this section, we detail experiments borne out to test our theoretical predictions. Section 7.1 tests predictions about meta-gradients without optimism, by comparing meta-learned variants of standard optimisers to their non-meta-learned counterparts. Section 7.2 test predictions about optmistic meta-gradients, by comparding an optimistic meta-gradient approach to gradient descent.

7.1 Convex Quadratic Experiments

Loss function. We consider the problem of minimising a convex quadratic loss functions f : R 2 → R of the form f ( x ) = x T Qx for some Q that is sampled such that it is ill-conditioned (see Appendix B for details).

Protocol. Given that the solution is always (0 , 0) , this experiment revolves around understanding how different algorithms deal with curvature. Given symmetry in the solution and ill-conditioning, we fix the initialisation to x 0 = (4 , 4) and run each algorithm for 100 iterations. For each Q and each algorithm, we sweep over the learning rate, decay rate, and the initialization of w (see Table 2 for values) and report results for the best performing hyper parameters.

Results. We report the learning curves for the best hyper-parameter choice for 5 randomly sampled problems in the top row of Figure 2 (columns correspond to different Q). We also study the sensitivity of each algorithm to the learning rate in the bottom row Figure 2. For each learning rate, we report the cumulative loss during training. While baselines are relatively insensitive to hyper-parameter choice, meta-learned improve for certain choices, but are never worse than baselines.

7.2 Imagenet Experiments

Protocol. We train a 50-layer ResNet following a standard protocol (Appendix C) with SGD as the baseline optimiser. We compare SGD to a variant that meta-learns an element-wise learning rate online, i.e. ( x, w ) ↦→ w glyph[circledot]∇ f ( x ) . We sweep over the learning rate (for SGD) or meta-learning rate and report results for the best hyper-parameter over three independent runs.

0

20

40

60

80

100

Iterations

10

28

10

16

10

4

Loss

0

20

40

60

80

100

Iterations

0

20

40

60

80

100

Iterations

0

20

40

60

80

100

Iterations

0

20

40

60

80

100

Iterations

Momentum

Meta-Momentum

AdaGrad

Meta-AdaGrad

0.1

0.3

0.5

0.7

0.9

0.990.9999 3.0

5.0

Learning rate

10

22

10

12

10

2

Loss

0.1

0.3

0.5

0.7

0.9

0.990.9999 3.0

5.0

Learning rate

0.1

0.3

0.5

0.7

0.9

0.990.9999 3.0

5.0

Learning rate

0.1

0.3

0.5

0.7

0.9

0.990.9999 3.0

5.0

Learning rate

0.1

0.3

0.5

0.7

0.9

0.990.9999 3.0

5.0

Learning rate

Momentum

Meta-Momentum

AdaGrad

Meta-AdaGrad

Standard meta-learning. In the standard meta-learning setting, we apply the update rule once before differentiating w.r.t. the meta-parameters. That is, the meta-update takes the form w t +1 = w t -β ∇ h t ( w t ) , where h t = f ( x t + w t glyph[circledot]∇ f ( x t )) . Because the update rule is linear in w , we can compute the meta-gradient analytically: ∇ h t ( w t ) = ∇ w f ( x + ϕ ( x, w )) = Dϕ ( x, w ) T ∇ f ( x ′ ) = ∇ f ( x ) glyph[circledot] ∇ f ( x ′ ) , where x ′ = x + ϕ ( x, w ) . Hence, we can compute the meta-updates in Algorithm 1 manually as w t +1 = max { w t -β ∇ f ( x t ) glyph[circledot]∇ f ( x t +1 ) , 0 . } , where we introduce the max operator on an elementwise basis to avoid negative learning rates. Empirically, this was important to stabilize training.

Optimistic meta-learning. For optimistic meta-learning, we proceed much in the same way, but include a gradient prediction ˜ g t +1 . For our prediction, we use the previous gradient, ∇ f ( x t +1 ) , as our prediction. This yields meta-updates of the form w t +1 = max { w t -β ∇ f ( x t +1 ) glyph[circledot] ( ∇ f ( x t +1 ) + ∇ f ( x t )) - ∇ f ( x t ) glyph[circledot] ∇ f ( x t ) , 0 . } , .

Results. We report Top-1 accuracy on the held-out test set as a function of training steps in Figure 1. Tuning the learning rate does not yield any statistically significant improvements under standard meta-learning. However, with optimistic meta-learning, we obtain a significant acceleration as well as improved final performance, increasing the mean final top-1 accuracy from 72% to 75% .

8 Conclusion

This paper explores a connection between convex optimisation and meta-learning. We construct an algorithm for convex optimisation that aligns as closely as possible with how meta-learning is done in practice. Meta-learning introduces a transformation and we study the effect this transformation has on the rate of convergence. We find that, while a meta-learned update rule cannot generate a better dependence on the horizon T , it can improve upon classical optimisation up to a constant factor.

An implication of our analysis is that for meta-learning to achieve acceleration, it is important to introduce some form of optimism. From a classical optimisation point of view, such optimism arises naturally by providing the meta-learner with hints. If hints are predictive of the learning dynamics these can lead to significant acceleration. We show that the recently proposed BMG method provides a natural avenue to incorporate optimism in practical application of meta-learning. Because targets in BMG and hints in optimistic online learning commute, our results provide first rigorous proof of convergence for BMG, while providing a general condition under which optimism in BMG yields accelerated learning.

References

M. Andrychowicz, M. Denil, S. Gómez, M. W. Hoffman, D. Pfau, T. Schaul, and N. de Freitas. Learning to Learn by Gradient Descent by Gradient Descent. In Advances in Neural Information Processing Systems , 2016.

Y. Bengio, S. Bengio, and J. Cloutier. Learning a Synaptic Learning Rule . Université de Montréal, Département d'informatique et de recherche opérationnelle, 1991.

A. Cutkosky. Anytime Online-to-Batch, Optimism and Acceleration. In International Conference on Machine Learning , 2019.

O. Dekel, A. Flajolet, N. Haghtalab, and P. Jaillet. Online learning with a hint. In Advances in Neural Information Processing Systems , 2017.

J. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research , 12(61):2121-2159, 2011.

C. Finn, P. Abbeel, and S. Levine. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. In International Conference on Machine Learning , 2017.

S. Flennerhag, P. G. Moreno, N. D. Lawrence, and A. Damianou. Transferring Knowledge across Learning Processes. In International Conference on Learning Representations , 2019.

S. Flennerhag, Y. Schroecker, T. Zahavy, H. van Hasselt, D. Silver, and S. Singh. Bootstrapped Meta-Learning. arXiv preprint arXiv:2109.04504 , 2021.

S. Flennerhag, Y. Schroecker, T. Zahavy, H. van Hasselt, D. Silver, and S. Singh. Bootstrapped Meta-Learning. In International Conference on Learning Representations , 2022.

G. E. Hinton and D. C. Plaut. Using Fast Weights to Deblur Old Memories. In Cognitive Science Society , 1987.

S. Hochreiter, A. S. Younger, and P. R. Conwell. Learning To Learn Using Gradient Descent. In International Conference on Artificial Neural Networks , 2001.

P. Joulani, A. György, and C. Szepesvári. A modular analysis of adaptive (non-) convex optimization: Optimism, composite objectives, and variational bounds. Journal of Machine Learning Research , 1:40, 2017.

P. Joulani, A. Raj, A. Gyorgy, and C. Szepesvári. A Simpler Approach to Accelerated Optimization: Iterative Averaging Meets Optimism. In International Conference on Machine Learning , 2020.

M. Khodak, M.-F. F. Balcan, and A. S. Talwalkar. Adaptive Gradient-Based Meta-Learning Methods. In Advances in Neural Information Processing Systems , 2019.

L. Kirsch, S. van Steenkiste, and J. Schmidhuber. Improving Generalization in Meta Reinforcement Learning Using Learned Objectives. arXiv preprint arXiv:1910.04098 , 2019.

J. Luketina, S. Flennerhag, Y. Schroecker, D. Abel, T. Zahavy, and S. Singh. Meta-gradients in non-stationary environments. In ICLR Workshop on Agent Learning in Open-Endedness , 2022.

A. R. Mahmood, R. S. Sutton, T. Degris, and P. M. Pilarski. Tuning-Free Step-Size Adaptation. In ICASSP , 2012.

H. B. McMahan. A survey of algorithms and analysis for adaptive online learning. The Journal of Machine Learning Research , 18(1):3117-3166, 2017.

M. Mohri and S. Yang. Accelerating Online Convex Optimization via Adaptive Prediction. In International Conference on Artificial Intelligence and Statistics , 2016.

Y. E. Nesterov. A method for solving the convex programming problem with convergence rate o (1/kˆ 2). In Dokl. akad. nauk Sssr , volume 269, pages 543-547, 1983.

A. Nichol, J. Achiam, and J. Schulman. On First-Order Meta-Learning Algorithms. arXiv preprint ArXiv:1803.02999 , 2018.

J. Oh, M. Hessel, W. M. Czarnecki, Z. Xu, H. P. van Hasselt, S. Singh, and D. Silver. Discovering Reinforcement Learning Algorithms. In Advances in Neural Information Processing Systems , volume 33, 2020.

B. T. Polyak. Some Methods of Speeding up the Convergence of Iteration Methods. USSR Computational Mathematics and Mathematical Physics , 4(5):1-17, 1964.

S. Rakhlin and K. Sridharan. Optimization, Learning, and Games with Predictable Sequences. In Advances in Neural Information Processing Systems , 2013.

J. Schmidhuber. Evolutionary Principles in Self-Referential Learning . PhD thesis, Technische Universität München, 1987.

T. van Erven and W. M. Koolen. MetaGrad: Multiple Learning Rates in Online Learning. In Advances in Neural Information Processing Systems , 2016.

J.-K. Wang and J. Abernethy. Acceleration through Optimistic No-Regret Dynamics. arXiv preprint arXiv:1807.10455 , 2018.

J.-K. Wang, J. Abernethy, and K. Y. Levy. No-regret dynamics in the fenchel game: A unified framework for algorithmic convex optimization. arXiv preprint arXiv:2111.11309 , 2021.

J. X. Wang, Z. Kurth-Nelson, D. Tirumala, H. Soyer, J. Z. Leibo, R. Munos, C. Blundell, D. Kumaran, and M. Botvinick. Learning to Reinforcement Learn. In Annual Meeting of the Cognitive Science Society , 2016.

Z. Xu, H. P. van Hasselt, and D. Silver. Meta-Gradient Reinforcement Learning. In Advances in Neural Information Processing Systems , 2018.

T. Zahavy, Z. Xu, V. Veeriah, M. Hessel, J. Oh, H. P. van Hasselt, D. Silver, and S. Singh. A Self-Tuning Actor-Critic Algorithm. In Advances in Neural Information Processing Systems , volume 33, 2020.

M. Zinkevich. Online Convex Programming and Generalized Infinitesimal Gradient Ascent. In International Conference on Machine Learning , 2003.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies a connection between optimization and meta-learning. For the case of a single task, it shows an equivalence between GD with momentum and GBML, and another equivalence between GD with Nesterov acceleration and the recent Bootstrapped Meta-Gradient algorithm. Theoretical analyses are done for these algorithm, showing that GBML speeds up convergence by a constant factor but is not able to improve on the $O(1/T)$ rate, while BMG is able to improve the rate to $O(1/T^2)$. Experiments are done on quadratic minimization and ImageNet image classification that confirm the theory.\n\nSTRENGTHS:\n1. The mathematical argument in the paper is very clearly presented, e.g. the authors roughly outline proof techniques.\n2. Overall the paper appears to be sound, although I did not check the math. Experiments were done on both \"toy\" examples and a more realistic ImageNet problem.\n3. As far as I know, the connection between BMG and GD with Nesterov acceleration is novel.\n\nWEAKNESSES:\nIn my opinion, the major weakness of the paper is the fact that it only considers the single task setting. However, one of the major motivations behind meta-learning is that information may be shared between tasks in order to improve convergence speeds for all tasks, and the multi-task setting is arguably more popular in the literature. It would greatly strengthen the work to consider the theory for a multi-task setting, although I understand that it might be more appropriate for future work.\n\nMinor:\n1. The objective function is convex is a fairly strong assumption.\n\nQUESTIONS:\nIs it fair to search over the range of hyperparameters in the experiments in Section 7.1? Usually when this is done the final results are provided for a held-out test set (a new quadratic function in this case), but this does not seem to be the case here.\n\nLIMITATIONS:\nThe authors do not have a broader impact section, and I agree since this work discusses optimization methods. However, they do not discuss the limitations/weaknesses of their work, which I think should be added to the final draft.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper shed an interesting perspective on meta-learning by studying the connection between gradient-based meta-learning and convex optimisation in the single task setting. It shows that meta-gradients contain gradient descent with momentum and Nesterov Acceleration as special cases. Furthermore, gradient-based meta-learning can be understood as a non-linear transformation of an underlying optimisation method. The authors establish the rates of convergence for meta-gradients in convex settings. For meta-learning to achieve acceleration of convergence, some form of optimization is needed. Specifically, it provides the first rigorous proof of BMG with acceleration rate of convergence, and highlights the underlying mechanics that enable faster learning with BMG.\n\nSTRENGTHS:\nThis paper provides us a novel view on meta-learning by studying the connection between gradient-based meta-learning and convex optimisation. It reveals that gradient descent with momentum and Nesterov Acceleration are special cases of gradient-based meta-learning with different update rules. This paper is of high technical quality. Based on the new understanding for meta-learning as a non-linear transformation of an underlying optimisation method, it establishes theoretical analysis and proof of the convergence rates of meta-gradients in convex settings. The theoretical analysis has insightful implications that we are able to further accelerate the rate of convergence with some form of optimization. As a result, the recently proposed BMG method is proven to achieve an accelerating convergence rate. These results are important and significant, providing us with new deep understanding about gradient-based meta-learning, and theoretical guidance to design fast meta-learning algorithms with update rules. Last, the paper is well organized and clearly written.\n\nWEAKNESSES:\nThis is a theory paper, it could be more impactful if the authors may add more experiment results for large scale problems either in convex optimization or deep learning settings besides ImageNet.\n\nQUESTIONS:\nCurious to know what is the computational cost of Optimistic meta-learning compared to SGD in the ImageNet experiment.\n\nLIMITATIONS:\n1. There is no code provided to reproduce the results.\n2. The authors do not describe limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-01"}

comments: SUMMARY:
This work discovers the connection between gradient-based meta-learning and convex optimization of the meta parameters. From there, the authors observe that common gradient descent and its variants with momentum are special cases. To match the conventional $O(1 / T^2)$ convergence rate, the authors propose the optimistic version.

STRENGTHS:
1. The observation that common accelerated methods can be viewed as meta-learning is interesting. The online convex optimization view presents a new perspective for showing the convergence of the accelerated methods, which could be itself interesting.
2. The proposed optimistic gradient-based meta-learning method is simple, but shows performance gain over vanilla baseline methods.

WEAKNESSES:
The major weakness of the analysis comes from the restrictive form of the meta-learner. This makes the analysis not applicable to all neural network based learn-to-optimize methods, while the meta-learning methods applied on learning rate or pre-conditioning matrix are well studied already in the literature.

QUESTIONS:
The figure 1 shows the result of top-1 accuracy of sgd and standard meta learning on resnet50 on imagenet. But to my knowledge SGD can achieve >75% top-1 accuracy on imagenet, can the author elaborate more on what codebase they use and the details of experimental setting for that experiment (e.g., how do you tune learning rate, what is the scheduling etc)?

LIMITATIONS:
The authors have adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

comments: SUMMARY:
The submission studies connections between recent advances in convex optimisation and heuristic meta-learning update rules. The provided framework contains standard methods such as heavy ball and Nesterov's momentum as special cases, while also containing rules that correspond to online meta-learning. Bootstrapped meta-gradients are also formalised within this framework. Some of the theoretical results are corroborated by empirical investigations.

STRENGTHS:
* The submission is well-written and interesting. I can see myself referring back to this paper in future when developing new meta-learning or bi-level optimisation methods.
* Framework provides a useful tool for conceptualising and analysing meta-learning algorithms, as well as comparing them with existing (or novel) convex optimisation approaches.
* The analysis shows how one can obtain optimal $O(1/T^2)$ rates of convergence by using optimism. E.g., by predicting that the current gradient will be similar to the previous gradient.
* Experiments are provided that empirically corroborate the predictions of the theoretical analysis. In particular, the meta-learning variants of momentum and adagrad converge substantially faster than the conventional approaches.

WEAKNESSES:
* The analysis is limited to the convex case, which is understandable. However, it would be interesting to at least empirically investigate the non-convex case as well.

QUESTIONS:
N/A

LIMITATIONS:
There is no substantial discussion of the limitations of the work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-10