SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Sebastian Flennerhag
flennerhag@google.com
Tom Zahavy
Brendan O'Donoghue
We study the connection between gradient-based meta-learning and convex optimisation. We observe that gradient descent with momentum is a special case of meta-gradients, and building on recent results in optimisation, we prove convergence rates for meta-learning in the single task setting. While a meta-learned update rule can yield faster convergence up to constant factor, it is not sufficient for acceleration. Instead, some form of optimism is required. We show that optimism in meta-learning can be captured through the recently proposed Bootstrapped Meta-Gradient [9] method, providing deeper insight into its underlying mechanics.
In meta-learning, a learner is using a parameterised algorithm to adapt to a given task. The parameters of the algorithm are then meta-learned by evaluating the learner's resulting performance [25, 10, 2]. As such, meta-learning features a complex interaction between the learner and the meta-learner. The learner's problem is to minimize the expected loss  $f$  of a stochastic objective by adapting its parameters  $x \in \mathbb{R}^n$ . The learner has an update rule  $\varphi$  at its disposal that generates new parameters  $x_{t} = x_{t - 1} + \varphi (x_{t - 1},w_{t})$ ; we suppress data dependence to simplify notation. A simple example is when  $\varphi$  represents gradient descent with  $w_{t} = \eta$  its step size, that is  $\varphi (x_{t - 1},\eta) = -\eta \nabla f(x_{t - 1})$  [17, 26]; several works have explored meta-learning other aspects of a gradient-based update rule [6, 21, 7, 30, 31, 9, 15, 22].  $\varphi$  need not be limited to a gradient-based update, it can represent some algorithm implemented within a Neural Network [25, 11, 1, 29].
The meta-learner's problem is to optimise the meta-parameters  $w_{t}$  to yield effective updates.
In a typical (gradient-based) meta-learning setting, it does so by treating  $x_{t}$  as a function of  $w$ . Let  $h_t$ , defined by  $h_t(w) = f(x_{t-1} + \varphi(x_{t-1}, w))$ , denote the learner's post-update performance as a function of  $w$ . The learner and the meta-learner co-evolve according to
$$
x _ {t} = x _ {t - 1} + \varphi \left(x _ {t - 1}, w _ {t}\right), \quad w _ {t + 1} = w _ {t} - \nabla h _ {t} \left(w _ {t}\right) = w _ {t} - D \varphi \left(x _ {t - 1}, w _ {t}\right) ^ {T} \nabla f \left(x _ {t}\right),
$$
Figure 1: ImageNet. We compare training a 50-layer ResNet using SGD against variants that tune an element-wise learning rate online using standard meta-learning or optimistic meta-learning. Shading depicts  $95\%$  confidence intervals over 3 seeds.
where  $D\varphi (x,w)$  denotes the Jacobian of  $\varphi$  with respect to  $w$ . The nested structure between these two updates makes it challenging to analyse meta-learning, in particular it depends heavily on the properties of the Jacobian. In practice,  $\varphi$  is highly complex and so  $D\varphi$  is almost always intractable. For this reason, the only theoretical results we are aware of specialise to the multi-task setting, where the learner must adapt to a new task  $f_{t}$ . Acceleration in this setup is driven by the tasks similarity. That is, if all tasks are sufficiently similar, a meta-learned update can accelerate convergence [14]. However, these results do not yield acceleration in the absence of a task distribution to the best of our knowledge.
This paper provides an alternative view. We study the classical convex optimisation setting of approximating the minimiser  $\min_x f(x)$ . We observe that setting the update rule equal to the gradient, i.e.  $\varphi : (x, w) \mapsto w \nabla f(x)$ , recovers gradient descent. Similarly, we show in Section 4 that  $\varphi$  can be chosen to recover gradient descent with momentum. This offers another view of meta-learning as a non-linear transformation of classical optimisation. An implication thereof is that task similarity is not necessary condition for improving the rate of convergence via meta-learning. While there is ample empirical evidence to that effect [30, 31, 9, 16], we are only aware of theoretical results in the special case of meta-learned step sizes [17, 26].
Given a function  $f$  that is convex with Lipschitz smooth gradients, meta-learning improves the rate of convergence by a multiplicative factor of  $\lambda$  to  $O(\lambda / T)$  via the smoothness of the update rule. To achieve accelerated convergence,  $O(1 / T^2)$ , some form of optimism is required, typically in the form of a prediction of the next gradient. We consider optimism with meta-learning in the convex setting and prove accelerated rates of convergence,  $O(\lambda / T^2)$ . Again, meta-learning affects these bounds by a multiplicative factor. Our main contributions are as follows:
1. We show that meta-gradients contain gradient descent with momentum (Heavy Ball [23]; Section 4) and Nesterov Acceleration [20] as special cases (Section 5).
2. We show that gradient-based meta-learning can be understood as a non-linear transformation of an underlying optimisation method (Section 4).
3. We establish rates of convergence for meta-learning in the convex setting (Section 4).
4. We show that optimism can be expressed through the recently proposed Bootstrapped MetaGradient method [BMG; 9]. Our analysis provides a first proof of convergence for BMG and highlights the underlying mechanics that enable faster learning with BMG (Section 6).
2 Meta-Learning as Convex Optimisation
Problem definition. This section defines the problem studied in this paper and introduces our notation (see Appendix A, Table 1). Let  $f: \mathcal{X} \to \mathbb{R}$  be a proper and convex function. The problem of interest is to approximate the global minimum  $\min_{x \in \mathcal{X}} f(x)$ . We assume a global minimiser exists and is unique, defined by
$$
x ^ {*} = \underset {x \in \mathcal {X}} {\arg \min } f (x). \tag {1}
$$
We assume that  $\mathcal{X} \subseteq \mathbb{R}^n$  is a closed, convex and non-empty set.  $f$  is differentiable and has Lipschitz smooth gradients with respect to a norm  $\| \cdot \|$ , meaning that there exists  $L \in (0,\infty)$  such that  $\| \nabla f(x) - \nabla f(y) \|_* \leq L \| x - y \|$  for all  $x,y \in \mathcal{X}$ , where  $\| \cdot \|_*$  is the dual norm of  $\| \cdot \|$ . We consider the noiseless setting for simplicity; our results carry over to the stochastic setting by replacing the key online-to-batch bound used in our analysis by its stochastic counterpart [13].
Algorithm. Let  $[T] = \{1,2,\dots,T\}$ . We are given weights  $\{\alpha_{t}\}_{t=1}^{T}$ , each  $\alpha_{t} > 0$ , and an initialisation  $(\bar{x}_0,w_1) \in \mathcal{X} \times \mathcal{W}$ . At each time  $t \in [T]$ , an update rule  $\varphi: \mathcal{X} \times \mathcal{W} \to \mathcal{X}$  generates the update  $x_{t} = \varphi(\bar{x}_{t-1},w_{t})$ , where  $\mathcal{W} \subseteq \mathbb{R}^{m}$  is closed, convex, and non-empty. We discuss  $\varphi$  momentarily. The algorithm maintains the online average
$$
\bar {x} _ {t} = \frac {x _ {1 : t}}{\alpha_ {1 : t}} = (1 - \rho_ {t}) \bar {x} _ {t - 1} + \rho_ {t} x _ {t}, \tag {2}
$$
where  $x_{1:t} = \sum_{s=1}^{t} \alpha_s x_s$ ,  $\alpha_{1:t} = \sum_{s=1}^{t} \alpha_s$ , and  $\rho_t = \alpha_t / \alpha_{1:t}$ . Our goal is to establish conditions under which  $\{\bar{x}_t\}_{t=1}^T$  converges to the minimiser  $x^*$ . While this moving average is not always used in practical applications, it is required for accelerated rates in online-to-batch conversion [27, 3, 13].
Algorithm 1: Meta-learning in practice.	Algorithm 2: Meta-learning in the convex setting.
input:Weights {βt}Tt=1	input:Weights {αt}Tt=1, {βt}Tt=1
input:Update rule φ	input:Update rule φ
input:Initialisation (x0, w1)	input:Initialisation (x̅0, w1)
for t = 1, 2, ..., T:	for t = 1, 2, ..., T:
xt = xt-1 + φ(xt-1, wt)	xt = φ(x̅t-1, wt)
ht(·) = f(xt-1 + ρtφ(xt-1, ·))	xt = (1 - αt/α1:t) x̅t-1 + (αt/α1:t) xt
wt+1 = wt - βt ∇ht(wt)	gt = Dφ(x̅t-1, wt)T∇f(x̅t)
return xt	wt+1 = arg minw∈W ∑s=1t αs<ngs, w> + 1/2βt ||w||2
return x̅T
Convergence depends on how meta-parameters  $w_{t}$  are chosen. The meta-learner faces a sequence of losses  $h_t: \mathcal{W} \to \mathbb{R}$  defined by the composition  $h_t(w) = f\big((1 - \rho_t)\bar{x}_{t-1} + \rho_t\varphi(\bar{x}_{t-1},w)\big)$ . As such, the meta-learner is facing an online optimization, which we model under Follow-The-Regularized-Leader (FTRL; reviewed in Section 3): given  $w_0$ , each  $w_{t}$  is chosen according to
$$
w _ {t + 1} = \underset {w \in \mathcal {W}} {\arg \min } \left(\sum_ {s = 1} ^ {t} \alpha_ {s} \left\langle \nabla h _ {s} \left(w _ {s}\right), w \right\rangle + \frac {1}{2 \beta} \| w \| ^ {2}\right). \tag {3}
$$
Note that Eq. 3 subsumes the standard meta-gradient; if  $\|\cdot\|$  is the Euclidean norm, an interior solution yields  $w_{t+1} = w_t - \alpha_t \beta \nabla h_t(w_t)$ . It is straightforward to extend Eq. 3 to account for meta-updates that use AdaGrad-like [5] acceleration by altering the norms (see [12]).
Update rule. It is not possible to prove convergence outside of the convex setting, since  $\varphi$  may reach a local minimum, where local changes to  $w$  do not yield better updates in  $x$ , yet the  $x$  sequence is not converging. Convexity means that each  $h_t$  must be convex, which requires that  $\varphi$  is affine in  $w$  (but may vary non-linearly in  $x$ ). We also assume that  $\varphi$  is smooth with respect to  $\| \cdot \|$ , in the sense that it has bounded norm; for all  $x \in \mathcal{X}$  and all  $w \in \mathcal{W}$  we assume that there exists  $\lambda \in (0, \infty)$  for which
$$
\left\| D \varphi (x, w) ^ {T} \nabla f (x) \right\| _ {*} ^ {2} \leq \lambda \left\| \nabla f (x) \right\| _ {*} ^ {2}.
$$
Limitations. Our analysis makes relatively strict assumptions. Most meta-learning systems are not affine in  $w$ . A notable case where our assumptions to hold is meta-learned step-sizes or preconditioning matrices. For other update rules, our analysis holds up to first-order Taylor approximation error. We carry out experiments in Section 7 to empirically verify our theoretical insights.
3 Preliminaries: Online Convex Optimisation
In this section, we present analytical tools from the optimisation literature that we build upon. In a standard optimisation setting, there is no update rule  $\varphi$ ; instead, the iterates  $x_{t}$  are generated by a gradient-based algorithm, akin to Eq. 3. Our problem setting reduces to standard optimisation if  $\varphi$  is defined by  $\varphi : (x, w) \mapsto w$ , in which case  $x_{t} = w_{t}$ . In this paper, we use batch-to-online conversion as our analytical tool. This strategy treats the iterates  $x_{1}, x_{2}, \ldots$  as generated by an online learning algorithm, for which we can obtain a regret bound. This regret bound can then be turned into a convergence rate, detailed momentarily.
Online Optimisation. In online convex optimisation [32], a learner is given a convex decision set  $\mathcal{U}$  and faces a sequence of convex loss functions  $\{\alpha_{t}f_{t}\}_{t = 1}^{T}$ . At each time  $t\in [T]$ , it must make a prediction  $u_{t}$  prior to observing  $\alpha_{t}f_{t}$ , after which it incurs a loss  $\alpha_{t}f_{t}(u_{t})$  and receives a signal—either  $\alpha_{t}f_{t}$  itself or a (sub-)gradient of  $\alpha_{t}f_{t}(u_{t})$ . The learner's goal is to minimise regret,  $R(T)\coloneqq \sum_{t = 1}^{T}\alpha_{t}(f_{t}(u_{t}) - f_{t}(u))$ , against a comparator  $u\in \mathcal{U}$ . An important property of a convex function  $f$  is  $f(u^{\prime}) - f(u)\leq \langle \nabla f(u^{\prime}),u^{\prime} - u\rangle$ . Hence, the regret is largest under linear losses:  $\sum_{t = 1}^{T}\alpha_{t}(f_{t}(u_{t}) - f_{t}(u))\leq \sum_{t = 1}^{T}\alpha_{t}\langle \nabla f_{t}(u_{t}),u_{t} - u\rangle$ . For this reason, it is sufficient to consider regret under linear loss functions. An algorithm has sublinear regret if  $\lim_{T\to \infty}R(T) / T = 0$ .
FTRL & AO-FTRL. The meta-update in Eq. 3 is an instance of Follow-The-Regularised-Leader (FTRL) under linear losses. In Section 6, we show that BMG is an instance of the Adaptive-Optimistic
FTRL (AO-FTRL) [24, 19, 13, 28]. In AO-FTRL, we have a strongly convex regulariser  $\| \cdot \| ^2$ . AO-FTRL sets the first prediction  $u_{1}$  to minimise  $\| \cdot \| ^2$ . Given linear losses  $\{g_s\}_{s = 1}^{t - 1}$  and learning rates  $\{\beta_t\}_{t = 1}^T$ , each  $\beta_{t} > 0$ , the algorithm proceeds according to
$$
u _ {t} = \underset {u \in \mathcal {U}} {\arg \min } \left(\alpha_ {t} \langle \tilde {g} _ {t}, u \rangle + \sum_ {s = 1} ^ {t - 1} \alpha_ {s} \langle g _ {s}, u \rangle + \frac {1}{2 \beta_ {t}} \| u \| ^ {2}\right), \tag {4}
$$
where each  $\tilde{g}_t$  is a "hint" that enables optimistic learning [24, 19]; setting  $\tilde{g}_t = 0$  recovers the original FTRL algorithm. The goal of a hint is to predict the next loss vector  $g_t$ ; if the predictions are accurate AO-FTRL can achieve lower regret than its non-optimistic counter-part. Since  $\| \cdot \| ^2$  is strongly convex, FTRL is well defined in the sense that the minimiser exists, is unique, and finite [18]. The regret of FTRL and AO-FTRL against any comparator  $u\in \mathcal{U}$  can be upper-bounded by
$$
R (T) = \sum_ {t = 1} ^ {T} \alpha_ {t} \left\langle g _ {t}, u _ {t} - u \right\rangle \leq \frac {\| u \| ^ {2}}{2 \beta_ {T}} + \frac {1}{2} \sum_ {t = 1} ^ {T} \alpha_ {t} ^ {2} \beta_ {t} \left\| g _ {t} - \tilde {g} _ {t} \right\| _ {*} ^ {2}. \tag {5}
$$
Hence, hints that predict  $g_{t}$  well can reduce the regret substantially. Without hints, FTRL can guarantee  $O(\sqrt{T})$  regret (for non-strongly convex loss functions). However, [4] show that under linear losses, if hints are weakly positively correlated—defined as  $\langle g_t,\tilde{g}_t\rangle \geq \epsilon \| g_t\| ^2$  for some  $\epsilon >0$  then the regret guarantee improves to  $O(\log T)$ , even for non-strongly-convex loss functions. We believe optimism provides an exciting opportunity for novel forms of meta-learning. Finally, we note that these regret bounds (and hence our analysis) can be extended to stochastic optimisation [19, 12].
Online-to-batch conversion. The main idea behind online to batch conversion is that, for  $f$  convex, Jensen's inequality gives  $f(\bar{x}_T) - f(x^*) \leq \sum_{t=1}^{T} \alpha_t \langle \nabla f(x_t), x_t - x^* \rangle / \alpha_{1:T}$ . That is, the suboptimality gap of the average iterate  $\bar{x}_T$  can be bounded by the regret in the sequence  $x_1, x_2, \ldots, x_T$ . Applying this bound naively yields  $O(1/T)$  rate of convergence. In recent work, [3] shows that one achieve tighter bounds by instead querying the gradient at the average iterate,  $f(\bar{x}_T) - f(x^*) \leq \sum_{t=1}^{T} \alpha_t \langle \nabla f(\bar{x}_t), x_t - x^* \rangle / \alpha_{1:T}$ . A tighter bound means a faster rate of convergence. Recently, [13] tightened the analysis further and proved that the sub-optimality gap can be bounded by
$$
\begin{array}{l} f (\bar {x} _ {T}) - f (x ^ {*}) \leq \\ \frac {1}{\alpha_ {1 : T}} \left(R ^ {x} (T) - \frac {\alpha_ {t}}{2 L} \| \nabla f (\bar {x} _ {t}) - \nabla f (x ^ {*}) \| _ {*} ^ {2} - \frac {\alpha_ {1 : t - 1}}{2 L} \| \nabla f (\bar {x} _ {t - 1}) - \nabla f (\bar {x} _ {t}) \| _ {*} ^ {2}\right), \tag {6} \\ \end{array}
$$
were we define  $R^x (T)\coloneqq \sum_{t = 1}^T\alpha_t\langle \nabla f(\bar{x}_t),x_t - x^*\rangle$  as the regret of the sequence  $\{x_{t}\}_{t = 1}^{T}$  against the comparator  $x^{*}$ . With this machinery in place, we now turn to deriving our main results.
4 Meta-Gradients without Optimism
The main difference between classical optimisation and meta-learning is the introduction of the update rule  $\varphi$ . To see how this acts on optimisation, consider two special cases. If the update rule just returns the gradient,  $\varphi = \nabla f$ , Eq. 3 reduces to gradient descent (with averaging). This inductive bias is fixed and does not change with experience, so acceleration is not possible: the rate of convergence is  $O(1 / \sqrt{T})$  [28]. The other extreme is an update rule that only depends on the meta-parameters,  $\varphi(x, w) = w$ . Here, the meta-learner has ultimate control and selects the next update without constraints. The only relevant inductive bias is contained in  $w$ . To see how this inductive bias is formed, suppose  $\| \cdot \| = \| \cdot \|_2$  so that Eq. 3 yields  $w_{t+1} = w_t - \alpha_t \rho_t \beta \nabla f(\bar{x}_t)$  (assuming an interior solution). Combining this with the moving average in Eq. 2, we may write the learner's iterates as
$$
\bar {x} _ {t} = \bar {x} _ {t - 1} + \tilde {\rho} _ {t} (\bar {x} _ {t - 1} - \bar {x} _ {t - 2}) - \tilde {\beta} _ {t} \nabla f (\bar {x} _ {t - 1}),
$$
where each  $\tilde{\rho}_{t} = \rho_{t}\frac{1 - \rho_{t - 1}}{\rho_{t - 1}}$  and  $\tilde{\beta}_t = \alpha_t\rho_t\beta$ . Setting  $\beta = 1 / (2L)$  and each  $\alpha_{t} = t$  yields  $\tilde{\rho}_{t} = \frac{t - 2}{t + 1}$  and  $\tilde{\beta}_t = t / (4(t + 1)L)$ , which recovers Polyak's canonical Heavy-Ball method [23]. Hence, gradient descent with momentum is a special case of meta-learning under the update rule  $\varphi :(x,w)\mapsto w$ . Because Heavy Ball carries momentum from past updates, it can encode a model of the learning dynamics that leads to faster convergence, on the order  $O(1 / T)$ . The implication of this is that the dynamics of meta-learning are fundamentally momentum-based and thus learns an inductive bias in the same cumulative manner. This similarity carries in our theoretical analysis, which we turn to next.
The central challenge in applying the bound in Eq. 6 to Algorithm 2 is that the iterates  $x_{t}$  are generated under the update rule  $\varphi$ . Hence, we cannot apply standard regret bounds directly. Instead, observe that
$$
\begin{array}{l} R ^ {x} (T) = \sum_ {t = 1} ^ {T} \alpha_ {t} \langle \nabla f (\bar {x} _ {t}), x _ {t} - x ^ {*} \rangle = \sum_ {t = 1} ^ {T} \alpha_ {t} \langle \nabla f (\bar {x} _ {t}), \varphi (\bar {x} _ {t - 1}, w _ {t}) - x ^ {*} \rangle \\ = \underbrace {\sum_ {t = 1} ^ {T} \alpha_ {t} \langle \nabla f (\bar {x} _ {t}) , \varphi (\bar {x} _ {t - 1} , w _ {t}) - \varphi (\bar {x} _ {t - 1} , w ^ {*}) \rangle} _ {\text {r e g r e t u n d e r l o s s e s} \ell_ {t} (\cdot) = \alpha_ {t} \langle \nabla f (\bar {x} _ {t}) , \varphi (\bar {x} _ {t - 1} , \cdot) \rangle} + \underbrace {\sum_ {t = 1} ^ {T} \alpha_ {t} \langle \nabla f (\bar {x} _ {t}) , \varphi (\bar {x} _ {t - 1} , w ^ {*}) - x ^ {*} \rangle} _ {\text {d i f f e r e n c e i n c o m p a r a t o r c a p a c i t y}}. \\ \end{array}
$$
The first term in the final inequality can be understood as the regret under convex losses  $\ell_t(\cdot) = \alpha_t\langle \nabla f(\bar{x}_t),\varphi (\bar{x}_{t - 1},\cdot)\rangle$ . Since  $\varphi$  is affine,  $\ell_t$  is convex and thus this regret can be upper-bounded by linearising the losses. The linearisation reads  $\langle D\varphi (\bar{x}_{t - 1},w_t)^T\nabla f(\bar{x}_t),\cdot \rangle$ , which is identical to the linear losses  $\langle \nabla h_t(w_t),\cdot \rangle$  faced by the meta-learner in Eq. 3. In other words, the regret component can be upper-bounded by the of the meta-learner,
$$
R ^ {w} (T) := \sum_ {t = 1} ^ {T} \alpha_ {t} \langle \nabla h _ {t} (w _ {t}), w _ {t} - w ^ {*} \rangle \geq \sum_ {t = 1} ^ {T} \alpha_ {t} \langle \nabla f (\bar {x} _ {t}), \varphi (\bar {x} _ {t - 1}, w _ {t}) - \varphi (\bar {x} _ {t - 1}, w ^ {*}) \rangle .
$$
The regret of the learner is therefore upper bounded by
$$
R ^ {x} (T) \leq R ^ {w} (T) + \sum_ {t = 1} ^ {T} \alpha_ {t} \left\langle \nabla f \left(\bar {x} _ {t}\right), \varphi \left(\bar {x} _ {t - 1}, w ^ {*}\right) - x ^ {*} \right\rangle . \tag {7}
$$
The last term captures the difference in comparator capacity and more specifically the amount of regret they can inflict. If the comparator  $w^{*}$  has more power than that of  $x^{*}$ , it can accumulate a lower total loss, in which case this term will be negative, allowing us to discard it. Intuitively, the comparator  $x^{*}$  is non-adaptive. It must make one choice  $x^{*}$  and suffer the average loss. In contrast, the comparator  $w^{*}$  becomes adaptive under the update rule; it can only choose one  $w^{*}$ , but on each round it plays  $\varphi(\bar{x}_{t-1}, w^{*})$ . If  $\varphi$  is sufficiently flexible, this gives the comparator  $w^{*}$  more power than  $x^{*}$ , and hence it can force the meta-learner to suffer greater regret. When this is the case, we say that regret is retained when moving from  $x^{*}$  to  $w^{*}$ . As long as  $\varphi$  is not degenerate, this is typically easy to satisfy by making  $\mathcal{W}$  sufficiently large.
Definition 1. Given  $f, \{\alpha_t\}_{t=1}^T$ , and  $\{x_t\}_{t=1}^T$ , an update rule  $\varphi: \mathcal{X} \times \mathcal{W} \to \mathcal{X}$  preserves regret if there exists a comparator  $w \in \mathcal{W}$  that satisfies
$$
\sum_ {t = 1} ^ {T} \alpha_ {t} \langle \varphi (\bar {x} _ {t - 1}, w), \nabla f (\bar {x} _ {t}) \rangle \leq \sum_ {t = 1} ^ {T} \alpha_ {t} \langle x ^ {*}, \nabla f (\bar {x} _ {t}) \rangle . \tag {8}
$$
If such  $w$  exists, let  $w^{*}$  denote the comparator with smallest norm  $\| w\|$ .
Lemma 1. Given  $f$ ,  $\{\alpha_t\}_{t=1}^T$ , and  $\{x_t\}_{t=1}^T$ , if  $\varphi$  preserves regret, then
$$
R ^ {x} (T) = \sum_ {t = 1} ^ {T} \alpha_ {t} \langle \nabla f (\bar {x} _ {t}), x _ {t} - x ^ {*} \rangle \leq \sum_ {t = 1} ^ {T} \alpha_ {t} \langle \nabla f (\bar {x} _ {t}), \varphi (\bar {x} _ {t - 1}, w _ {t}) - \varphi (\bar {x} _ {t - 1}, w ^ {*}) \rangle = R ^ {w} (T).
$$
Proof: Appendix D. From Eq. 8, it is clear that for  $\varphi$  to retain regret, it must admit a parameterisation that correlates negatively with the gradient. In other words,  $\varphi$  must be able to behave as a gradient descent algorithm. However, this must not hold on every step, only sufficiently often. For instance,  $\varphi(x, \cdot)$  affine can be made to satisfy this condition if  $\mathcal{X}$  and  $\mathcal{W}$  are chosen appropriately.
Theorem 1. Let  $\varphi$  preserve regret and satisfy the assumptions in Section 2. Then
$$
\begin{array}{l} f \left(\bar {x} _ {T}\right) - f \left(x ^ {*}\right) \leq \frac {1}{\alpha_ {1 : T}} \left(\frac {\| w ^ {*} \| ^ {2}}{\beta} + \sum_ {t = 1} ^ {T} \frac {\lambda \beta \alpha_ {t} ^ {2}}{2} \| \nabla f \left(\bar {x} _ {t}\right) \| _ {*} ^ {2} \right. \\ \left. - \frac {\alpha_ {t}}{2 L} \| \nabla f (\bar {x} _ {t}) - \nabla f (x ^ {*}) \| _ {*} ^ {2} - \frac {\alpha_ {1 : t - 1}}{2 L} \| \nabla f (\bar {x} _ {t - 1}) - \nabla f (\bar {x} _ {t}) \| _ {*} ^ {2}\right). \\ \end{array}
$$
If  $x^{*}$  is a global minimiser of  $f$ , setting  $\alpha_{t} = 1$  and  $\beta = \frac{1}{\lambda L}$  yields  $f(\bar{x}_T) - f(x^*) \leq \frac{\lambda L \, \mathrm{diam}(\mathcal{W})}{T}$ .
Algorithm 3: BMG in practice.
input: Weights  $\{\beta_t\}_{t=1}^T$
input: Update rule  $\varphi$ , divergence  $B^{\mu}$
input:Targetoracle
input:Initialisation  $(x_0,w_1)$
for  $t = 1,2,\dots ,T$
$x_{t} = x_{t - 1} + \varphi (x_{t - 1},w_{t})$
Query  $z_{t}$  from target oracle
$d_{t}(\cdot) = B_{z_{t}}^{\mu}(x_{t - 1} + \varphi (x_{t - 1},\cdot))$
$w_{t + 1} = w_t - \beta_t\nabla d_t(w_t)$
return  $x_{T}$
Algorithm 4: Convex optimistic meta-learning.
input: Weights  $\{\alpha_{t}\}_{t = 1}^{T},\{\beta_{t}\}_{t = 1}^{T}$
input: Update rule  $\varphi$
input:Hints  $\{\tilde{g}_t\}_{t = 1}^T$
input:Initialisation  $(\bar{x}_0,w_1)$
for  $t = 1,2,\ldots ,T$
$x_{t} = \varphi (\bar{x}_{t - 1},w_{t})$
$\bar{x}_t = (1 - \alpha_t / \alpha_{1:t})\bar{x}_{t-1} + (\alpha_t / \alpha_{1:t})x_t$
$g_{t} = D\varphi (\bar{x}_{t - 1},w_{t})^{T}\nabla f(\bar{x}_{t})$
$v_{t} = \alpha_{t + 1}\tilde{g}_{t + 1} + \sum_{s = 1}^{t}\alpha_{s}g_{s}$
$w_{t + 1} = \arg \min_{w\in \mathcal{W}}\langle v_t,w\rangle +\frac{1}{2\beta_t}\| w\|^2$
return  $\bar{x}_T$