SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof: Appendix D. Compared to Heavy Ball, meta-learning introduces a constant  $\lambda$  that captures the smoothness of the update rule. Hence, while meta-learning does not achieve better scaling in  $T$  through  $\varphi$ , it can improve upon classical optimisation by a constant factor if  $\lambda < 1$ .
That meta-learning can improve upon momentum is borne out experimentally. We consider the problem of minimizing an ill-conditioned convex quadratic and compare standard momentum to a version with meta-learned step-size, i.e.  $\varphi : (x, w) \mapsto w \odot \nabla f(x)$ , where  $\odot$  is the Hadamard product. We find that introducing a non-linearity  $\varphi$  leads to a sizeable improvement in the rate of convergence. See Section 7.1 for further details.
5 Meta-Gradients with Optimism
It is well known that minimizing a smooth convex function admits convergence rates of  $O(1 / T^2)$ . Our analysis of meta-learning does not achieve these rates. Previous work indicates that we should not expect it to either; to achieve the theoretical lower-limit of  $O(1 / T^2)$ , some form of optimism (c.f. Section 3) is required. A typical form of optimism is to predict the next gradient. This is how Nesterov Acceleration operates [20], and is the reason for its  $O(1 / T^2)$  convergence guarantee.
From our perspective, meta-learning is a non-linear transformation of the iterate  $x$ . Hence, we should expect optimism to play a similarly crucial role. Formally, optimism comes in the form of hint functions  $\{\tilde{g}_t\}_{t=1}^T$ , each  $\tilde{g}_t \in \mathbb{R}^m$ , that are revealed to the meta-learner prior to selecting  $w_{t+1}$ . These hints give rise to Optimistic Meta-Learning (OML) via meta-updates
$$
w _ {t + 1} = \underset {w \in \mathcal {W}} {\arg \min } \left(\alpha_ {t + 1} \tilde {g} _ {t + 1} + \sum_ {s = 1} ^ {t} \alpha_ {s} \langle \nabla h _ {s} (w _ {s}), w \rangle + \frac {1}{2 \beta_ {t}} \| w \| ^ {2}\right). \tag {9}
$$
If the hints are accurate, meta-learning with optimism can achieve an accelerated rate of  $O(\tilde{\lambda} / T^2)$ , where  $\tilde{\lambda}$  is a constant that characterises the smoothness of  $\varphi$ , akin to  $\lambda$ . Again, we find that meta-learning behaves as a non-linear transformation of classical optimism and its rate of convergence is governed by the geometry it induces.
For a complete description, see Algorithm 4. These updates do not correspond to the typical meta-update in Algorithm 1; however, we show momentarily that they can be interpreted as the targets in the BMG method, summarised in Algorithm 3. Before turning to BMG, we establish that optimistic meta-learning in the convex setting does indeed yield acceleration.
Theorem 2. Let  $\varphi$  preserve regret and assume Algorithm 4 satisfy the assumptions in Section 2. Then
$$
\begin{array}{l} f (\bar {x} _ {T}) - f (x ^ {*}) \leq \frac {1}{\alpha_ {1 : T}} \left(\frac {\| w ^ {*} \| ^ {2}}{\beta_ {T}} + \sum_ {t = 1} ^ {T} \frac {\alpha_ {t} ^ {2} \beta_ {t}}{2} \| D \varphi (\bar {x} _ {t - 1}, w _ {t}) ^ {T} \nabla f (\bar {x} _ {t}) - \tilde {g} _ {t} \| _ {*} ^ {2} \right. \\ \left. - \frac {\alpha_ {t}}{2 L} \| \nabla f (\bar {x} _ {t}) - \nabla f (x ^ {*}) \| _ {*} ^ {2} - \frac {\alpha_ {1 : t - 1}}{2 L} \| \nabla f (\bar {x} _ {t - 1}) - \nabla f (\bar {x} _ {t}) \| _ {*} ^ {2}\right). \\ \end{array}
$$
Proof; Appendix D. From Theorem 2, it is clear that if  $\tilde{g}_t$  is a good predictor of  $D\varphi (\bar{x}_{t - 1},w_t)^T\nabla f(\bar{x}_t)$ , then the positive term in the summation can be cancelled by the negative term. In a classical optimisation setting,  $D\varphi = I_{n}$ , and hence it is easy to see that simply
choosing  $\tilde{g}_t$  to be the previous gradient is sufficient to achieve the cancellation [13]. Indeed, this choice gives us Nesterov's Accelerated rate [28]. The upshot of this is that we can specialise Algorithm 4 to capture Nesterov's Accelerated method by choosing  $\varphi : (x, w) \mapsto w$  as in the reduction to Heavy Ball—and setting the hints to  $\tilde{g}_t = \nabla f(\bar{x}_{t-1})$ . Hence, while the standard meta-update without optimism contains Heavy Ball as a special case, the optimistic meta-update contains Nesterov Acceleration as a special case.
In the meta-learning setting,  $D\varphi$  is not an identity matrix, and hence the best targets for meta-learning are different. Naively choosing  $\tilde{g}_t = D\varphi (\bar{x}_{t - 1},w_t)^T\nabla f(\bar{x}_{t - 1})$  would lead to a similar cancellation, but this is not allowed. At iteration  $t$ , we have not computed  $w_{t}$  when  $\tilde{g}_t$  is chosen, and hence  $D\varphi (\bar{x}_{t - 1},w_t)$  is not available. The nearest term that is accessible is  $D\varphi (\bar{x}_{t - 2},w_{t - 1})$ .
Corollary 1. Let each  $\tilde{g}_{t + 1} = D\varphi (\bar{x}_{t - 1},w_t)^T\nabla f(\bar{x}_t)$ . Assume that  $\varphi$  satisfies
$$
\left\| D \varphi (x ^ {\prime}, w) ^ {T} \nabla f (x) - D \varphi (x ^ {\prime \prime}, w ^ {\prime}) ^ {T} \nabla f (x ^ {\prime}) \right\| _ {*} ^ {2} \leq \tilde {\lambda} \| \nabla f (x ^ {\prime}) - \nabla f (x) \| _ {*} ^ {2}
$$
for all  $x^{\prime \prime},x^{\prime},x\in \mathcal{X}$  and  $w,w^{\prime}\in \mathcal{W}$ , for some  $\tilde{\lambda} >0$ . If each  $\alpha_{t} = t$  and  $\beta_{t} = \frac{t - 1}{2t\tilde{\lambda}L}$ , then
$$
f (\bar {x} _ {T}) - f (x ^ {*}) \leq \frac {4 \tilde {\lambda} L \operatorname {d i a m} (\mathcal {W})}{T ^ {2} - 1}.
$$
Proof: Appendix D.
These predictions hold empirically in a non-convex setting. We train a 50-layer ResNet using either SGD with a fixed learning rate, or an update rule that adapts a per-parameter learning rate online,  $\varphi : (x, w) \mapsto w \odot \nabla f(x)$ . We compare the standard meta-learning approach without optimism to optimistic meta-learning. Figure 1 shows that optimism is critical for meta-learning to achieve acceleration, as predicted by theory (experiment details in Appendix C).
6 Bootstrapped Meta-Gradients as a form of Optimism
Given Theorem 2, it is of interest to study practical ways of implementing optimism in meta-learning. We study a recently proposed variant of meta-gradients, Bootstrapped Meta-Gradients (BMG) [8]. Informally, instead of directly minimising the loss  $f$ , the meta-objective in BMG is the distance between the meta-learner's output  $x_{t}$  and a desired target  $z_{t}$ . The target is computed by unrolling the meta-learner for a further number of steps, thus implicitly embodying a form of optimism, before a gradient step is taken:  $z_{t} = x_{t} + \varphi(x_{t}, w_{t}) - \nabla f(x_{t} + \varphi(x_{t}, w_{t}))$ . This encodes optimism via  $\varphi$  because it encourages the meta-learner to build up momentum (i.e. to accumulate past updates). To see this formally, we turn to AO-FTRL. First, we provide a more general definition of BMG. Let  $\mu : \mathcal{X} \to \mathbb{R}$  be a convex distance generating function and define the Bregman Divergence  $B^{\mu} : \mathbb{R}^{n} \times \mathbb{R}^{n} \to \mathbb{R}$  by
$$
B _ {z} ^ {\mu} (x) = \mu (x) - \mu (z) - \langle \nabla \mu (z), x - z \rangle .
$$
Given initial condition  $(x_0, w_1)$ , the BMG updates proceed according to
$$
x _ {t} = x _ {t - 1} + \varphi \left(x _ {t - 1}, w _ {t}\right)
$$
$$
\begin{array}{l} w _ {t + 1} = w _ {t} - \beta_ {t} \nabla d _ {t} (w _ {t}) \\ = w _ {t} - \beta_ {t} D \varphi \left(x _ {t - 1}, w _ {t}\right) ^ {T} (\nabla \mu \left(x _ {t}\right) - \nabla \mu \left(z _ {t}\right)), \tag {10} \\ \end{array}
$$
where  $d_t: \mathbb{R}^n \to \mathbb{R}$  is defined by  $d_t(w) = B_{z_t}^\mu(x_{t-1} + \varphi(x_{t-1}, w_t))$ ; each  $z_t \in \mathbb{R}^n$  is referred to as a target. See Algorithm 3 for an algorithmic summary. To show how this relates to AO-FTRL, let  $\mu = f$ . In this case, the BMG update reads  $w_{t+1} = w_t - \beta_t D\varphi(x_{t-1}, w_t)^T (\nabla f(z_t) - \nabla f(x_t))$ . We can obtain these updates via our convex framework (i.e. Algorithm 4) by setting  $\tilde{g}_{t+1} = \nabla f(z_t)$ . In this case, we have that (Corollary 3, Appendix E) AO-FTRL reduces to
$$
w _ {t + 1} = \frac {\beta_ {t + 1}}{\beta_ {t}} w _ {t} - \beta_ {t} D \varphi (\bar {x} _ {t - 1}, w _ {t}) ^ {T} (\alpha_ {t + 1} \nabla f (z _ {t}) - \alpha_ {t} \nabla f (\bar {x} _ {t})) + \xi_ {t},
$$
where  $\xi_{t} = \beta_{t}\alpha_{t}D\varphi (\bar{x}_{t - 2},w_{t - 1})^{T}\nabla f(z_{t - 1})$  denotes an error correction term that removes the previous target. This error correction term is theoretically important for stability [13], as the accumulation of hints can otherwise dominate the true signal. That the original BMG does not feature this error term may explain the instabilities the authors observed when setting too aggressive targets [8]. Since Algorithm 3 does not average its iterates—while Algorithm 4 does—we see that these updates (ignoring  $\xi_{t}$ ) are identical up to scalar coefficients (that can be controlled for by scaling each  $\beta_{t}$  and each  $\tilde{g}_{t + 1}$  accordingly). Our next results present formulas for constructing targets in BMG or hints in AO-FTRL so that the two commute.
Theorem 3. Targets in Algorithm 3 and hints in algorithm 4 commute in the following sense.  $\mathbf{BMG} \rightarrow \mathbf{AO}\text{-FTRL}$ . Let BMG targets  $\{z_t\}_{t=1}^T$  by given. A sequence of hints  $\{\tilde{g}\}_{t=1}^T$  can be constructed recursively by
$$
\alpha_ {t + 1} \tilde {g} _ {t + 1} = D \varphi (\bar {x} _ {t - 1}, w _ {t}) ^ {T} (\nabla \mu (\bar {x} _ {t}) - \nabla \mu (z _ {t}) - \alpha_ {t} \nabla f (\bar {x} _ {t})) + \alpha_ {t} \tilde {g} _ {t}, \quad t \in [ T ], \tag {11}
$$
so that interior updates for Algorithm 4 are given by
$$
w _ {t + 1} = \frac {\beta_ {t}}{\beta_ {t - 1}} w _ {t} - \beta_ {t} (\nabla \mu (z _ {t}) - \nabla \mu (\bar {x} _ {t})).
$$
AO-FTRL  $\rightarrow$  BMG. Conversely, assume a sequence  $\{\tilde{y}_t\}_{t=1}^T$  are given, each  $\tilde{y}_t \in \mathbb{R}^n$ . If  $\mu$  strictly convex, a sequence of BMG targets  $\{z_t\}_{t=1}^T$  can be constructed recursively by
$$
z _ {t} = \nabla \mu^ {- 1} (\nabla \mu (x _ {t}) - (\alpha_ {t + 1} \tilde {y} _ {t + 1} + \alpha_ {t} \nabla f (x _ {t}))) \qquad t \in [ T ],
$$
so that BMG updates in Eq. 10 are given by
$$
w _ {t + 1} = w _ {t} - \beta_ {t} \left(\alpha_ {t + 1} \tilde {g} _ {t + 1} + \alpha_ {t} \left(D \varphi (\bar {x} _ {t - 1}, w _ {t}) ^ {T} \nabla f (\bar {x} _ {t}) - \tilde {g} _ {t}\right)\right),
$$
where each  $\tilde{g}_{t + 1}$  is given by
$$
\alpha_ {t + 1} \tilde {y} _ {t + 1} = \alpha_ {t + 1} D \varphi \left(x _ {t - 1}, w _ {t}\right) ^ {T} \tilde {y} _ {t + 1} + \alpha_ {t} \tilde {y} _ {t}.
$$
Proof; see Appendix E. As an immediate consequence of this, we can apply our optimistic metagradient analysis (Theorem 2) to BMG to obtain a rate of convergence. This is captured in the following corollary.
Corollary 2. Let each  $\tilde{g}_{t + 1} = D\varphi (\bar{x}_{t - 1},w_t)^T\tilde{y}_{t + 1}$ , for some  $\tilde{y}_{t + 1}\in \mathbb{R}^n$ . If each  $\tilde{y}_{t + 1}$  is a better predictor of the next gradient than  $\nabla f(\bar{x}_{t - 1})$ , in the sense that
$$
\| D \varphi (\bar {x} _ {t - 2}, w _ {t - 1}) ^ {T} \tilde {y} _ {t} - D \varphi (\bar {x} _ {t - 1}, w _ {t}) ^ {T} \nabla f (\bar {x} _ {t}) \| _ {*} \leq \tilde {\lambda} \| \nabla f (\bar {x} _ {t}) - \nabla f (\bar {x} _ {t - 1}) \| _ {*},
$$
then Algorithm 4 guarantees convergence at a rate  $O(\tilde{\lambda} / T^2)$ .
In other words, for certain choices of targets, BMG yields accelerated rates of convergence.
In this section, we detail experiments borne out to test our theoretical predictions. Section 7.1 tests predictions about meta-gradients without optimism, by comparing meta-learned variants of standard optimisers to their non-meta-learning counterparts. Section 7.2 test predictions about optimistic meta-gradients, by comparing an optimistic meta-gradient approach to gradient descent.
7.1 Convex Quadratic Experiments
Loss function. We consider the problem of minimising a convex quadratic loss function  $f: \mathbb{R}^2 \to \mathbb{R}$  of the form  $f(x) = x^{T}Qx$  for some  $Q$  that is sampled such that it is ill-conditioned (see Appendix B for details).
Protocol. Given that the solution is always  $(0,0)$ , this experiment revolves around understanding how different algorithms deal with curvature. Given symmetry in the solution and ill-conditioning, we fix the initialisation to  $x_0 = (4,4)$  and run each algorithm for 100 iterations. For each  $Q$  and each algorithm, we sweep over the learning rate, decay rate, and the initialization of  $w$  (see Table 2 for values) and report results for the best performing hyper parameters.
Results. We report the learning curves for the best hyper-parameter choice for 5 randomly sampled problems in the top row of Figure 2 (columns correspond to different Q). We also study the sensitivity of each algorithm to the learning rate in the bottom row Figure 2. For each learning rate, we report the cumulative loss during training. While baselines are relatively insensitive to hyper-parameter choice, meta-learned improve for certain choices, but are never worse than baselines.
7.2 Imagenet Experiments
Protocol. We train a 50-layer ResNet following a standard protocol (Appendix C) with SGD as the baseline optimiser. We compare SGD to a variant that meta-learns an element-wise learning rate online, i.e.  $(x,w)\mapsto w\odot \nabla f(x)$ . We sweep over the learning rate (for SGD) or meta-learning rate and report results for the best hyper-parameter over three independent runs.
Figure 2: Convex Quadratic. We generate convex quadratic loss functions with ill-conditioning and compare gradient descent with momentum and AdaGrad to meta-learning variants. Meta-Momentum uses  $\varphi : (x, w) \mapsto w \odot \nabla f(x)$  while Meta-AdaGrad uses  $\varphi : (x, w) \mapsto \nabla f(x) / \sqrt{w}$ , where division is element-wise. Top: loss per iteration for randomly sampled loss functions. Bottom: cumulative loss (regret) at the end of learning as a function of learning rate; details in Appendix B.
Standard meta-learning. In the standard meta-learning setting, we apply the update rule once before differentiating w.r.t. the meta-parameters. That is, the meta-update takes the form  $w_{t + 1} = w_t - \beta \nabla h_t(w_t)$ , where  $h_t = f(x_t + w_t \odot \nabla f(x_t))$ . Because the update rule is linear in  $w$ , we can compute the meta-gradient analytically:  $\nabla h_t(w_t) = \nabla_w f(x + \varphi(x, w)) = D\varphi(x, w)^T \nabla f(x') = \nabla f(x) \odot \nabla f(x')$ , where  $x' = x + \varphi(x, w)$ . Hence, we can compute the meta-updates in Algorithm 1 manually as  $w_{t + 1} = \max \{w_t - \beta \nabla f(x_t) \odot \nabla f(x_{t + 1}), 0\}$ , where we introduce the max operator on an element-wise basis to avoid negative learning rates. Empirically, this was important to stabilize training.
Optimistic meta-learning. For optimistic meta-learning, we proceed much in the same way, but include a gradient prediction  $\tilde{g}_{t + 1}$ . For our prediction, we use the previous gradient,  $\nabla f(x_{t + 1})$ , as our prediction. This yields meta-updates of the form  $w_{t + 1} = \max \left\{w_t - \beta \nabla f(x_{t + 1})\odot (\nabla f(x_{t + 1}) + \nabla f(x_t)) - \nabla f(x_t)\odot \nabla f(x_t),0.\right\}$
Results. We report Top-1 accuracy on the held-out test set as a function of training steps in Figure 1. Tuning the learning rate does not yield any statistically significant improvements under standard meta-learning. However, with optimistic meta-learning, we obtain a significant acceleration as well as improved final performance, increasing the mean final top-1 accuracy from  $72\%$  to  $75\%$ .
This paper explores a connection between convex optimisation and meta-learning. We construct an algorithm for convex optimisation that aligns as closely as possible with how meta-learning is done in practice. Meta-learning introduces a transformation and we study the effect this transformation has on the rate of convergence. We find that, while a meta-learned update rule cannot generate a better dependence on the horizon  $T$ , it can improve upon classical optimisation up to a constant factor.
An implication of our analysis is that for meta-learning to achieve acceleration, it is important to introduce some form of optimism. From a classical optimisation point of view, such optimism arises naturally by providing the meta-learner with hints. If hints are predictive of the learning dynamics these can lead to significant acceleration. We show that the recently proposed BMG method provides a natural avenue to incorporate optimism in practical application of meta-learning. Because targets in BMG and hints in optimistic online learning commute, our results provide first rigorous proof of convergence for BMG, while providing a general condition under which optimism in BMG yields accelerated learning.
[21] A. Nichol, J. Achiam, and J. Schulman. On First-Order Meta-Learning Algorithms. arXiv preprint ArXiv:1803.02999, 2018.
[22] J. Oh, M. Hessel, W. M. Czarnecki, Z. Xu, H. P. van Hasselt, S. Singh, and D. Silver. Discovering Reinforcement Learning Algorithms. In Advances in Neural Information Processing Systems, volume 33, 2020.
[23] B. T. Polyak. Some Methods of Speeding up the Convergence of Iteration Methods. USSR Computational Mathematics and Mathematical Physics, 4(5):1-17, 1964.
[24] S. Rakhlin and K. Sridharan. Optimization, Learning, and Games with Predictable Sequences. In Advances in Neural Information Processing Systems, 2013.
[25] J. Schmidhuber. Evolutionary Principles in Self-Referential Learning. PhD thesis, Technische Universität München, 1987.
[26] T. van Erven and W. M. Koolen. MetaGrad: Multiple Learning Rates in Online Learning. In Advances in Neural Information Processing Systems, 2016.
[27] J.-K. Wang and J. Abernethy. Acceleration through Optimistic No-Regret Dynamics. arXiv preprint arXiv:1807.10455, 2018.
[28] J.-K. Wang, J. Abernethy, and K. Y. Levy. No-regret dynamics in the fenchel game: A unified framework for algorithmic convex optimization. arXiv preprint arXiv:2111.11309, 2021.
[29] J. X. Wang, Z. Kurth-Nelson, D. Tirumala, H. Soyer, J. Z. Leibo, R. Munos, C. Blundell, D. Kumaran, and M. Botvinick. Learning to Reinforcement Learn. In Annual Meeting of the Cognitive Science Society, 2016.
[30] Z. Xu, H. P. van Hasselt, and D. Silver. Meta-Gradient Reinforcement Learning. In Advances in Neural Information Processing Systems, 2018.
[31] T. Zahavy, Z. Xu, V. Veeriah, M. Hessel, J. Oh, H. P. van Hasselt, D. Silver, and S. Singh. A Self-Tuning Actor-Critic Algorithm. In Advances in Neural Information Processing Systems, volume 33, 2020.
[32] M. Zinkevich. Online Convex Programming and Generalized Infinitesimal Gradient Ascent. In International Conference on Machine Learning, 2003.