MetaMD: Principled Optimiser Meta-Learning for Deep Learning
Authors: Boyan Gao
Henry Gouk
Jan Stuehmer
Massimiliano Pontil
Timothy Hospedales
Venue: Submitted to ICLR 2023 (2023)
Keywords: Meta-learning
Optimiser Learning

Abstract: Optimiser design influences learning speed and generalisation in training machine learning models. Several studies have attempted to learn more effective gradient-descent optimisers via solving a bi-level optimisation problem where generalisation error is minimised with respect to optimiser parameters. However, most existing neural network oriented optimiser learning methods are intuitively motivated, without clear theoretical support, and focus on learning implicit biases that improve generalisation, rather than speed of convergence. We take a different perspective starting from mirror descent rather than gradient descent, and meta-learning the corresponding Bregman divergence. Within this paradigm, we formalise a novel meta-learning objective of optimising the rate of convergence. The resulting framework, termed Meta Mirror Descent (MetaMD), learns to accelerate optimisation speed. Unlike many meta-learned neural network optimisers, it also supports convergence guarantees and uniquely does so without requiring validation data. We empirically evaluate our framework on a variety of tasks and architectures in terms of convergence rate and generalisation error and demonstrate strong performance.

Under review as a conference paper at ICLR 2023
METAMD: PRINCIPLED OPTIMISER META- FOR DEEP LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Optimiser design influences learning speed and generalisation in training machine learning models. Several studies have attempted to learn more effective gradient- descent optimisers via solving bi-level optimisation problem where generalisation error is minimised with respect to optimiser parameters. However, most existing neural network oriented optimiser learning methods are intuitively motivated, with- out clear theoretical support, and focus on learning implicit biases that improve generalisation, rather than speed of convergence. We take different perspective starting from mirror descent rather than gradient descent, and meta-learning the corresponding Bregman divergence. Within this paradigm, we formalise a novel meta-learning objective of optimising the rate of convergence. The resulting frame- work, termed Meta Mirror Descent (MetaMD), Iearns to accelerate optimisation speed. Unlike many meta-learned neural network optimisers, it also supports con- vergence guarantees and uniquely does without requiring validation data. We empirically evaluate our framework on a variety of tasks and architectures in terms of convergence rate and generalisation error and demonstrate strong performance.
INTRODUCTION
Gradient-based optimisation algorithms, such as stochastic gradient descent (SGD), are fundamental building blocks of many machine learning gegorithms-notale those focused on training linear models and deep neural networks. These methods are typically developed to solve a broad class of problems, and therefore the method developers make as few assumptions about the target problem as possible. This leads to a variety of general purpose techniques for optimisation, but such generality often comes with slower convergence. By taking advantage of more information about the target problem, one is typically able to design more efficient- but less general- ootimllsstiiiiiion algorithms. Another challenge in a non-convex deep learning context, is that that many of the empirically fastest optimisers such as Adam (Kingma & Ba 2015) lack convergence guarantees.
While one line of research hand-designs optimisers to exploit known properties of particular problems, a complementary line of research focuses on situations where optimisation problems come in families. This allows the use of meta-learning techniques to fit optimiser to the given problem family with the goal of maximising convergence speed or generalisation performance. For example, in the many-shot regime, /Andrychowicz et al. (2016) and Wichrowska et al. (2017) learn black-box neural optimisers to accelerate training of neural networks, while Bello et al. (2017) learn symbolic gradient-based optimisers improve generalisation. MAML (Finn et al.) 2017) and Meta-SGD (Li et al. 2017) learned initialisation and learning rate for SGD training of neural networks with good generalisation performance the few-shot regime. Later generalisations focused learning problem family-specific curvature information (Park & Oliva 2019 Flennerhag et al., 2020). Nevertheless, most existing learned optimisers such as Andrychowicz et al. (2016); Wichrowska al. (2017):
Flennerhag et al. (2020): Bello et al. (2017) can not provide convergence guarantees.
In this work, we revisit the optimiser learning problem from the perspective of mirror descent. Mirror descent introduces a Bregman divergence that regularises the distance between current and next iterate, introducing strongly convex sub-problem that can be optimised exactly. In mirror descent, the choice Bregman divergence determines optimisation dynamics. In a meta-learning context, the Bregman divergence thus provides interesting representation of an optimisation strategy that can be fit to a given family of optimisation problems, leading to our learned optimiser termed Meta Minror Descent (MetaMD). Many existing learned optimisers do not have formal notion of convergence

---

Under review as a conference paper at ICLR 2023
rate, and in practice typically optimise a meta-objective reflecting training validation loss after fixed number of iterations. In contrast, MetaMD is directly trained to optimise the convergence rate bound for mirror descent. Importantly, this means we can adapt theoretical guarantees from mirror descent to provide convergence guarantees for MetaMD, an important property not provided by most learned optimsers, and many hand-designed optimisers widely used in deep learning.
An important issue in meta-learning mirror descent is specifying the family of Bregman divergences to learn. Meta-learning with general Bregman divergences leads to an intractable tri-level optimi- sation problem. Thus, we seek a family of divergences for which the innermost optimisation has a closed form solution. The chosen paramaterisation should be complex enough to exhibit interesting optimisation dynamics, simple enough to provide a closed form solution, and always provide a valid Bregman divergence. We provide a parameterisation that meets all these desideratum in the form of Kronecker factorised preconditioner (eeitiiiioneZZrrusirss been successfully exploited in meta- meta-learning methods such as Meta-Curvatuus Park & Oliva (2019), and WarpGrad Flennerhag et al. (2020), and are richer than the parameter-wise learning rates Li et al. ( Khodak et al. (2019), which is special case. Importantly, we also show how to train the divergence efficiently with implicit gradient computation Lorraine et al. (2020) Uniquely, this means that our framework can effectively train learning-rate (as a special case of preconditioning) with implicit gradient, an idea that was previously suggested to be impossible (Lorraine et al. (020).
(888); Martens & Grosse (2015). Preconditioners have
Empirically we demonstrate that we can train MetaMD given a model architecture and suite of train- ing tasks, We then deploy it to novel testing tasks and show that MetaMD provides fast convergence and good generalisation compared to several existing hand-designed and learned optimisers.
RELATED WORK
Meta-learning aims to extract some notion of 'how to learn' given task or distribution of tasks (Hospedales et al. 2020), such that new learning trials are better or faster. These two stages are often called meta-training, and meta-testing respectively. Key dichotomies include: meta-learning from single task vs a task distribution (as formalised, e.g., by Baxter (2000)); the type of meta-knowwwa knowledge to be extracted; and long- vs short-horizon meta-learning For few-shot problems with short optimisation horizons, the seminal model-agnostic meta-learning (MAML) Finn et al. (011)) learns an initial condition from which only few optimisation steps required solve new task. Meta-SGD Li et al. (017) and Meta-Curvature) Park & Oliva (2119) extend MAML by learning a parameter-wise learning rate, and preconditioning matrix, respectively. Another group of methods focus on larger scale problems in terms of dataset size and optimisation horizon. For example, neural architecture search (NAS) Real et al. (2019) Zoph & Le (2017) discovers effective neural architectures. MetaReg Balaji et al. 2018) meta-leama regularisation parameters to improve domain generalisation.
Several studies focus specifically on optimiser meta- meta-learning for many-shot batch learning problems, which we address here. In this case, the extracted wetta-nnowlddge spans learning rates for SGD (Micaelli & Storkey (02T), symbolic gradient- descent rules (Bello et al. (017) neural network gradient-descent rules (Andrychowicz et al. 2016 Li & Malik 2017), and gradient-free optimisers (Sandler et al. 2021 Chen et al. 2017). Differently to the gradient-descent based methods, we start from the perspective of mirror descent, where mirror descent's 's Bregman Divergence provides a target for meta-learning. This perspective has several benefits, notably the ability to derive learned optimiser with convergence and generalisation guarantees. Our suggested practical instantiation is meta-learn a Coonecker-factorise pre-conditioner, which is effective and efficient. However, the framework is general and can be used to learn particular special cases such as element-wise learning rates Li et al. (2017); Micaelli & Storkey (202T). Beyond supporting this richer representation. we provide convergence guarantees, do not rely on validation set, and demonstrate cross-dataset generalisation, enabling us to amortise meta-learning cost. contrast, the single task meta-learne of Micaelli & Storkey (2021) needs to repeat meta-learning on each specific dataset to optimise per-dataset validation performance.
In addition to the work on deep meta-learning of optimisers and optimiser hyperparameters, there is also a rich literature on principled development of online meta-learning algorithms that aim minimise various notions of meta meta-regret[ regget(Finn& et al. (2019); Denevi et al. 2011b a. 2018): Balcan et al. (2019): Khodak et al. (2019). Several of these build online mirror descent, but there is an important distinction between their work and ours: we focus on the batch setting, with a view to ensuring the resulting algorithm can be tractably applied to deep neural networks. ARUBA (Khodak al. 2019)
2

---

Under review as a conference paper at ICLR 2023
is perhaps closest to work. A closed-form update rule for the Bregman divergence is derived from a similar convergence bound. However, (i) it addresses online rather than the batch training regime relevant for neural network learning, and (ii) it is only instantiated with a Bregman divergence parameterised by a diagonal matrix.
3 STOCHASTIC MIRROR DESCENT
We formalise the problem of learning optimiser using the Stochastic Mirror Descent (SMD) framework, which can be thought of as a generalisation of stochastic subgradient descent. SMD optimisers produce a series of progressively better estimates for the optimal parameters of the objective function. This is accomplished by solving convex optimisation problem at each step, t,
())( D.+1 = arg (0 + 21
(1)
where (s((s)) provides an unbiased stochastic approximation of the loss based on the random variable and training set S. v is the step size. Bg denotes Bregman divergence, which can be thought of as way of measuring distance in parameter space, and each choice of Bregman divergence leads to a different optimisation algorithm. One can define Bregman divergences as
B (0 = x - (00)) - (Vo(0),
(2)
where à is a A-strongly convex function. There are several choices of ø that result in existing algorithms specialised for various types of optimisation problems in machine learning. For example, if one chooses e to be -11 112. then mirror descent becomes gradient descent, while choosing e to be the negative entropy results in the bullback-Leiblee divergence, leading to the exponentiated gradient algorithm (Kivinen& & Warmuth, 1997). A significant benefit of deriving new algorithms that fit into the mirror descent framework is that, in the convex setting, one can obtain bounds on the rate of convergence towards minima 0, for any valid choice of Φ @-including those that are learned. In this work we make use the following convergence analysis:
Theorem 3.1 (Corollary 8 from Orazio et al. (2021)). If (s(+g) convex and L-smooth to the Euclidean norm for all &, and is essentially smooth, then stochastic mirror descent with constant step size 1) S X/2L guarantees
T w.r.t.
Ee( - (0.;EE) < (B.10.100
+ 2202,
1/T
where 0 = - ET=1 8, and 0² = info [(: E) - (info Cs(
We note that one can obtain a trivial corollary where instead of 0 appearing the left-hand side, one can instead use the best iterate obtained after T steps, arg ming [E The above bound assumes convexity, but the convergence of stochastic mirror descent for neural networks is an open problem, with recent work focusing on narrower families of optimisers (e.g.. g.,Zhang et al. (2020)).
4
META-LEARNING A BREGMAN DIVERGENCE
4.1 OPTIMISER LEARNING FRAMEWORK
We propose meta-learning algorithm to learn mirror descent optimisers. We consider the multi-task meta-learning setting (Hospedales et al. 2020 Finn et al.) 2017), assuming that a task distribution p(T) is available from which learned optimizer by meta-testing on novel tasks from the same distribution. For gradient-basee meta-learning, the meta-training procedure is conventionally framed as bilevel optimization problem where the inner problem solves learning tasks given the optimiser, and the outer problem updates the optimiser (Hospedales al. 2020). We do this by using parameterised potential function, ΦMs to construct the Bregman divergence. The outer problem is to minimise some meta objective denoted E(M) with respect to parameters, M. Since our mirror descent optimizer is defined by a Bregman
draw tasks for meta-training, and that we will evaluate the
In D' O'Orazio al. 2021), function, f, is said be L-smooth if {+) < L. for all o..
3

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Meta Mirror Descent learning algorithm.
1: Input: Si,... S., M,, k
[Meta-train set, initial divergence parameters, minibatch size]
Output: .M
Leeared divergence parameters) )
3: while not converged or reached max steps do
4: H= O
(Set hypergradient accumulator)
5: for all St minibatch((11..... Sn, do
6:
Init 6:
(Sct random weights for base model)
7:
02 arg ming Beu (0, 0...0)s.. E arg ming (0()))
(Train the base model with mirror descent)
8:
H = H + hypergradientt E OM.
[Obtain in Eq 13 detailed in Appx A.12
9:
end for
10: M (., H)
(Update M using SGD (or similar) update rule)
end while
divergence B this leads to a tri-level optimisation problem with new layer corresponding to the problem given in Eq. required complete a single mirror descent step,
min (MM.(MM) M
(4)
s.t. 0, (M) = arg min LG( = (Tzi Thy o TTs))
(5)
1 2nj
s.t. π Tas (0,) arg min ( VoLs(0.). 8) + Bas (01(6.). (f1l0)).
(6)
In the outer problem (Eq. 4) the algorithm aims to learn divergence by optimising the meta- objective E, which evaluates the optimiser performance. To achieve this requires getting the best response from the mid-level problem (Eq.5) where the base model is trained from the initialisation Op to 0, by recursively applying the Text operation, which is defined in the innermost problem (Eq, [5]. The inner problem corresponds to performing stochastic mirror descent update using a Bregman divergence based on Φı. Compared with the standard bilevel problems in meta-learning. introducing this third layer adds significant cost to both meta-train and meta-test stages. However, with a suitable choice of divergence, we can obtain a closed-form solution for the inner problem, thus reducing the cost to be similar to that of standard bilevel optimisation problem. This section explains the main components of our method, which is summarised in Algorithm I
4.2 DIVERGENCE PARAMETERISATION
The paramaterisation of the divergence is important for practical instantiation of our framework. Ideally it should be expressive enough to represent interesting optimisation dynamics, while being simple enough to provide an efficient or closed form solution to the innermost convex mirror descent optimisation. We describe reasonable compromise as follows.
We define our Bregman divergences via the squared norm, Φx (0) = _ M0, where M is learnable symmetric positive definite matrix. In this case, the inner problem has a closed form solution,
0z+1 = 0, nM voCs()))
The derivation of this closed-form mapping is given in Appx. be interpreted as preconditioner, which has high representation potentia[Pat Park Olival (2019):
This parameterisation can
Flennerhag et al. (2020), but it can be inefficient to compute - when the dimensionality of # is high-as is typically the case for neural networks. To provide a better trade-off between capacity and efficiency, we reduce the capacity of dM by using block diagonal matrix M = diag(M, M2, .... Mz)
with each block operating on the gradient of one layer in a neural network. We additionally represent each Mj by Kronecker factorisation (Zehfuss
[858]
Hensel
1889 Martens & Grosse 2015).
M₁- A X Bç,
where Aç E R and Bg E R.XI are two symmetric positive definite matrices parameterised by the column vectors ag and bę,
B bebb + I., To avoid explicitly constructing M,, make use of the fact that
T In. A açug
M V.. Cs(()) (Aç @ Bc) vo. = (ee
4

---

Under review as a conference paper at ICLR 2023
where 0g denotes the weights in layer Ç. A and B can be easily computed by composing the eigenvector matrices and inverting the diagonal eigenvalue matrix generated by Singular Value Decomposition from A and Be respectively.
4.3 META-OBJECTIVE
The next step is define the meta-objective in terms of the Bregman divergence parameters, M. An advantage of the SMD framework is that we have a formal notion of convergence from an initialisation 0, solution 0. (from Theorem 3.1). By defining the meta objective {M, 0) bound on the convergence rate, meta-learning ΦM leads to a faster optimíser. In particular, we design meta-objective based on the quantities that appear in Theorem 3.1 Expanding the definition of the condition on the step size in the bound yields
2BoM (0-1101)L 20²
Be(Cs(0:E) (o.; S
XI
Minimising the right-hand side of this inequality with respect to M ensures that the learned optimiser finds solution closer to minima after I iterations of training. As our goal is to optimise the expected speed of convergence on future tasks, we design a meta-objective that considers the average convergence rate over different meta-train tasks. We also discard terms that do not depend on M, and separate into a regularisation term that allows us to control the importance optimising the strong convexity coefficient. Thus, we define the meta-objective as
k (0)) (10 +M().
E(M, 1 1
Σ 1=1
Bon
(8)
where 0 and 0f the initial and final weights for the task i, and leave k as a hyperparameter that can be tuned heuristically. We will shortly define regularisation function, g(-), which is used prevent meta-overfitting. The strong convexity parameter is given by X ming (0min(M)) where Vnii denotes the smallest eigenvalue and the details are given in Appx. A.5
We solve the outer loop optimisation problem (Eq. 4 by gradient descent using 0 This gradient computation relies on unrolling training trajectories the inner loop where it is expensive to compute the gradient with standard reverse-mode differentiation. In this work, we apply implicit differentiation (Lorraine et al. 2020) to solve this problem, which will be discussed in Section 4.4
Generalisation of Convergence Rate Selecting the Bregman divergence parameters using a set of training tasks will result in optimistic view of how well would expect the learned optimiser to converge on novel tasks. That is, when Eq. g is evaluated on the meta-train tasks, it will provide an overly optimistic estimate of the convergence rate simple family of Bregman divergences we employ, is possible to construct a high-confidence bound on how biased this estimate will be, and therefore provide convergence guarantees for novel tasks in terms of the meta-train objective:
expect future tasks. Given the relatively
Theorem 4.1. Under the same conditions as Theorem 3.1 for any M, O2 and 0,, such that || T|e < C and ||0 0.112 I r, the following holds with probability at least 1 -8,
2B au (0.0) 11(1))
2r
In(2/8)
Eg (CEeeCC((0E)) Cs(0.
+ 20² +
2/n -30r2
3C
(9)
1/
8n
This tells us that the expected convergence rate on novel tasks depends on the learned divergence on training tasks, plus complexity term depending on the Frobenius distance the meta-learned optimiser weights M from the identity. Note that restricting the diameter r of the parameter space or neural networks is usually required to obtain generalisation guarantees (Bartlett et al. 2017 Long Sedghi] [0200] Gouk et al.) 2021), so this is not an unusual or counterproductive requirement. We therefore use g (M) = M T|r as a regulariser. The details for computing || /|| are given
in Appendix A.4 Note that as M approaches I, mirror descent will behave similarly to SGD.
4.4
GRATA-GT COMPUTATION WITH IMPLICIT GRADIENT
We optimise the outer objective using a gradient- based optimisation method. Typical meta-learning pipelines will either unroll the sequence of updates performed by an iterative optimiser to solve the
5

---

Under review as a conference paper at ICLR 2023
inner problem (Finn et al. 2017), make use of the Implicit Function Theorem (IFT) to compute derivatives w.r.t. parameters that appear in the inner objective (Lorraine et al. 2020). The number of solution in use-case (i.e., many shot learning) makes
inner loop iterations required to arrive the memory and computation requirements of unrolling this loop infeasible, so we use our prospective MD optimiser to find solution and then apply the IFT method. However, the Bregman divergence parameters do not appear in the inner problem objective, which blocks the hypergradient computation through IFT. To this end, when computing the hypergradients, we use rephrased inner objective expressed in terms of the Bregman divergence parameters. This is possible by using the property that mirror descent finds the fixed point closest to the initial guess of the network parameters, as measured by the corresponding Bregman divergence (Azizan et al. (021). The alternative objective function is
arg min B (0,
(10)
0
s.t. 0 E arg min (6()).
(11)
6'
Framing the constrained optimisation problem as its Lagrangian and replacing the constraints with (Cs(0.) - Ls))) = 0, we obtain
arg arg sup sup B (0,(0,0)) + p(Cs(0.) (s(
(12)
0
1
where 0- is the solution obtained via solving the inner problem with our prospective MD optimiser. and we note that = 0., provides us with an immediate solution to this transformed problem. This means the second term always evaluates to zero when evaluated at the solution, and we therefore only need to apply the IFT to the first term, yielding
38
0² Bos
8d Bod 80 aM
(13)
00
00 00
dM 10 (M)
where the inverse Hessian matrix can be approximated by Neumann series (Lorraine et al. 2020)
EXPERIMENTS
We evaluate our learned optimiser and compare its performance on variety of tasks against commonly used standard optimisers including SGD, SGD-M (with momentum), Adam (Kingma & Ba 2015)
RMSProp (Tieleman & Hinton 2012) and KFAC (Martens & Grosse 2015). and meta-learning
based methods, L20 (Andrychowicz et al. 2016). Meta SSD (Li et al. 2017). Reta-Curvature (MetaCur) (Park & Olival 2019), ARUBA (Khodak et al., 2019), ARUBA+ (Khodak et al., 2019),
PowerSign (Bello et al. (017) and AddSign (Bello et al. (017). We first explore synthetic tasks, followed by shallow neural networks on digit datasets, and then training ResNet on CIFAR-10, before finally evaluating deep neural networks on high resolution images.
Algorithm deployment pipeline: For each set of experiments, we train MetaMD on set of meta- train datasets, and evaluate it on disjoint set of meta-test datasets. In the meta-test stage, models are trained by MetaMD (or competitors) using each dataset's standard training set, and evaluated on the corresponding test splits. We emphasize that for meta-testing, each optimiser consumes the same amount of data, and a comparable amount of computing per iteration, except KFAC is much more expensive due to its inverse operation in every iteration. The learned optimisers such as MetaMD, MetaSGD, MetaCur, and X-Sign use additional data and compute for the prior meta-training stage, but this is one-off cost that can be amortized across different meta-test problems of interest.
5.1 SYNTHETIC PROBLEM: META- QUAARAQUACATIC OPTIMISATION
We start with a synthetic experiment to illustrate the impact of the proposed meta objective, and simply using exact Forward Mode Differentiation (FMD) for hypergradient for learning. Meta Mirror Descent is deployed on family of 2D quadratic optimisation problems from which we can sample a disjoint set of meta- ceta-training and meta-testing optimisation problems. We sample tasks of the form:
min 0J Q0 5TO 0
6

---

Under review as a conference paper at ICLR 2023
NelaND
SGD
SCD-M
Adam
1055
MetaND
5G0
Adom
23
30
30
22 30
30
Figure 1: Trajectory comparison of different optimisers on two quadratic optimisation problems (rows). Columns represent optimisers from left right: MetaMD, SGD, SGD-M and Adam. The point green point denotes the starting point and the orange point denotes the minima. Top row: Iterations convergence 23, 928, 104 and 45 respectively. Bottom Row: Iterations convergence are 280, 27, ,232, 1,838 and 324 respectively. MetaMD is the fastest.
2.3
Traiinn tear BotstedeNISS
1.2
Learning Gltr10 ResMet3
SGD
SCD.M
2.0
300-M
Adar
1.0
AdantV
Adam
100
3.8
KFAC
Loc
Powerrign
MotoCur
3.6
AddSign
Met15GD
P.4
MetaMD
05
5.2
0.
750 10300
1250
1500
3750
2000
Thiniin
Figure 2: Comparison training loss curves for different optimisers. Left: RotatedMNNIST, averaged over all held-out domains. Right: Training loss curve of ResNet18 8 trained on CIFAR10
where Q and are random variables. We give the details of the setting in Appx. A.1 of optimisation trajectories on two kinds of meta-test quadratic problems is shown in Fig. [ All the optimisers are initialised in the same position and stopped when the stop criterion is met. All the optimisers reach the minima, but those that have not meta-learned the curvature information in this family of problems require more iterations to converge compared with MetaMD.
comparison
5.2 LEARNING MIRROR DESCENT FOR NEURAL NETWORKS
RotatedMNIST and MLPs We first evaluate optimiser learning for neural networks using the RotatedMNIST dataset and a 3-layer MLP architecture. RotatedMNISST defines 6 domains by rotating the original MNIST dataset by 0, 15, 30, 45, 60 and 75 degrees. We 5 domains for meta-training, and train MetaMD to convergence in the inner loop, and evaluate the performance on the held-out domain. This process repeated, holding curve is shown in Fig. lefft, and the testing performance in Table toop). We can see that MetaMD converges rapidly and trains models with strong testing performance. The hyperparameter tuning protocol for this and other experiments in this section is explained in Appx A.7
each domain in turn as meta-test. The convergence
Next we explore applying MetaMD to more diverse set of datasets and CNN classifiers. We collect set of datasets, which we denote as DiverseDigits. They include: (c & Cortes (2010), QMNIST Yadav & Bottou| (2019), KMNIST Clanuwat et al. (2018). FasHionMNIST Xiao et al. (2017). USPS Hull (1994) and SVHN Netzer et al. (2011). We train LeNet classifier using MetaMD, resizing all images to 28 x 28 greyscale. The same leave- one-dataset-out protocol used: Each dataset held out turn for evaluation after MetaMD is trained on the other datasets. Compared the previous RotatedMNIST experiment, the distribution
Diverse Digit Datasets and Small CNNs
7

---

Under review as a conference paper at ICLR 2023
Table 1: Test Accuracy (%) on RotatedMNIST and DiverseDigits with 3-layer MIP and LeNet
respectivly. Each column is test dataset, and MetaMD is trained on the other datasets.
Test domain
3-Layer ML.P
SGD
92.23 + 0.57
91.91
+ 0.49
92.57
+ 0.32
92.89
+ 0.35
92.73 + 0.32
92.36 + 0.87
SGD-M
94.77
0.58
94.64
0.14
94.66
0.29
94.67
0.47
0.47
94.47
0.63
Adam
92.96 + 0.58
93.29
+ 0.92
93.51
0.84
93.69 + 0.99
93.67
+ 0.35
92.98
+ 1.17
RMSprop
92.48 1: 0.49
93.56
1: 0.51
92.77
1: 0.50
93.58
1: 0.32
93.43
0.332
93.14
1 0.31
93.48
0.74
93.46
0.44
93.44
0.61
93.84
0.51
93.04
0.22
93.64
0.23
MotaCar
93.92 + 0.45
94.62
+: 0.71
94.98
=: 0.30
94.43
4: 0.43
94,98
4 0.71
93.83
+: 0.92
MctaMD
95.67
0.39
95.59
0.28
95.50
0.61
95.33 0.45
95.03
0.63
94,99
0.52
Tost domain
MNIST
QMNIST
KMNIST
PashionMNIST
USPS
SVHN
SGD
96.44 + 0.91
96.23
0.73
61
1.87
88,95 + 1,43
92.73 + 1.13
85.44
1.22
SGD+M
98.47 1: 0.16
97.21 1: 0.15
92.54
1: 0.62
86.44 1: 0.45
95.37
1: 0.24
86.26 1: 0.48
LeNet
Adam
98.49 1: 0.17
98.
4: 0.33
93.20
:: 0.82
87.36 1: 0.55
93.68 1: 0.38
87.07
1: 0.61
RMSprop
98.65
0.87
87.45 0.13
87.01
0.18
L.0
90.80 + 0.27
92.60 + 0.34
87:27
+ 0.68
82.34 0.56
91.23 + 0.54
34.42
+ 0.43
MetaCur
97.45 1 0.48
97.62
0.67
93.74
0.27
87.50 1: 0.58
93.42 1: 0.67
86.27
0.53
MetaMD>
98.72 + 0.33
98.74 + 0.24
96.56 +: 0.49
87.33 + 0.36
95.96 +: 0.14
86.43 + 0.64
WVST
OMMIST
5004
Sku
Tobre
Teah Rendoon
Thandna lemkee
Figure 3: Learning curves for DiverseDigits. From left to right: MNIST, QMNIST and SVHN.
of tasks used for meta-training and meta-testing is now more diverse and challenging. Due to the greater cost of training the base model here, use T = 500 iterations for the inner loop, and leave efficient meta-learning under longer-horizons as future work. We compare all methods fairly using a common "yyper-paramet tuning protocol for meta-test. Specifically, we perform grid search with respect to meta-test validation accuracy for each competitor, and more detail is given in Appx. A.7
The results averaged over 3 meta-test trials are shown as testing performance at convergence in Table[[[botoom and selected meta-test learning curves in Fig. 3 with the remaining learning curves given in Appx. A.8 We can see that MetaMD is clearly faster than SGD and SGD-M in training con- vergence (Fig. 3. while typically producing models with the strongest generalisation error (Table [ It is noteworthy that MetaMD exhibits strong cross-dataset generalisation here, corroborating our Theorem on cross-task optimiser generalisation.
Application to ResNet18 and CIFARIO We next explore scaling up to deeper architectures, aiming to train ResNet18 on CIFARI0 a held out testing task. To this end, we construct suite of meta-training datasets by combining STL10 (Coates et al.) 2011) and DiverseDigits from the previous setting. ResNet & 10 is a well-studied problem with lots of known tuning tricks for standard optimisers. For fair comparison, we therefore tune all methods with exactly the same BayesOpt- hyperparameter tuning protocol, based on CIFAR 10 validation performance. For this experiment we also compare some other learned optimisers, including: (i) PowerSign and AddSign that are not directly comparable. They were trained on CIFARIO (for us a held out dataset), but on a different architecture Bello et al. (2017). (ii) KFAC which is directly comparable insofar as using the same structure of preconditioner during meta-test via the Fisher Information Matrix of the target problem while MetaMD amortizes this cost and meta-learns it from a disjoint set of tasks, without tuning to the target task. (iv) MetaSGD which we re-implement to use exactly the same meta-train data MetaMD for direct comparability.
MetaMD, but not comparable in that computes the preconditioner
Table 2: Test accuracy CIFARIO using Resnet18 and various optimisers
Mathed Accuracy
KFAC
SGD.M
Adam
AdamW
PowerSign
AddSign
MensGD
MetaMD 93.56 0.34
87,54 1 0.23
91.37 1 0.45
91.64
1 0.56
92.66
0.32
87.77 1 0.32
88.29 1. 0.67
91.54 1 0.17
8

---

Under review as a conference paper at ICLR 2023
Table 3: Test Accuracy (%) on high resolution datasets. Comparison with various optimiser.
Datasets
Caltech
DTD
Flowers
Publig
SGD-M
26.95 + 0.35
32.66
0.61
47.18 : 0.17
74.10 + 0.11
Adam
27.21 +: 0.53
28.09
0.44
41.03 = 0.48
73.31 + 0.18
PowerSign
26.11 0.39
31.91
44.66 +
68.25
0.42
AddSign
27.21 + 0.37
29.52
44.79 = 0.47
67.77 0.29
Meta-SGiD
25.41 +: 0.58
27.93 0 0.34
48.77 + 0.44
66.51 0.36
MetaCur
12.19 2.04
15.96
1.42
6.46
28,98
1.87
ARUBA
21.84 0.74
21.17
1.31
20.26 1 0.79
52.35 + 1.16
ARUBA++
24.26 +: 0.97
25.64
0.64
39.24 + 0.67
67.29
0.63
MetaMD
37.50 + 1.59
41.01 0.87
54.84 + 1.62
77.77 0.87
Publig
Publig
0.0
0.7
F.22
0.0
25
50
75
120
150
200
125 150 175
200
75 120 125 150 175 200
Figure 4: Learning curves on Pubfig. The left, middle and right column represent Training loss, Training accuracy and Test accuracy respectively.
From the results in Fig. (right), we can see that MetaMD converges rapidly as expected; and from the results in Table we see that it also provides the best generalisation performance.
High Resolution Image Application Finally, we explore benchmarks with higher resolution
images with ResNet18. 88 We construct set of datasets including Aircraft (Maji et al. (2013), But-
terfly (Chen al.] 2018), Pets (Parkhi al. 2012), Caltech (Griffin et al. (200)). DTD (Cimpoi
et al. 2014). Flowers (Nilsback & Zisserman, 2008 Pubfig (Pinto et al. 2011). See Appx. A.9 for the details about each. Compared with the (28 x and 32x) images used in the previous experiment
settings, these datasets are much higher resolution images. We resize them to standard 224 x 224 for ResNet18. We divide this suite of datasets into meta-train and meta-test groups. Meta-train contains Aircraft, Butterfly and Pets. Meta-test has Caltech, DTD, Flowers, and Pubfig. Compared with Rotated MNIST and Diverse Digit settings, the meta-train and meta-test domains are much more different, which affects the competitors whose initialisations are part of the meta-training objectives. MetaMD and the competitors including ARUBA, ARUBA++, MetaCur and Meta-SGD, are trained on the meta-train domains. MetaMD requires the model trained to be converged in each outer loop it- eration. To improve meta-training efficiency, therefore we train the base model on 20 subcategories of each domain which are randomly sampled in each outer loop iteration. All the competitors on the meta-test tasks by training ResNet18 8 for each dataset individually. We did fair hyperparameter tuning for every competitor detailed in Appx. A.7 We compare the learning curve and the test set performance of the learned model on each dataset. The learning curves on Pubfig are shown in Fig, 4 and the others are in Appx. AA.9 Online algorithms ARUBA and ARUBA+ converge poorly in terms of the training loss during meta-test. This is due to their prior poor convergence during meta-train, in contrast to the batch competitors that better exploit the meta-train tasks. In contrast, MetaCur converged well during meta-train, but suffers from meta-overfitting and converges poorly during meta-test. Mcanwhile, MctaMD exhibits rapid convergence as expected. In terms of test accuracy on the downstream tasks, Table shows that MetaMD performs best, thanks to knowledge transfer from the source tasks reducing overfitting to these comparatively small target tasks.
tested
CONCLUSION
We explored meta-learning neural network optimisers from the Mirror Descent perspective - in practice by meta-learning Bregman Divergence manipulate the gradient during learning. Our approach has clear theoretical motivation by optimizing a bound on the convergence rate, and has both a convergence guarantee and cross-dataset generalisation guarantee. Our practical implicit- gradient based algorithm surprisingly (Lorraine et al.) 2020) enables meta-learning of learning rates via implicit gradient, and empirically fast and effective compared SGD and other competitors.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Marein Andrychowiez, Misha Denil, Sergio Gomez, Matthew Hoffman, David Pfau, Tom Schaul, Brendan Shillingfrd,, and Nando De Freitas. Learning learn by gradient descent by gradient descent. Advances in neural information processing systems, 2016.
Navid Azizan, Sahin Lale, and Babak Hassibi. Stochastic mirror descent on overparameterized nonlinear models. IEEE Transactions on Neural Networks and Learning Systems, 2021.
Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. Metareg: Towards domain general- ization using meta-regalarizzatonn Advances in Neural Information Processing Systems, 2018.
Maximilian Balandat, Brian Karrer, Daniel R. Jiang, Samuel Daulton, Benjamin Letham, An- drew Gordon Wilson, and Eytan Bakshy. BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization. In NeurlPS, 2020.
Maria-Florina Balcan, Mikhail Khodak, and Ameet Talwalkar. Provable guarantees for gradient-based meta-learning. In International Conference on Machine Learning, pp. 424-433 PMLR, 2019.
Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov)::43334822 2002.
Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. Spectrally-nnormalizedd margin bounds for neural networks. Advances in neural information processing systems, 30, 2017.
Jonathan Baxter. A model of inductive bias learning. Journal of artificial intelligence research, 12: 149-198, 2000.
Irwan Bello, Barret Zoph, Vijay Vasudevan, and Quoc V Le. Neural optimizer search with rein- forcement learning. In International Conference on Machine Learning, 459-468. PMLR, 2017.
Kuilin Chen and Chi-Guhn Lee. Meta-free few-shot learning via representation learning with weight averaging. 2021.
Tianshui Chen, Wenxi Wu, Yuefang Gao, Le Dong, Xiaonan Luo, and Liang Lin. Fine-grained repre- sentation learning and recognition by exploiting hierarchical semantic embedding. Proceedings of the 26th ACM International Conference Multimedia, pp. 2023-2031, 2018.
Yutian Chen, Matthew W. Hoffman, Sergio Gomez Colmenarejo, Misha Denil, Timothy P. Lillicrap, Matt Botvinick, and Nando de Freitas. Learning to learn without gradient descent by gradient descent. In ICML, 2017.
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.
Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature. In NeurIPS (Workshop), 2018.
Adam Coates, Andrew Ng. and Honglak Lee, An analysis of single-layer networks in unsupervised feature learning. In AISTATS, 2011.
Giulia Denevi, Carlo Ciliberto, Dimitris Stamos, and Massimiliano Pontil. Incremental learning-to- learn with statistical guarantees. arXiv preprint / laoe:o00330000 2018
Giulia Denevi, Carlo Ciliberto, Riccardo Grazzi, and Massimiliano Pontil. Learning-to-ler -t-iinn-tt-learn stochas- tic gradient descent with biased regularization. In International Conference on Machine Learning, pp. 1566-1575. PMLR, 2019a.
Giulia Denevi, Dimitris Stamos, Carlo Ciliberto, and Massimiliano Pontil. meta-learning. Advances in Neural Information Processing Systems, 32, 2019b.
Ryan D' D'Orazio, Nicolas Loizou, Issam Laradji, and Ioannis Mitliagkas. Stochastie mirror descent: Convergence analysis and adaptive variants via the mirror stochastic polyak stepsize. arXiv preprint arXiv:2/10.1 15412, 2021.
10

---

Under review as a conference paper at ICLR 2023
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. arXiv preprint arXiv:1703.03400 2017.
Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine. Online meta- meta-learning In International Conference on Machine Learning, pp. 9920-1930 PMLR, 2019.
Sebastian Flennerhag, Andrei Rusu, Razvan Pascanu, Francesco Visin, Hujun Yin, and Raia Hadsell. Meta-leaming with warped gradient descent. ICLR, 2020.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-awar minimization for efficiently improving generalization. In International Conference on Learning Representations, 2021. /ww./ttteee///eeeeee.......... fort:/ooumm?0 htt.:///eeooruuotooooooomoommooa
Luca Franceschi, Michele Donini, Paolo Frasconi, and Massimiliano Pontil. Forward and reverse gradient-ban based hyperparameter optimization. In International Conference on Machine Learning. 2017.
Henry Gouk, Timothy M Hospedales, and Massimiliano Pontil. Distance-hased regularisation of deep networks for fine-tuning. In ICLR, 2021.
Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. 2007.
Hensel. Ueber gattungen, welche durch composititon aus zwei anderen gattungen entstehen. 1889.
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey. arXiv preprint htt::2004.054.399 2020.
Jonathan Hull. A database for handwritten text recognition research. IEEE Transactions on pattern analysis and machine intelligence, 16(5):500-554, 1994.
Mikhail Khodak, Maria-Florina F Balcan, and Ameet S Talwalkar, Adaptive gradient-based meta- learning methods. Advances in Neural Information Processing Systems, 32, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International
Conference on Learning Representations, 2015.
Jyrki Kivinen and Manfred K Warmuth. Exponentiated gradient versus gradient descent for linear predictors. information and computation, 132(1):1-6) 1997.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010.
Ke Li and Jitendra Malik. Learning to optimize. In 2017.
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few-shot learning. arXiv preprint arXiv: 2017.
Philip M Long and Hanie Sedghi. Generalization bounds for deep convolutional neural networks, In ICLR, 2020.
Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. In A/STATS, 2020.
S. Maji, Kannala, F. Rahtu, M. Blaschko, A. Vedaldi, Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. Technical report, 2013.
James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored approximate curvature, In International conference on machine learning, pp. 2408-2417. PMLR, 2015.
Paul Micaelli and Amos J Storkey Gradient-based hyperparameter optimization over long horizons. Advances in Neural Information Processing Systems, 34, 2021.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. 201 1.
11

---

Under review as a conference paper at ICLR 2023
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification a large number of classes. Proceedings of the 6th Indian Conference on Computer Vision, Graphics Image Processing, pp- 722-729. IEEE, 2008.
Eunbyung Park and Junier Oliva. Meta-curvature. In NeurIPS, 2019.
Omkar Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, 2012.
Nicolas Pinto, Zak Stone, Todd Zickler, and David Cox. Scaling up biologically-inspiee computer vision: case study in unconstrained face recognition on facebook. IEEE CVPR Workshops, pp. 35-42. IEEE, 2011.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. In AAAI, 2019.
Jan N van Rijn, Salisu Mamman Abdulrahman, Pavel Brazdil, and Joaquin Vanschoren. Fast algorithm selection using learning curves. In International symposium on intelligent data analysis, pp. 298-309. Springer, 2015.
Mark Sandler, Max Vladymyrov, Andrey Zhmoginov, Nolan Miller, Tom Madams, Andrew Jackson, and Blaise Agüera Arcas. Meta-learning bidirectional update rules. In JCML, 2021.
Nikunj Saunshi, Arushi Gupta, and Wei Hu. A representation learning perspective on the importance of train-validation splitting in meta-learning. In International Conference Machine Learning, pp. 9333-9343. PMLR, 2021.
Shai Shalev-Shwards and Shai Ben-David. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.
Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning. 4(2):26-31, 2012.
Olga Wichrowska, Niru Maheswaranathan, Matthew W Hoffman, Sergio Gomez Colmenarejo. Misha Denil, Nando Freitas, and Jascha Sohl-Dickstein. Learned optimizers that scale and generalize. In International Conference on Machine Learning, pp. 3751-3760, PMLR, 2017.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv: arXiv: 778.007747 2017.
Chhavi Yadav and Léon Bottou. Cold case: The lost mnist digits. Advances in Neural Information Processing Systems 32. Curran Associates, Inc., 2019.
Zehfuss. Über eine gewisse determinante. Zeitschrift für Mathematik und Physik, 3(8851))298-31, 1858.
Jingzhao Zhang, Hongzhou Lin, Stefanie Jegelka, Suvrit Sra, and Ali Jadbabaie. Complexity of finding stationary points of nonconvex nonsmooth functions. In ICML, 2020.
Barret Zoph and Quoc Le. Neural architecture search with reinforcement learning. ICLR, 2017.
12

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
A.1 QUADRATIC SETTING
Meta Mirror Descent is deployed on family of 2D quadratic optimisation problems from which we can sample disjoint set of meta-training and meta-testing optimisation problems. We sample tasks of the form:
min 0J TTQ0-=QQ--0000 6
where Q and are random variables. b follows Gaussian distribution with mean vector [1, 1J" and identity covariance. To generate Q, which controls the flatness of the quadratic function, we sample Q0,o and 11.1, independently with different Gaussian distributions. For example, on the top Figure T the mean of Q0,o and Q1,1 are 0.3 and 14 while on the bottom row, the mean of Qo,o and Q1,1 are 0.02 and 14 respectively. In addition, the initialisation of each optimisation problem is also a random variable.
in
A.2 DERIVE OF THE CLOSED FORM MIRROR LOOP
In our setting, the mirror loop described as
1
0c+1 arg min((v + - (0)00)
(14)
2n
or, equivalently,
0c+1 arg min (VoC,, B3
(15)
Setting the gradient w.r.t. zero, we have
vVL(0), Vp(00+1) - O,
(16)
which when rearranged yields
vo(0.+1) = Vo(0)- (V(() 8141 VΦ- V vV..))
(17)
(18)
In our case
8T MO, 6(0) OF
(19)
where M is a block diagonal matrix. Therefore,
Vo(0) M0
(20)
VΦ (0) 1-10.
(21)
As result, we also have
0++1 V (Vo(0)) V(.)))
(22)
M MM ((.))
(23)
0e -1
(24)
A.3 PROOF OF THEOREM 4.1
We will make use of well-known bound suprema of empirical processes due to Bartlett & Mendelson (2002).
Theorem A.I. For all functions f E with || S a, and ii.d. Z, the following holds with probability at least 8.
In(2/8)
E[f(Z)] < n 1=1
Σ
f(Z,) + +2), + 3ay
(25)
2n
13

---

Under review as a conference paper at ICLR 2023
In this theorem, (F) is the empirical Rademacher complexity of the class F, defined
Rs (F) Ex
sup Σ fEr im1
fif (Zi)
(26)
where e; a Rademacher random variable, so P(e; = -1) P(es = =
Proof. It suffices to bound, with high confidence, the difference between the first of the meta- objective, and the expected Bregman divergence between initializations and solutions on new tasks sampled from the same task distribution. We will obtain such bound using Rademacher complexity, and the main result will follow from standard applications of Rademacher complexity- gener- alisation bounds (Bartlett & Mendelson 2002), along with the observation that (g [|0)) Cr- That is, we will treat the mittrll and final points of an optimisation trajectory for each task as random variable, and will bound what extent the mean Bregman divergence between initial and trained parameters on some meta-trai tasks can deviate from the expected Bregman divergence on unseen tasks. In particular, we analyse the following class:
1 M0,
== ((0,,0.) (0.0)100:) 6(0)
00
|M
- I|IE S CC..
(27)
2
Denoting 0.¹) 0 by 0.. can bound the Rademacher complexity of this class from above by
(J) = E.
sup BBEF
e (o.
(28)
n
n
E.
sup
O Mo;
(29)
2n
f=1
E.
sup
e, (M, 0,
(30)
2n
i=1
E.
Sup(M, ,,,00F)
(31)
2n
i=1
E.
sup((M - 1), e0,00 +
E. 2n
(1,
-.>
0,0
(32)
2n
M
where (r is the Frobenius inner product. Note that the second term in the final equality is equal to zero. As such we can continue by further bounding the first term using the Cauchy- inequality,
C Ex 2n
RA S
00,0%'
(33)
The remainder of the proof follows well-known sequence of steps using when bounding the expected norm of a Rademacher sum (see, e.g., (Shalev-Shwarzz & Ben-David 2014)), which we include here for completeness. Jensen's 's inequality tells us that
E,
Σ
00,0
<
E,
0,
(34)
i=1
F
E2
E 10.0 0,05)r
(35)
14

---

Under review as a conference paper at ICLR 2023
Noting that E/c,e,] is one if = j and zero otherwise, obtain
n
E
Ge (0,0 0,
110,0 11:
(36)
i=1
n
(10.112110.112²
(37)
$ nrr.
(38)
Substituting this back into our earlier derivation yields
CV
(F)
(39)
2n
Cr2
(40)
n/n'
which concludes the proof.
A.4 META REGULARISER COMPUTATION
We provide the meta regualriser computation details in this section. The proposed meta regulariser is expressed as
|| - 112 || + || - 111 + 2(M,
(41)
||M + ||/||² 2tr(M)
(42)
-1IM112 + 11113 - 2(r(A))r(B)
(43)
Σ (A) + |1 - 2tr(A)tr((F)
(44)
where u-(-) denotes the i-th singular value of matrix. To get from Eq tr(A & B) = (B))) and from Eq. 43 to Eq. 44 we use that M Σig (B)
42
Eq
43
that
Σ
A
&
B)
A.5 COMPUTING À
In this section, we provide the computation details for A, used in Equation g
À min Umin (Mg)
(45)
min Unin (Ac Ø Bc)
(46)
min min 12
0z (Ac)u,
(47)
where vz(-) denotes the i-th singular value of matrix. From Eq, 46 Eq. 47 we that v(A@B)
(AA) v vBB)" where v(.) gives all the eigenvalues in the column vector form.
A.6 GRADIENT COMPUTATION FOR DIAGONAL MATRIX
Diagonal matrix is special case of block diagonal matrix, which we applied in the 2D quadratic setting in section 5.2 With this simple parameterisation, the exact hypergradient w.r.t the target optimiser can be computed with Forward Mode Differential (FMD). The gradient of the second term in the proposed meta-objective in Eq Eqgin is easy to compute while the first term with respect to is expressed as:
ÖB (0., 01) 2M
0Box (0T, 01)
(48)
aM a Bo m (0r, 01)
a aBo. (0r:0)) 00
(49)
aM
00r
0M
direct grudient
indirect gradient
15

---

Under review as a conference paper at ICLR 2023
when T is large enough to satisfy that 8, 8r. The computation of the direct gradient can be easily solved by the existing auto auto-differentiation library. The indirect gradient in Eq usually termed hypergradient, is much more computationally chanllenging as it is expressed in the form:
80r
Σ
II
Ar Br
(50)
0M
tml
1-1++
s.t. A, = @ToN (02-1) = - 111 002
0
for (1 (0(-1);
(51)
00,-1
B. Tnou (0.- 2n 00 Cir (0, (0--))
(52)
0M
Forward-Moce Differentiation (FMD) and Moverse-Mode Differentiation Franceschi et al. (2017)
two algorithms to compute Eq 50 RMD computes the gradient from the last to the initial step.
requiring one to store the entire optimisation trajectory in memory. When the optimisation trajectory
is long, this computation is very expensive. In comparison, FMD updates the hypergradient in parallel with in inner loop optimisation by:
a0; Teor (0c-1)
00,-1
@Toar (01-1)
(53)
0M
09;-1
0M
0M
where it only requires the information from step t 1.
A.7 HYPERPARAMETER TUNING
Grid Search For tuning the hyperparameters on 3-layer MLPs model settings in section 5.2
we sweep over the learning rates {0.1, 0.05, 0.01, 0,000,,000000 and weight decay parameters of (0.001, 0.0001, 0.0005} for the SGD, SGD-M and RMSprop. In terms of Adam, we do grid search over the learn rates {0.3, 0.2, 0.1, 0.01, 0.001) and weight decay {0.001, 0.0001, 0.0005]. For L20, MetaCur, ARUBA and ARUBA++, we tune the models on the meta-test setting with the learning rates {0.1, 0.01, 0.005, 0.001, 0.0005, 0.0001) and weight decay (0.001, 0.0001, 0.000,00000100000055
Bayesian Optimisation We implement our BayesOpt using (Balandat et al. 2020) for the hy- perparameter tuning on the ResNet18 8 and CIFARIO setting. The model the expected performance using a Gaussian process with RBF kernel, which maps the learning rate and weight decay to the estimated validation accuracy. This also provides uncertainty information to the Upper Confidence Bound (UCB) acquisition function for exploring/explottngg the hyperparameter space. For each model selection in the meta-test stage, run the Bayesian optimisation for 25 iterations.
A.8 TRAINING LOSS LEARNING CURVE FOR DIVERSEDIGITS DATASET
We give all the training loss learning curves on DiverseDigits in Fig 5 It can be noticed that the conclusion we drew that MetaMD is clearly faster than SGD and SGD-M in training convergence in Section 5.2 is further supported.
ONNIST
KORUST
2.
Nodoo
6.0
0.0
5.-
2075
$3.5
6.5
85
Figure 5: Convergence comparison of different optimisers on DiverseDigits.
16

---

Under review as a conference paper at ICLR 2023
Cattech
Poua
a.5:
23 30
73
150
123
150
275
250
25
30
385
125
150 375 200
0.00
25
55
15
150
115
150
215
200
PTD
RTD
STD
12 340
$0.30
3 20225-
3.200
7 20.15
020
50
1:0
150
I7S
250
25
105
150 175 200
25
55 75
125
153
175 200
Falnin (002h
Rowers
Rowers
1.0
0.5-
Pewentinn
T1 109
2.4
0.0
0.0
Trainina 1in Cooch
Trahnin eauch
Fainin (000)
1.0
20.0
20.0
0.4
Foz
Misido
75 150
200
375
2000h
Trai DO esech
Transin eaach
Figure 6: Learning curves on Caltech, DTD, Flowers and Pubfig. The left, middle and right collumn represent Training loss, Training accuracy and Test accuracy respectively
A.9 EXPERIMENT DETAILS ON HIGH RESOLUTION IMAGE DATASETS
We provide summary of each dataset applied in our High Resolution tasks and give a clear split for the meta-train and meta-test datasets in Table + training. testing splits. As a result, we build our own split with 60 and 20 examples for each classes in training and testing stage respectively. Both MetaMD and competitors are tuned on learning rates,
All the datasets except Caltech-256 has standard
Table 4: Statistics for the datasets used throughout the experiments. The Train, and Test columns contain the number instances in each of the corresponding subsets.
Dataset
Train
Test
Classes
Aircraft (Maji et al. 2013
6,667
3,333
100
Meta-Train
Butterfly (Chen et al.
2018
10,270
15,009
200
Pets (Parkhi et al. 2012
4,000
3,390
37
Caltech Griffin et al. 2007
12,800
5,120
256
DTD Cimpor et al.
2014)
3,760
1,880
47
Meta-Test
Flowers (Nilsback & Zisserman
2008
2,040
6,149
102
PubFig (Pinto et al.
2011)
12,178
1,660
83
10.1, 0.01, 0.001, 0.0001) and weight decays, (0.001, 0.0005, 0.0001, 0.0000.., The learning curve for Caltech, DTD, Flowers and Pubfig are shown in Fig 6
A.10 Loss LANDSCAPE ANALYSIS
The generalisation ability is reflected by the flatness of the converged loss landscape (Foret et al. 2021). Motivated by this, compare the converged loss landscapes achieved by MetaMD with those by SGD on the High Resolution Image setting. From Figure [ it can be observed that the landscapes reached by MetaMDD much more flattened than SGD.
17

---

Under review as a conference paper at ICLR 2023
Caltech
DTD
50000
MetaMD
140000
MetaMD
40000
SGD
120000
SGD
100000
CE Loss
30000
CE Loss
80000
20000
60000
40000
10000
20000
0
-1.5
-1.0
-0.5
0.0
0.5
1.0
1.5
1.5
-1.0
-0.5
0.0
0.5
1.0
1.5
Perturbation
Perturbation
Flowers
Pubfig
50000
MetaMD
80000
MetaMD
40000
SGD
SGD
60000
Loss
30000
CE Loss
40000
20000
10000
20000
1155
-1.0
-0.5
0.0
0.5
1,0
1.5
-1.5 -1.0
-0.5
0.0
0.5
1.0
Perturbation
Perturbation
Figure 7: 1D Training Loss Landscape on Caltech, DTD, Flowers and Pubfig, where perturbations performed in the direction of the eigenvector of the loss Hessian matrix corresponding to the largest eigenvalue.
A.11 LOw RANK REPRESENTATION ANALYSIS
The recent research (Saunshi et al. 2021 Chen Lee 2021) claim that low rank representation generalizes well. Following the low-rank representation analysis proposed in (Chen & Lee 202T), we compare the learned feature space of the model trained by SGD and that trained by MetaMD in the high resolution setting in Fig. 8 Σ; Ágogğ, denotes the low rank level, where the smaller value represents the lower rank and X = >: Aa/Amaz represent the normalized singular values. One can observe that MetaMD has the lower rank feature spaces on all the meta-test tasks than SGD. The decay speed of the normalized singular values also indicates that MetaMD learns the lower rank feature space than SGD.
A.12 HYPERGRADIENT COMPUTATION
In this section, we introduce details of the inverse Hessian computation. The hypergradient of the meta-objective & w.r.t. to the optimiser parameters M is computed through
30 3E ( Box
2 Bass
82 Bass
82 Bos ae am
Σ
lim 1-
0M
00
08 00
de 9M
00
de 0e
PM (...8((M)
3=0
ΦM (ad)
where following Lorraine et al. 2020p, the inverse Hessian matrix is approximated by Neumann series:
8 Bost 30 00
8 B 30 a0
lim Σ
In practice, the approximation iteration j does not need to go to infinity_ Lannitt_addiia and in our case we set it as 20. We give the implementation details for computing ØE in Algorithm by adapting the algorithm proposed (Lorraine et al. 2020) to our problem, where a is the hyperparameter controlling the iteration step length.
18

---

Under review as a conference paper at ICLR 2023
testects
singg
zitupé
Neovo 16.32
siags 23,60
2.6
0.5
30.4
0.2
0.3
130
150
235
250
532
165
130
200
370
Singuaa vaiue
5.ggu
Plonaz nic Neshit18
Pobtly with taplects
5G0 22 30.50
op 35.28
Meuut 24loph 20
VidaNo 22/ohh 2759
R2
0.
Suguu or valaa dod
Singuler Value rosx
Figure 8: Low rank representation Analysis on Caltech, DTD, Flowers and Pubfig.
Algorithm 2 Computing the hypergradient of the meta-objective &, with respect to the optimiser parameter M. The "rad function from PyTorch computes Jacobian-vector product when called with non-scalar first argument. Inspired by Lorraine et al. (2020). we use this to efficiently compute the Hessian required for approximating the Neumann series.
Input: Bos E, OM, 02
0" B Bos Output: a00M ag for all j
bb 8
do
0 B 00
wー grad
at
0, v)
p -+=00 V
end for
A.13 CONVERGENCE RATE COMPARISON
We aim to learn optimisers with fast convergence speed. To evaluate the training speed of the learned MetaMD and to compare it with other methods, we measure the area under the curve (AUC) of the training loss (Rijn et af. 2015) where small AUC value indicates an optimiser converges fast. Table 5 shows that the learned MetaMD has the fastest convergence speed in the CIFARIO 10 and ResNet 8 setting while MetaMD ranks second in the High resolution setting in Table To further illustrate the property of MetaMD,, we compute the wall clock time of each optimisation iteration for each model in Table 7 Our model can still outperform other meta-learm precondition methods, such MetaCur.
A.14 FINE TUNING WITH LEARNED METAMD
We studied the transferability of the learned MetaMD by shifting dataset between the meta-train and meta-test stage. A different kind of domain shift is to deploy the learned MetaMD for fine-tuning the base model, which different to the meta-train stage where the base model is learned from scratch in each each inner loop. We reuse the High resolution setting in Section 5.2 MetaMD is trained on all the meta-tran train datasets with the Algorithm but deployed to fine-tune the ResNet18 pre-trained ImageNet on each individual meta-tess test dataset in the meta-test stage. Compared with the hand-craft optimisers including, SGD-M and Adam, with fair hyperparameters selection, MetaMD has competitive performance in terms of test accuracy in Table g
19

---

Under review as a conference paper at ICLR 2023
Table 5: Area under the learning of the training loss for CIFARIO and Resnet18. Comparison with various optimisers.
Method SGD-M
Adam
AdamW
KFAC
PowerSign
AddSign
Meta-SGD
MetaMD
CIFARIO
32.04
15.27
20.05
27.77
34.06
32.90
11.82
Table 6: Area under the learning curve of the training loss for high resolution task. Comparison with various optimisers.
Method
SGD-M
Adam
PowerSign
AddSign
Mcta-SGD
ARUBA
ARUBA ++
MetaCur
MetaMD
Caltech
106.80
69.77
108.23
111.66
176.39
409.97
327.83
736.83
90.20
DTD
434.23
94.35
138.44
133.88
153.68
509.23
379.49
469.52
132.84
Flowers
111.32
74.78
145.12
142.23
112.06
672.46
386.41
332.81
94.25
Publig
56.80
44.19
82.33
68.77
81.08
416.87
274.47
512.78
47.88
Average Rank
4.00
1.00
5,00
4.25
5.00
8,50
7.00
8.25
2.00
A. NUMBER OF META-PARAMETERS
We compute the number of the meta- pattmeaaarmeerss for every meta-optimiser including Meta-SGI MetaCur and MetaMD in Table9 It can be noticed that MetaMD has a significantly small number of meta-parameters to learn compared with other meta-optimiser, which is one of the reasons that MetaMD suffers less from meta-overfitting.
A.16 WEIGHT DRCAY STRENGTH AS DOMAIN SHIFT
We simulate the domain shift by changing the weight decay strength to evaluate the transferability of the learned optimisers. During the meta-train stage, the optimisers learned by setting the weight decay strength as zero, and we test the model on the meta-train dataset with various weight strengths including (0.0, 10-5, 10-1, 10-33.. In Figure
notice that MetaMD quite stable with the simulated domain shift compared with SGD and Adam and has the lowest AUC on all the settings.
20

---

Under review as a conference paper at ICLR 2023
Table 7: Wall clock time iteration (mean and standard deviation) for High resolution task with ResNet18. Comparison with various optimisers.
Method
SUD.M
Adam
PowerSign
AddSiga
Meto SGD
ARUBA
ARUBA
MetiCCr
MeiaMD
Time
18.21 2377
12L16 + 2,62
121.30 2.38
122.45 3.21
120.29 3.32
118.45 = 3,63
118.38 3,42
172.63 3,6
131,22 3,62
Table 8: Test Accuracy (%) and AUC on high resolution datasets by fine-tuning ImageNet pre-trained ResNet18. Comparison with various optimisers.
Datasets
Caltech
DTD
Flowers
Pubfig
Avg Rank
SGD-M (Fine-tuning)
77.81 0.73
67.13
0.58
91.56
0.54
89.52
0.39
2.5
Adam (Fine-tuning)
72.44 0.19
68.46
0.77
90.12 0.28
2.5
MetaMD (Fine-tuning)
77.93 + 0.44
68.66
0.93
91.67
0.75
0.41
1.5
SGD-M (AUC)
154.98
78.58
22.12
59.06
2.5
Adam (AUC)
70.64
42.39
26.86
59.62
2.0
MetaMD (AUC)
150.40
77.23
21.63
54.29
1.5
Table 9: Number of Meta-parametton for various Meta-optimisers
Model
MetaSGD
MetaCur
MetaMD
11176512
2974717
37664
Meta-train: Airplane
Meta-train: Butterfly
250
SGD-M
240
SGD-M
Adam
Adam
240
MetaMD
220
MetaMD
230
200
220
210
180
200
190
140
0.0
1e-5
1e-4
1e-3
0.0
1e-5
1e-4
1e-3
Weight Decay Strength
Weight Decay Strength
Figure Area Under the Curve (AUC) comparison with various optimisers on different Weight Decay strength.
21


Reviews:
[{"review_id": "gjaNnh3kQaD", "reviewer": "Reviewer_bxcN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper approaches meta-learning from a mirror descent perspective, considering a setting where the potential of the Bregman divergence is meta-learned. It proposes a method that can be efficiently implemented by restricting the form of the potential and using the IFT to be practical. Experiments show improved performance compared to competing methods.", "summary_of_the_review": "The proposed method strengthens the idea that the MD framework can be successfully adopted for meta-learning, and while a compromise is made in terms of expressivity (the form of the potential is still somewhat restrictive) to achieve reasonable computational costs, the experiments show improvements over competing methods nonetheless.\n\nThe main weaknesses, in my opinion, are the lack of results on more standard tasks such as miniImageNet and MetaDataset, and a missing comparison against WarpGrad. These would be extremely valuable and would enable a more rigorous and complete evaluation of MetaMD.\n\n------------\n\nI believe reviewer Ced8 raises key concerns which I had overlooked in my initial review. The authors' response does not fully address these points, especially regarding the suboptimal performance of baseline methods due to the adoption of lightweight HPO instead of using hyperparameter values recommended by the corresponding papers. This could result in an unfair advantage to MetaMD due to a possible increased robustness to hyperparameter settings -- an advantage that might only manifest when a low budget HPO is deployed which is effectively choosing poor hyperparameter values. In light of this, I have decided to decrease my score.", "strength_and_weaknesses": "- The idea is well-motivated and the approach is promising. Adopting the Mirror Descent framework for meta-learning is something that has been discussed in the community but a practical yet expressive approach was still to be found, which I think this paper accomplishes. The form of the potential is not too constrained (although being block diagonal is somewhat restrictive) and the use of the IFT seems to fit well in this setting.\n- The description of the method is mostly clear, but Algorithm 1 could have more details. Line 8 could be expanded to have the actual form of the hypergradient (following Eq 13) and what steps are used to compute it (Neumann series, as mentioned right before Section 5?). H seems to be an accumulator for the (hyper)gradients of M, and this could also be stated somewhere in Algorithm 1.\n- Experimentally, the proposed method comfortably outperforms competing approaches in most tasks, but it would be very valuable to also consider commonly-adopted few-shot learning problems such as miniImageNet and MetaDataset. Some of the tasks used in the comparison seem less standard. Some of the considered methods also perform very poorly on the considered tasks but are known to perform well on miniImageNet (like MetaCurvature): it's unclear whether these methods are incompatible with such tasks for some reason, or if they were trained under less than optimal settings.\n- A comparison against WarpGrad is expected but not given in the paper. This is an important evaluation since WarpGrad is somewhat motivated by MD and is more recent than most, if not all, of the considered competing methods.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "W6pRF4JZVD", "reviewer": "Reviewer_c6CM", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper provides a method for meta-learning a preconditioner for gradient descent. The authors motivate the method from the mirror descent perspective, and use a convergence rate bound as a training objective. The authors parameterize the preconditioner in the K-FAC format [1]. Empirically, the method achieves good results across a range of problems.", "summary_of_the_review": "Overall, this is a nice paper with good empirical results. I hope the authors can provide clarifications to my questions above.\n\n## References\n\n[1] Optimizing Neural Networks with Kronecker-factored Approximate Curvature\nJames Martens, Roger Grosse\n\n[2] Meta-Learning with Warped Gradient Descent\nSebastian Flennerhag, Andrei A. Rusu, Razvan Pascanu, Francesco Visin, Hujun Yin, Raia Hadsell", "strength_and_weaknesses": "## Strengths\n\n**S1**: The paper is clearly written and easy to follow, especially given the fairly technical subject. I enjoyed reading it.\n\n**S2**: The proposed method is well-motivated theoretically, with connections to mirror descent.\n\n**S3**: The authors derive a new bound on the generalization performance of their learned optimizer in Theorem 4.1 (I did not check the proof of this bound).\n\n**S4**: Empirically, the method performs well on a range of tasks.\n\n## Weaknesses\n\n**W1**: While the authors present the method quite generally, as learning the Bregman  divergence for mirror descent, they restrict the class of these divergences so that the method is learning a preconditioner for SGD, which has been considered before.\n\n**W2**: The motivation for the training objective in Eq. 8 is not immediately obvious to me.\n\nI discuss the weaknesses in more detail below.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HhF_JjUOerv", "reviewer": "Reviewer_Ced8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a meta-optimised mirror descent scheme for large-scale optimization problems, based on tuning of a parametric Bregman divergence on a set of curriculum meta-training task. The expected benefits of this meta-learning procedure is to produce an optimiser with accelerated convergence that is transferable to similar optimization problems, while retaining theoretically provable convergence guarantees. The authors provide a theoretical discussion regarding Stochastic mirror descent as well as an optimization framework leveraging the Implicit Function Theorem for meta-optimizing the Bregman divergence used in the mirror descent scheme. The authors further describe a particular construction of the Bregman divergence based on block-diagonal Kroenecker factorization of a quadratic form, aiming at finding a trade-off between computational complexity of the inner optimization problem and meta-adaptability, as well a convergence result of this approach in the convex case. The paper provide experiments showing the doability and the potential benefit of the method (in terms of test accuracy) on a set of synthetic problems / MLP and Convolutional NN training on MNIST-like datasets classification/ ResNet-18 on higher resolution images dataset classification.", "summary_of_the_review": "Overall, while I personally find this line of work very interesting and would like proposals such as MetaMD to come through, I feel that the current experimental section has too many weaknesses which makes this submission not ready for publication. I am willing to discuss more specifically with the authors the points (I-IV) raised above and to increase my score should they provide more compelling experimental evidence, in priority regarding transferability and acceleration. ", "strength_and_weaknesses": "Strength : From a formal point of view, I feel that this work is clear and well written : understanding is straightforward, the authors motivates well the proposal, which is further discussed in comparison with the relevant meta-optimization literature. The theoretical and experimental results are also clear without ambiguity. \n\nHowever, I have several concerns regarding the applicability of the method and the execution of the experimental section that I list below in order of importance: \n\n - I) In my opinion, an important question is how transferable is the learnt divergence when applied to new optimization problems, as it determines the genuine applicability of the method and the benefit of the expensive meta-training procedure. It seems that this is very partially answered in the experimental sections with mixed results: In the DiverseDigits experiment, the training loss function seems to plateau at a slightly higher level than two off-the-shelf optimizers without meta-adaptation: RMSProp and Adam, which calls into question the transferability of $B_{\\phi}$ for datasets that are most dissimilar to the meta-training curriculum. In order to form an idea of how general is the learnt meta-optimizer, It would be beneficial to the paper to provide a quantification of the method transferability when controlling tasks dissimilarities (by varying datasets or the loss function for instance).\n\n- II) Another point that weakens the paper argument is that there is no empirical results regarding the acceleration proposed by this method, whereas it is one claim of the abstract. Apart from the iteration count in the toy dataset, no quantification of the acceleration induced by metaMD is provided. Worse, the different optimization baselines are not controlled for computational budget (such as total number of flops or time for reaching stopping criterion for instance). The meta-training procedure is compute-intensive and the argument that it constitutes a \"one-off cost that can be amortized across different meta-test problems of interest\" would hold only if transferability (point ii) is valid which is hardly decidable from the current set of experiments.\n\n - III) Some of the performance metrics seems surprisingly low compared to what is usually reported in the literature. For instance the accuracy regime of ResNet-18 for Caltech and Flowers dataset is lower than 30% for all baselines. By inspecting more closely the training curves in Appendix (figure 6) It seems that this low test performance mainly comes from substantial overfitting. To the author point, the benefit of MetaMD is to provide an implicit regularization of the optimization problem (since meta-training is performed on the other datasets). Since classification test accuracy is the only selected metric, the unusual overfitting regime in which the tested optimization scheme are compared seems to artificially favor MetaMD. \n\n - iv) One explicit motivation of the paper for meta-optimizing a stochastic mirror descent scheme (SMD) is to retain the well-established theoretical guarantees offered by SMD method. However, the authors propose a convergence rate in the convex setting, which is fairly restrictive and ill-aligned with the experiments over non-convex problems carried in the paper. There exists a rich literature on convergence of SMD in the non-convex case and I feel that such theoretical results provided are uninformative. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6a3MAsZOfC", "reviewer": "Reviewer_f8bx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In summary, this paper:\n1. Proposes to attack the problem of learning gradient-based optimizers by considering the space of gradient updates which can be represented in the Stochastic Mirror Descent (SMD) framework. \n2. In the SMD framework, the 1-step gradient updates are all parameterized by the expression of the Bregman Divergence. In the current paper's setting, quadratic costs $\\theta^{T} M \\theta$ with learnable matrix $M$ are considered.\n3. Quite a few techniques are used to make learning $M$ feasible; they are primarily:\n    * Expressing $M$ as block diagonal matrices to reduce the size, where each block corresponds to a layer in the neural network. Each block is also parameterized by a Kronecker factorization to further reduce computation.\n    * Using an upper bound on convergence rate, as the actual objective to differentiate $M$ across.\n    * Using implicit gradient descent to further reduce the explosive memory/computation blowups from auto-differentiating through a gradient update chain.\n4. Experiments show solid results on toy quadratic functions, meta-training MNIST variants, and meta-testing on CIFAR10 + ResNet-18. Lastly, \"larger\" tasks were also experimented on with their own meta-train / meta-test, by increasing image resolutions over a distribution of datasets (Aircraft, Butterfly,...etc.) to 224x224. \n   ", "summary_of_the_review": "The paper is a solid application of well-known optimization techniques in the literature, and has demonstrated this with both conceptual and experimental contributions. The main boost to the paper's score and impact, is if my question about applying the method over larger-scale experiments are answered. ", "strength_and_weaknesses": "# Strengths\n* Even though the proposed method has multiple steps, the paper is very well-written and conveys all such steps clearly, and makes itself accessible to even non-experts.\n* The method appears to be quite natural, theoretically motivated, and utilizes well-known techniques from the literature (convex optimization, bilevel optimization, etc).\n* The experimental results appear to be quite strong, with performance gains over numerous baselines within the few percentages, sometime within the tens of percentages on the \"High Resolution Image Application\" benchmark.\n\n# Weaknesses\n* While I am not an expert on the particular topic of Stochastic Mirror Descent, it appears that the method is only applicable to SGD-like optimizers, and momentum-based optimizers such as Adam are not represented here. If so, is there a way to extend the method to the momentum case? Momentum-based optimizers such as Adam are still the de-facto choice for applications such as reinforcement learning.\n* The largest network used in this paper, ResNet-18, even if applied to 224 x 224 image sizes, is still quite small in the field of deep learning at the end of the day. While I am not explicitly requiring the authors to try the method on much higher compute budget experiments (e.g. training large Transformers, ResNet-101 on ImageNet-sized datasets), I do want to know if the method would still hold up, or if certain parts may need to be changed. \n    * Answering this would greatly improve its impact over today's large-scale training, which in all honesty are probably the intended audience for such a work anyways, since the potential benefits over e.g. large language model training are far bigger than simply improving accuracies over MNIST or CIFAR10.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject