Generalizable Learning to Optimize into Wide Valleys
Authors: Junjie Yang
Tianlong Chen
Mingkang Zhu
Fengxiang He
Dacheng Tao
Yingbin Liang
Zhangyang Wang
Venue: ICLR 2022 Submitted (2022)
Keywords: L2O
Generalization
Flatness
Entropy-SGD

Abstract: Learning to optimize (L2O) has gained increasing popularity in various optimization tasks, since classical optimizers usually require laborious, problem-specific design and hyperparameter tuning. However, current L2O approaches are designed for fast minimization of the objective function value (i.e., training error), hence often suffering from poor generalization ability such as in training deep neural networks (DNNs), including ($i$) disappointing performance across unseen optimizees $\textit{(optimizer generalization)}$; ($ii$) unsatisfactory test-set accuracy of trained DNNs ($\textit{optmizee generalization}$). To overcome the limitations, this paper introduces $\textit{flatness-aware}$ regularizers into L2O for shaping the local geometry of optimizee's loss landscape. Specifically, it guides optimizee to locate well-generalizable minimas in large flat regions of loss surface, while tending to avoid sharp valleys. Such optimizee generalization abilities of $\textit{flatness-aware}$ regularizers have been proved theoretically. Extensive experiments consistently validate the effectiveness of our proposals with substantially improved generalization on multiple sophisticated L2O models and diverse optimizees. Our theoretical and empirical results solidify the foundation for L2O's practically usage. All codes and pre-trained models will be shared upon acceptance.

Under review as a conference paper at ICLR 2022
GENERALIZABLE LEARNING TO OPTIMIZE INTO WID VALLEYS
Anonymous authors Paper under double-blind review
ABSTRACT
Learning optimize (L20) has gained increasing popularity in various optimiza- tion tasks, since classical optimizers usually require laborious, problem-specific design and hyperparameter tuning. However, current 1.20 approaches are de- signed for fast minimization of the objective function value (i.e., training er- ror), hence often suffering from poor generalization ability such as in training deep neural networks (DNNs), including (i) disappointing performance across seen optimizees (optimizer generalization): (ti) unsatisfactory test-set accuracy trained DNNs (optmizee generalization). To overcome the limitations, this paper introduces flatness-aware regularizers into 1.20 for shaping the local geometry of optimizee's 's loss landscape. Specifically, it guides optimizee to locate well- generalizable minimas in large flat regions of loss surface, while tending to avoid sharp valleys. Such optimizee generalization abilities of flatness-wware regular- izers have been proved theoretically. Extensive experiments consistently validate the effectiveness of our proposals with substantially improved generalization on multiple sophisticated L20 models and diverse optimizees. Our theoretical and empirical results solidify the foundation for L20's practically usage. All codes and pre-trained models will be shared upon acceptance.
INTRODUCTION
One cornerstone deep learning's success ar- guably the stochastic gradient-based optimization methods, such as SGD (Robbins & Monro, 1951), Adam (Kingma & Ba, 2014), AdaGrad (Duchi et al., 2011), RProp (Riedmiller & Braun, 1993), and RM- SProp (Tieleman & Hinton, 2012). The performance of deep neural networks (DNNs) hinges on the choice of optimization methods and the corresponding pa- rameter settings. Thus, intensive human labor often required empirically select the best optimization method and its parameters for each specific problem.
Flatness-avare Regularizer
Learnable Optimizer
Update Rule
Training
Dynamnies
Optimizee NNs
A seemingly promising data-driven approach, learn- ing to optimize (L20), arose from the meta learning community alleviate this issue. It aims replace traditional optimizers tuned by human hand with neu- ral network based optimizers that can learn update rules from data. Existing works have demon- strated that learned optimizer is not only able to decrease the objective function faster, but also able to tremendously reduce the required human labor. Andrychowicz al. (2016a) first proposed to parameterize the update rules using a long short-term memory (LSTM) network. The LSTM optimizer tries to simulate the behavior of iterative methods by unrolling. By aggregating the loss of the function to be optimized (optimizee) at each time step, it aims to minimize the overall loss along the optimization path. Wichrowska et al. (2017) enlarged the optimizer neural network to a hierarchical recurrent neural network (RNN) architecture, which improves its capability larger or unseen optimization problems. Li & Malik (2016) also proposed a reinforcement learning based approach for this sub-field.
The pipeline illustration of our pro- posed Flatness-aware regularizer in L20. Loss plots are credits to et al. (2018).
Figure 1:

---

Under review as a conference paper at ICLR 2022
Although L20 methods are able to achieve better performance than analytical optimizers in many traditional optimization tasks, they are yet mature to serve as practical optimizers for deep neural networks. The main reason is that all existing L20 methods are designed for solving traditional optimization problems where the generalization ability is not a concern. Generalization ability, one of the core problems in machine learning, is neither guaranteed for deep neural networks optimized by L20, nor studied by literature in the context of L20 before. It has been shown that for today's 's heavily overparameterized networks, is easy memorize the entire training data which lead to zero training loss (Zhang et al., 2021). Therefore, for a L20 method, fast minimization of the training error cannot really lead to generalization ability of the optimizee.
The generalization ability of machine learning methods has been extensively studied both theoret- ically and empirically in the past decades (Keskar et al., 2017: Hochreiter & Schmidhuber, 1997: 1994; Jiang et al., 2020; Chaudhari et al., 2017: Damian et al., 2021). Among them, prevailing the- ory that links the generalization ability of models to the geometry of the loss landscape has recently shown its empirical effectiveness. Most methods taking this idea measure the geometry of the loss landscape using Hessian spectrum, while some others use the local entropy (Damian et al., 2021; Chaudhari et al., 2017). Further, Foret et al. (2021) proposes the SAM method, which minimizes the loss value and the loss sharpness simultaneously. Specifically, an optimization method that prefers to converge wide valleys (i.e. flat basin) in loss landscape shows better generalization ability.
Inspired by this, we propose flatness-aware regularizers i.e., Hessian regularizer and Entropy reg- ularizer for learning to optimize. The Hessian regularization is the pseudonorm of the optimizee's Hessian matrix and the Entropy regularization measures the local Entropy of the optimizee. Both of them characterize the geometry of the loss landscape which intends teach optimizer favor update rules that lead to wide valleys in loss landscape. To summarize, the contributions of this paper can be outlined below:
We propose to use the flatness-aware regularizers in the training of L20 optimizers. The update rules learned with such regularizers would favor converging to wide valleys in loss landscape when minimizing loss functions. Thus, L20 optimizers trained with flamess- aware regularizers can be deemed as a plug-and-plas optimizer that favors generalization and requires more time calculating Hessian or Entropy information while in use.
We demonstrate theoretically that adopting flatness- awwre regularizers in 1,20 can enhance the generalization ability of optimizees trained by regularized optimizers. Note that the the- oretical result of Entropy regularizer implies Entropy-SCr SGD favors wide valleys in original loss landscape rather than only in Entropy energy landscape.
Comprehensive experiments over various tasks demonstrate the effectiveness of our meth- ods. Empirical results show that our methods significantly improve the generalization abil- ity of existing L20 methods, enabling them to outperform current state-of-the-ae by a large margin.
RELATED WORK
Learning to Optimize (L20) As a special case of learning to learn, L20 has been widely inves- tigated in various machine learning problems, including black-box optimization (Chen et al., 2017), Bayesian swarm optimization (Cao et al., 2019), minmax optimization (Shen al., 2021). domain adaptation (Li et al., 2020: Chen et al., 2020b), adversarial training (Jiang et al., 2018: Xiong & Hsieh, 2020), graph learning (You et al., 2020), and noisy label training (Chen et al., 2020c). The first 1.20 framework dates back to Andrychowicz et al. (2016b), in which the gradients and update rules of optimizee are formulated as the input features and outputs for a RNN optimizer. respectively. It proposes a coordinate- wise design which enables trained L20 to be applicable for different neural networks with diverse amount of parameters. Li & Malik (2016) proposes an alternative reinforce- ment learning frameworks for L220, leveraging gradient history and objective values as observations and step vectors as actions. Later on, more advanced variants arise to power up the generaliza- tion ability of L20. For example, (i) regularizers like random scaling, objective convexifying (Lv et al., 2017), and Jacobian regularization (Li et al., 2020), (ii) enhanced L20 model like hierarchical RNN architecture (Wichrowska et al., 2017), and (iii) improved training techniques like curriculum learning and imitation learning (Chen et al., 2020a), A more comprehensive literature is referred to a recent survey paper of learning to optimize.
2

---

Under review as a conference paper at ICLR 2022
Flatness Generalization of Neural Network Generalization analysis of neural networks has been widely studied by various methods, including VC-dimension (Bartlett et al., 2019), covering number (Bartlett et al., 2017), stability (Hardt al., 2016; Zhou et al., 2018a), Rademacher com- plexity (Golowich et al., 2018; Ji & Liang, 2018; Ji et al.. 2021; Arora et al., 2018; 2019), etc. In particular, the landscape flatnesy has been known be associated with better generalization. On the empirical side, Keskar et al. (2017) and He et al. (2019) showed that minima in wide valleys often generalize better than those in sharp basins. Further, Wilson et al. (2017) and Keskar Socher (2017) showed empirically that SGD favors better generalization solutions than Adam. On the the- ory side, Zhou et al. (2020) showed that SGD is more unstable at sharp minima than Adam and explained why SGD generalize better than Adam theoretically and Zou et al. (2021) explained that the inferior generalization performance of Adam connected to nonconvex loss landscape. To improve the generalization performance, Entropy-SGD was introduced in Chaudhari et al. (2017) which was shown to outperform SGD in terms of the generalization error and the training time. Meanwhile, the spectral norm regularization has been proposed in Yoshida & Miyato (2017) to improve the generalization ability of neural networks empirically.
METHODOLOGY
In this section, we provide basic notations and the detailed formulations about our flatness-aware regularizers, i.e., Hessian and Entropy regularizers.
3.1 PRELIMINARY
We define (0; <) and (()) () respectively as the non-negative meta-traning and meta-testing functions, where 0 E IP is the optimizee parameter, and & and ( respectively denote training and testing data samples. Suppose there are N training data samples & E {E:, (1...., N)} and M testing data samples Ç E {Gy.j = (1.....M) Then we define the empirical meta- vettattaining and meta-testing functions and their corresponding population risk functions follows:
(Mata-Training) (0)
1
Σ
(O;G),
I.) = Eg (O,E))
N
M
(Meta-Tee Testing) (
1
(0)),
()) = (2)
M
j=1
An L20 algorithm aims to learn an update rule for optimizee based on the meta-training function.
An update rule can be expressed as ((1) (0) (() + m=(2(");)) where = 0, 1.....,,7 T
1 denotes the iteration index over one epoch, the variable = captures the information (e.g.. loss
values, gradients) that we collect on the optimization path, and the optimizer function m o) is parameterized by & and captures how the update of the optimizee parameter depends on the loss landscape information included in z. In order to find desirable optimizer parameter Φ, L20 solves the following meta- meta-training training problem:
min ( (o (+)) where
-f) + m(=("; ())
A popular L20 meta-training algorithm applies the gradient descent method, which updates & based on the gradient of the objective function T (o (o()())) each update of d requires T iterations of the optimizee parameter 08² (() to obtain ( (Ø).
(Φ)) with respect to Φ. As suggested by eq. (3),
In meta-testing, we apply the output & of meta-training and its corresponding optimizer to update the optimizee as o(2) (Ø) = of2) (Ø) + m(z(3), Φ) = 0, 1...., T - 1). Note that we differentiate the optimizee updates training and testing by superscripts (1) and (2), respectively.
3.2 HESSIAN REGULARIZER
Motivated by the idea that the optimizee in a flat area of the training objective has superior general- ization capability, we propose to incorporate flatness-awar regularizers into L20 meta-nraining, in order to learn optimizers that favors to land the optimizee into a flat region. We introduce two such regularizers in this and next subsection.
3

---

Under review as a conference paper at ICLR 2022
The first regularizer we introduce is based the spectral norm of the Hessian, smaller values of which corresponds to flatter landscape. Thus, the new L20 meta- meta-training objective is given by:
min(L (0+) (o)) + AIV
! (0f (+)) where (0)
+ m( ;
(4)
where À is the regularizer hyperparameter. Note that the Hessian regularizer is adopted for training the optimizer parameter Φ. and its impact on the update rule m(= ()) +) is only through ø, i.e., the information in 2, does not include such regularization. We let o* be the optimal optimizer parameter, which can be written as
(0f 6 = arg min JN (0) (6)) + V (OF) (0))
Due to the computational intractability of directly penalizing V2L (0) (o)), we investigate three approximation variants in the implementation. ① Hessian EV: the eigenvalue of largest module of Hessian matrix, computed by power iteration (Yao al., 2020); Hessian Trace: the trace of Hes- sian matrix, calculated via Hutchinson method (Yao et al., 2020); 3 Jacobian Trace: the trace of
Hessian's 's Jacobian approximation VOLN (2) (0)) V ( (0 (o)). Note that such Hessian ap- proximation methods do not involve computing Hessian explicitly which helps to reduce the memory and computational cost. In our case, we perform 10 iterations for Hessian norms' approximation.
3.3 ENTROPY REGULARIZER
The second flutesssuts regularizer we incorporate to L20 is based on the local entropy function (0) (0) 3110 0'11²) d0' proposed in Chaudhari al. (2017). Due the exponential decay with respect to ||0 0'112, the integral mainly captures the value of the loss function _
G (0:7) log Jor exp (-L
(1) (0') over the neighborhood of 0. Thus, the value of G (0; 7) measures the flatness of the local area around 0. Thus, the L20 meta training objective with Entropy regularizer given by:
m (1) (b)) a
(Φ)) - x (of) (0):))} where
ef ( m(2 m
(6)
We let oi be the optimal optimizer parameter, which can be written
Φ" = arg min L (o) (0) AG
In order implement the gradient descent algorithm for meta-training. the gradient GN (0 (02) ()): can be calculated by the entropy gradient G N' (0:7) and the chain rule.
In partieular, as given in Chaudhari et al. (2017), the entropy gradient takes the following form
-V (0:7) 7(0 g))
(8)
where E {Ei, = (1,..., N)} are training samples and the distribution of 0' is given by P(O';0,7) x exp - L
(W (0') 1 - 0'1
4 THEORETICAL ANALYSIS
In this section, we first introduce several assumptions, and then present the generalization analysis of L20 with Hessian and Entropy regularizers.
4.1 ASSUMPTIONS
We first define the local basin of 0 with the radius as (² = (0' 110- S d). As have been observed widely in training variety of machine learning objectives, the convergent point enters into a local neighborhood where the strong convexity (or similar properties such as gradient dominance condition, reguarity condition, etc) holds (Du et al., 2019; Li & Yuan, 2017; Zhou et al., 2018b; Safran & Shamir, 2016; Milne, 2019). We thus make the following assumption on the geometry of the meta-training function.
4

---

Under review as a conference paper at ICLR 2022
Assumption 1. We assume that there exist a a local basin (A (o))d 0) of the conver- gence point 04 (6) that in such local basin, (0) and LN (f) are u-strongly convex w.Et. 8. Futhermore, there exist unique optimal point (e(1) of function I. (0) and a optimal point (1) of function L (0) in local basin ( ))
Assumption 2. We assume that L (0) function is M M-Lipschitz: (O) and ()) are L-Lipschitz; (O), V2) VOL (0) and ViL (0) are p-Lipschitz.
We further adopt the following assumptions introduced in Mei al. (2018), in order guarantee the similarity between the landscape of the empirical and population objective functions.
The gradient of the training loss V (F)) E) is (--sub-Gaussia and the Hessian of the loss function is 72 --su-eeppooentill. See Appendix B. for more details.
Assumption 3.
Assumption 4. There exist a basin Dr D² (0) that the meta- veeta-training functions (0) is (e, -ttronylyy
Morse in DJ (0) and the local basins Dd (8))) ) (o")) for = 1,2 of convergence points 8 (0'((- 1, are in D" (0). See Appendix B.I for more details.
4.2 GENERALIZATION ANALYSIS HESSIAN-REGULARIZEN L20
In this section, we adopt the optimizer learned by regularized L,20 to train a new optimizee, and analyze the advantages of the Hessian regularizer on the optimizee generalization ability.
We first note that by optimization theory, the regularized optimization problem eq. (5) is equivalent to the following constrained optimization
min ( N (0 (o)) where
0 = =0(²) + +m(=( :
subject 1I (1) (0f) (o)) BHession (A),
where Beeessinn (A) is the constraint bound on the Hessian determined by A. Thus, the opti- mizer parameter 6* learned by the Hessian-regularizeed L20 meta- meta-training in eq. (5) is also solution to eq. (9), i.e., its Hessian satisfy the constraint. Then let of denote the opti- mizee parameters trained by optimizer 6* in meta-testing, and 8-(2) denote the optimal point of the population meta-testing function (-). We then characterize the generalization error as (0(2) (Ø²)) (0)12),, which capture how well the optimizer d² performs on a testing task with respect to the best possible testing loss value.
The following theorem characterizes the generalization performance of the optimizee trained with Hessian regularized optimizer as defined above.
Theorem 1 (Generalization Error of hold. We let N Z max log N/ni,, Cp Cplogr} where C' = Co maxx(ca, lo } n).
Regularized L20). Suppose Assumptions 1, 2, 3, 4
2 n² 1/" minf 721 74: p=r2 } and Co is an universal constant. Then, with probability at least 1 - 25 we
have
1 I (0F (0') - (8)(2)) s A2
(10)
where A -Btession (A) + pAz pA + A + -ww=-=2 T-
-T') + +0 with w = L- Dn = (0 - ViLL A (x)
log N N
A = A + A +
O(w--T'+ + O Clog N [10-11) 11(2)11, A (0) ( and T' is the minimum gradient descent iterations
N
Tata'
for o(. (GD) to enter into the local basin of ol. (
In Theorem I the generalization error is bounded by the terms A, and A2. where the Hessian regular- izer affects the generalization error through the constraint BHessian ()) in Al. Clearly, by choosing the regularization hyperparameter X, we control the value of Biessian (A), which then makes an impact on the generalization. Specifically. larger À corresponds to smaller Bressian (A) and hence smaller generalization error. This also explains that flatter landscape (i.c., smaller Bitessia (A) on Hessiam) yields better generalization performance (i.e., smaller generalization error).
5

---

Under review as a conference paper at ICLR 2022
The generalization error in Theorem 1 also contains other terms which we explain follows: (a) pAi + مهم + AA capture the similarity between the training and testing tasks; more similar tasks yields better generalization; (b) O(wT-T) captures the exponential decay rate of the optimizee's
C' log N arises due to iteration due to the strong convexity, and vanishes for large T; and (c) O(V N
the difference between the empirical and population loss functions, and vanishes as the sample size N gets large.
4.3 GENERALIZATION ANALYSIS OF ENTROPY-REGULARIZA L20
In this section, we analyze the generalization error of the Entropy regularizer on the optimizee generalization ability.
Similarly to the Hessian regularizer, the regularized optimization problem eq. (6) is equivalent to the following constrained optimization:
minL (1)
(09 (o)) where
044 0 mm2))) m=(())))
subject to - ( (0f (0): S B Bentrop (A),
(11)
where BEatrop (A) is the constraint bound on the Entropy determined by X. Thus, the optimizer d learned by the Entropy-regularize L20 meta-training in eq. (6) is also a solution to eq. (11), i.c. the local entropy satisfies the constraint.
In the following, we first establish an important connection between the Entropy constraint bound Bearopy (A) and the Hessian || 1 (0 (+))] of the optimizee.
Theorem (Connection between Hessian and Entropy Regularizer), Suppose Assumptions and
hold. We define C(y, p.pmm. m) log [gr: -1l--m exp(-311 I (1()))d'' where is constant and 0 E T.P Then, we have
L (0)II S D (B (A))
where D(z) = (1) (f) (0) (p - 1) log(y + p1) mM - 2 log(2x) - 1 pm - C(r,p, m) +
log(æ +7) and (x) denotes the inverse function of D(x).
can be observed that the function D(x) in monotonically increasing w.r.t. x and is its inverse (=), as determined by the only z-dependent term log(æ + 7). Hence, Theorem shows that the entropy bound constraint Beatrops (A) implies corresponding Hessian constraint 1 D (Beanopy(A) (A)). Thus, Theorem 1 can be applied to provide a bound on the generalization error here.
Corollary (Generalization Error of Entropy-Rese Regularized L.20). Suppose the same conditions of Theorem hold. Then the generalization error of 1.20 with Entropy regularizer takes the bound in eq. (10) with Beessiau (X)
()) in A₁ being replaced by D 1-1 (Beatopy(X))
Theorem 2 and Corollary establish that the bound D (BEntoog (A)) serves the same role as the Hessian bound in the generalization performance. Thus, by controlling the hyperparameter À to be large enough the L.20 training. B (A) as well as D- (BEnoooy and Hessian can be controlled to be sufficiently small. In this way, the optimizee will be landed into flat basin (due to small Hessian) to enjoy better generalization.
To compare with the result in Chaudhari et al. (2017), note that Chaudhari et al. (2017) proposed the Entropy-SGD method and showed that Entropy-SGD favors better generalization solutions in terms of the Entropy energy landscape. As comparison, Theorem 2 and Corol- lary establish the generalization eтror for Entropy-SGD in terms of the un- un-regularized loss (0 (Φ*)) (0++2)) in original loss landscape, which is the ultimate goal of generaliza- tion. Such a favorable result is established by exploiting the equivalence between the regularized optimization and the un-regularized constrained optimization problems.
We further note that Dinh et al. (2017) theoretically shows that sharp minimas can also generalize well for deep neural networks. However, such a result does not contradict the fact that flat minima generalizes well, which has strong evidence (He et al., 2019; Keskar et al., 2017), and is the property that we exploit in the paper.

---

Under review as a conference paper at ICLR 2022
Conv-CIFAR
MLP-ReLU
Conv-l
24
ADAM
120-Scate
2.5
Traninn Loss
120-Seste Hescan Tirre
1.5
1.-
1.0
ext
aboo
200
10000
2209
106
3900
12
Training
Training
Training
Traininn
1.0
Tegging Accurary
ADIN
120-Scale
Training Iterations
Training Iterations
Training Iterations
Training Iterations
Figure 2: Comparison of the training loss/testing accuracy of optimizees trained using analytical optimizers L20-Scale (Wichrowska et al., 2017) with/without the proposed Hessian regularization.
CORV-NNIST
MNIST
3.
25
120.0MM-CL
25
Training Loss
120-04ec
120kp/eci derropsi
35
200
ext
1000
220
<0c
200
220
400
1203
32.5
1000
Training
Training Iterations
Training
Training Beeations
3.0
Tecting Accurary
6.0
14
0.0
5G0
ADAM
L20 O1ACL
L20 01A Mossan Trece
120 O1 Enougy
<it Training Iterations
200 1000
2009
Training
400
603 300
1000
Iterations
12.0
1000
Rerations
Figure 3: Comparison of the training loss/testing accuracy of optimizees trained using analytical optimizers and L20-DM-CL (Chen et al., 2020a) with/without the proposed Hessian regularization.
5 EXPERIMENT
In experiments, we consider two advanced L20 algorithms, i.c., L20-DM-CL' (Chen et al., 2020a) and L20-Scale (Wichrowska et al., 2017) on four diverse meta testing optimizees.
Meta Training Optimizees. For training L20-Scale, we use a three-layer convolutional neural network (CNN) which has one fully-comnected layer, and two convolutional layers with eight 3 x 3 and 5 x kernels respectively. For training 1.20DDM, we adopt the same meta training optimizee from Andrychowiczen et al. (201 6b), which is a simple Multi-Layer Perceptron (MLP) with one hidden layer of 20 dimensions and the sigmoid activation function. MNIST (LeCun et al., 1998) dataset used for all the meta-training.
Meta Testing Optimizees. We select four distinct and representative meta testing optimizees from Andrychowicz al. (2016b) and Chen et al. (2020a) to evaluate the generalization ability of the
It is an enhanced version of the earliest L20-DM DM introduced by DeepMind Andrychowicz et al. (2016b). We choose much stronger baseline instead of the vanilla 1.20-DM with "poor-generalization base- line" (Wichrowska et al., 2017; I.v al., 2017).

---

corresponds to meta testing (2) and test accuracy corresponds to I,(2) in Section 3.1. We conduct ten independent replicates with different random seeds and report the average performance. All of Under review as a conference paper at ICLR 2022
learned optimizer. Specifically, ① MLP-ReLU: single layer MLP with 20 neurons and the ReLU activation function on MNIST. @ Conv-MNIST: a CNN has one fully-connected layer, and two convolutional layers with 16 3 x 3 and 32 5 x 5 kernels a large CNN has one fully-connected layer, and four convolutional layers with two 32 3 x 3 and two 32 5 x 5 kernels on MNIST. @ Conv-CIFAM AR: CNN has one fully-connected layer, and two convolutional layers with 163 3 x 3 and 32 5 x 5 kemels on CIFAR-I0 10 (Krizhevsky & Hinton, 2009). Optimizees @ @,, and @ are for evaluating the generalization of L20 across network architectures. Then, @ evaluates the generalization of L20 across both network architectures and datasets.
MNIST. @ Conv-Large-Ma SM:
Training and Evaluation details. During the meta training stage of L20, L20-Scale is trained with 5 epochs, where the number of each epoch's iteration is drawing from a heavy tailed distri- bution (Wichrowska et al., 2017). L-0-DM-CL is trained with a curriculum schedule of training epochs and iterations, following the default setup in Chen al. (2020a). RMSprop with the learn- ing rate 1 x 10-6 is used to update 1.20s. For the (Hessian, Entropy) regularization coefficients (AHessian AEntropy: 7} we perform a grid search and choose (5 10 6---//111 10 1,1 10 $.1 x 10-4} for L200-Sell/1220-DM-CLL
In the meta testing stage of L20,, we compare our methods with classical optimizers like SGD and Adam, and state-of-the-art (SOTA) L20s such as L.20-ccale and L20-DM-CI.. Hyperparameters of both classical optimizers and 1.20 baselines are carefully tuned through the grid search and all other irrelevant variables are strictly controlled for fair comparison. We run 10, 000 iterations for the meta testing, and the corresponding training loss and test accuracy on all unseen optimizees are collected to evaluate the optimizer and optimizee generalization. Note that the training loss
our experiments are conducted on computing facility of NVIDIA GeForce GTX 1080Ti GPUs.
5.1 IMPROVE GENERALIZATION WITH HESSIAN REGULARIZATION
In this section, we conduct extensive evaluations of our proposed Hessian regularization on previous sfate-of-the-ar L20 methods. i.c., L20-Scale (Wichrowska al.. 2017) and L20-DM-CL (Chen et al.. 2020a). Achieved training loss and testing accuracy are collected in Figure 2 and 3 which also include comparisons with representative analytical optimizers like SGD (Ruder, 2016) and Adam (Kingma& & Ba, 2014). Several consistent observations can be drawn from our results: O Hes- sian Trace regularizer consistently enhances the generalization abilities of learned L2Os and trained optimizees. Specifically, L20s with Hessian Trace enable fast training loss decay and much lower final loss on all four unseen meta-testing optimizees, demonstrating the improved optimizer gen- eralization ability. Furthermore, all unseen optimizees trained by Hessian regularized L2Os enjoy substantial testing accuracy which boosts up to 31%, showing the enhanced optimizee generalization ability. Such impressive performance gains effective evidence of our proposal, which again suggests that Hessian regularization potentially leads to well-generalizablee minimas in wide valleys in loss landscape. Adopting vanilla L20-DM-CL to train meta-testing optimizees (e.g.. CONV-MNIST and Conv Conv-CIFAR) suffers from instability as shown in Figure 3, and it can be significantly mit- igated by introducing our flatness-awe aware regularization. Conv-Large-MNII ST is an exception, where the L20-DM-CL DM-CL fails to train this optimizee and ends up with random guessed accuracy, i.e., 10% Although plugging Hessian Trace into L20-DM-CL greatly improves its test accuracy from 10% to 95%++ it still undergoes a unsatisfactory training loss. Potential reasons may lie in the rough model architecture and limited input features of 1.20-DM-II, coincided with the findings in Chen et al. (2020a). We will further investigate this interesting phenomenon in the future. 2 For advanced L20-Scale, Hcssian Trace regularization facilitates it to converge a significantly lower minima and obtain considerable accuracy improvements. It enlarges the advantages of L20 methods compared to analytical optimizers, SGD and Adam, unleashing the power of parameterized optimizers.
5.2 IMPROVE GENERALIZATION WITH ENTROPY REGULARIZATION
We investigate the generalization ability improvements from the Entropy regularization. Generally, it boosts both optimizee and optimizer generalizations of L20 in most cases, shown in Figure 3. Hessian v.s. Entropy Regularization. We compare our two kinds of flatness-aware regularizers from both computational cost and performance benefits perspectives. local entropy's 's gradient in eq. (8), it involves gradients from multiple unroll steps for the estima-
In order to calculate the
8

---

Under review as a conference paper at ICLR 2022
Conv-MNIST
CoR--IFARR
MIP-ReLU
6.9
015
Tecting Accuracy
0.3
030
L2C-ON-LL
029
L20.01 LIC-ON-CL: Nossss tV
L:0-0N-CL: Jacciaan Tracs
L20.0NCLL: Mosian Tracs
630
Iterations
226
620
Iterations
IED 1990
X53 Training
4000 6600
10
0.
Testing Accurary
0.7
04
120-Srale
Jassann
Scale jacch Trcce
03
Training Itarations
Training Iarations
Training Itarations
Training Iterations
Figure 4: Comparison of the testing accuracy of optimizees trained using analytical optimizers and SOTA L20 with/without different Hessian regularization, Hessian EV, Hessian Trace, and Jacobian Trace.
tion (Chaudhari al., 2017). leading extra memory and computing outlays. Compared to Hessian augmented L20, it costs ~ 2.6x memory and ~ 3x running time for L20-DM-CL experiments". @ As for generalization gains, Entropy regularizer performs slightly better COnV-MNIST and Conv-CIFAR, while behaves marginally worse on MLP-ReLL and Conv-Large-MNIIS com- pared Hessian regularizer. We would like draw reader's attention Lonv-Large-MN which Entropy regularized 1.00-DMCLL capable of decaying the training loss and finding a much lower minima than Adam. Note that on this optimizee, both L20-DM-CL and its Hessian variant can not decrease the training loss. The possible reason is that multi-layer convolutional neural net- work without BN cannot be stably trained MNIST. However, our LLO--MMCC+Entropy is more stable in training and improves testing accuracy compared with L20-DDMCL. This indicates that L20-DM-CL Entropy may also produce more trainable loss surface for optimizees.
in
Based on the above experiments as well as the experiment on ResNet20 in Appendix A.1, we ob- serve that 1.20+Entropy is preferred when we adopt 1.20 to train large neural networks, where 1.20+Entropy yields better optimizers and optimizee generalization abilities. On the other hand, L20+Hessain optimizer requires less time per iteration to train and achieves lower training loss as well as higher test accuracy than L20+Entropy in small networks, e.g. MLP. The possible reason is that Entropy takes account of the landscape over a large range of loss to measure the flatness, and can hence capture complex landscape information in large neural networks. On the other hand, Hessian regularizer captures the flatness information only for the individual point, but in a more accurate manner, and thus is suitable smaller neural networks with relative simple landscape.
5.3 ABLATION AND VISUALIZATION
In this section, carefully examine the effect of Hessian regularization's different approximation variants, including Hessian EV, Hessian Trace, and Jacobian Trace. Results are presented in Fig- 4. We find that Hessian Trace regularizer achieves the most stable and substantial performance boosts across all optimizees. Jacobian Trace performs the worst which is within expectation since provides the roughest estimation of Hessian.
CONCLUSION
In this paper, we propose several flatness-awar regularizers to improve both optimizer and op- timizee generalization abilities of current tfthe-tf L20 approaches. Such regularizers capable of shaping the local geometry of optimizee's loss surface, and leading to well-generalizable minimas in wide valleys which have been proved theoretically. Our empirical results validate the ef- fectiveness of our proposal, taking a further step for L20's practically usage in real-world scenarios.
We conduct entropy-relate experiments light 120-DM-CI. rather heavy 1.20-Scale models, since RTX TTTAN with 24G memory the largest GPU access and afford.

---

Under review as a conference paper at ICLR 2022
7 REPRODUCIBILITY CHECKLIST
To ensure reproducibility, we use the Machine Learning Reproducibility Checklist v2.0, Apr. 2020 (Pineau et al., 2021). An earlier version of this checklist (v1.2) was used for NeurIPS 2019 (Pineau et al., 2021).
For all models and algorithms presented,
clear description of the mathematical settings, algorithm, and/or model. We clearly describe all of the settings, formulations, and algorithms in Section 3. A clear explanation of any assumptions. All assumptions are stated in Section 4.1 and details are clearly explained in Appendix B.1.
An analysis of the complexity (time, space, sample size) of any algorithm. We do not make the analysis.
For any theoretical claim,
A clear statement of the claim. A clear statement of theoretical claims are made in Section 4.2 and Section 4.3.
A complete proof of the claim. The complete proofs of all claims available in Appendix B and Appendix C.
For all datasets used, check if you include:
The relevant statistics, such as number of examples. We use widely adopted datasets MNIST and CIFAR-10 in Section 5. The related statistics can be and https: h/wp://ww......
seen http:: lecun. com/ htt///xxdmmmmmm//e
toronto. wwt./d/ii//iiaaaa.e.eeeaeaaa html.
The details of train/validationntess splits. We give this information in our repository
in the supplementary material.
An explanation of any data that were excluded, and all pre-processing step. We did not exclude any data or perform any pre-processing.
-
A link to downloadable version of the dataset or simulation environment. Our repository contains all instructions to download and run experiments on the datasets. For new data collected,a complete description of the data collection process, such as instructions to annotators and methods for quality control. We do not collect or release new datasets.
For all shared code related this work, check you include:
Specification of dependencies. We give installation instructions in the README of our repository.
Training code. The training code is available in our repository.
Evaluation code. The evaluation code is available in our repository. (Pree)trained model(s). We do not release any pre-trained models.
README file includes table of results accompanied by precise command to run to produce those results. We include README with detailed instructions to repro- duce all of our experimental results.
For all reported experimental results, check if you include:
The range of hy per-parmmeterss considered, method to select the best hyper- parameter configuration, and specification all hyper-parameters used to gen- erate results. We provide all details of the typer-paramest tuning in Section 5.
The exact number of training and evaluation runs. The details about training and evaluation can be seen Section 5.
clear definition of the specific measure or statistics used to report results. We use the classification accuracy test-set and the loss the train-set. A description of results with central tendency (e.g. mean) & variation (e.g. error bars), We do not report the mean and standard deviation for experiments.
The average runtime for each result, or estimated energy cost. We do not report the running time or energy cost.
- A description of the computing infrastructure used. All detailed descriptions are presented in Section 5.
10

---

Under review as a conference paper at ICLR 2022
REFERENCES
Marcin Andrychowicz, Misha Denil, Sergio Gómez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando de Freitas. Learning learn by gradient descent by gradient descent. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems (NeurlPS), volume 29. Curran Associates, Inc., 201
Marein Andrychowicz, Misha Denil, Sergio Gomez, Matthew Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. In Advances in neural information processing systems (NeurIPS), 2016b.
Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via compression approach. In International Conference on Machine Learning (ICML), pp. 254-263, 2018.
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of op- timization and generalization for overparameterized two-layer neural networks. In International Conference on Machine Learning (ICML), pp. 322-332, 2019.
Peter L Bartlett, Dylan J Foster, and Matus Telgarsky. Spectrally-normaiized margin bounds for neural networks. In Advances in Neural Information Processing Systems (NeurIPS), volume 30, pp. 6240-6244 2017.
Peter L Bartlett, Nick Harvey, Christopher Liaw, and Abbas Mehrabian. Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks. The Journal of Machine Learning Research (JMLR),
2301, 2019.
Yue Cao, Tianlong Chen, Zhangyang Wang. and Yang Shen. Learning to optimize in swarms. In Advances in Neural Information Processing Systems (NeurIPS), pp.
2019.
Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-SGID: Biasing gradi- ent descent into wide valleys. In International Conference on Learning Representations (ICLR), 2017.
Tianlong Chen, Weiyi Zhang, Jingyang Zhou, Shiyu Chang. Sijia Liu, Lisa Amini, and Zhangyang Wang. Training stronger baselines for learning to optimize. arXiv preprint arXiv:2010. 09089, 2020a.
Wuyang Chen, Zhiding Yu, Zhangyang Wang, and Animashree Anandkumar. Automated synthetic- to-real generalization. In International Conference Machine Learning (ICML), pp. 1746-1756, 2020b.
Xuxi Chen, Wuyang Chen, Tianlong Chen, Ye Yuan, Chen Gong, Kewei Chen, and Zhangyang Wang. Self-pu: Self boosted and calibrated positive-umlabeled training. In International Confer- ence on Machine Learning (ICML), pp. 1510-1519, 2020c.
Yutian Chen, Matthew W Hoffman, Sergio Gómez Colmenarejo, Misha Denil, Timothy P Lillicrap, Matt Botvinick, and Nando De Freitas. Learning to learn without gradient descent by gradient descent. In International Conference on Machine Learning (ICML), pp. 748-756, 2017.
Alex Damian, Tengyu Ma, and Jason D. Lee. Label noise SGD provably prefers flat global mini- mizers. CoRR, abs/2106.006500. 2021.
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In International Conference on Machine Learning (ICML), pp. 1019-1028, 2017.
Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. International Conference on Learning Representations (ICLR), 2019.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research (JMLR), 12(61):1111-2159 2011.
11

---

Under review as a conference paper at ICLR 2022
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness- aware minimiza- tion for efficiently improving generalization. International Conference on Learning Represen- tations (ICLR), 2021.
Noah Golowich, Alexander Rakhlin, and Ohad Shamir. Size-independent sample complexity of neural networks. In Conference On Learning Theory (COLT), pp. 297-299, 2018.
Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability stochastic gradient descent. In International Conference on Machine Learning (ICML), pp. 1225-1234, 2016.
Haowei He, Gao Huang, and Yang Yuan. Asymmetric valleys: Beyond sharp and flat local minima. In Advances in Neural Information Processing Systems (NeurIPS), volume 32, pp. 2553-2564, 2019.
Sepp Hochreiter and Jürgen Schmidhuber. Simplifying neural nets by discovering flat minima. In Advances neural information processing systems (NeurIPS), pp. 529-536, Cambridge, MA, USA, 1994. MIT Press.
Sepp Hochreiter and Jürgen Schmidhuber. Flat Minima. Neural Computation, 9(1::144, 01 1997.
ISSN 0899-7667. doi: 10. (..1116//cc..997.. 1997.911..
Kaiyi Ji and Yingbin Liang. Minimax estimation neural net distance. In Advances in Neural Information Processing Systems (NeurIPS), pp. 3849-3858, 2018.
Kaiyi Ji, Yi Zhou, and Yingbin Liang. Understanding estimation and generalization error of gener- ative adversarial networks. IEEE Transactions on Information Theory, 07(5):311433129 2021.
Haoming Jiang. Zhehui Chen, Yuyang Shi, Bo Dai, and Tuo Zhao. Learning to defense by learning to attack. arXiv preprint arXiv:1811.01213 2018.
Yiding Jiang. Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantas- tic generalization measures and where to find them. In International Conference on Learning Representations (ICLR), 2020.
Nitish Shirish Keskar and Richard Socher. Improving generalization performance by switching from Adam to SGD. arXiv preprint arXiv:1712.07628 2017.
Nitish Shirish Keskar, Jorge Nocedal, Ping Tak Peter Tang, Dheevatsa Mudigere, and Mikhail Smelyanskiy. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representations (ICLR), 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6998 69800, 2014.
A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Master's thesis, Depariment of Computer Science, University of Toronto, 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):228--2324 1998.
Chaojian Li, Tianlong Chen, Haoran You, Zhangyang Wang, and Yingyan Lin. Halo: Hardware- aware learning to optimize. In European Conference Computer Vision (ECCV), pp. 500-518. Springer, 2020.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss land- scape of neural nets. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Ke Li and Jitendra Malik. Learning optimize. arXiv preprint arXiv: 1r0vv01606.01855 2016.
Yuanzhi Li and Yang Yuan. Convergence analysis of two-layer neural networks with ReLU activa- tion. Advances in Neural Information Processing Systems (NeurIPS), 30, 2017.
Kaifeng Lv, Shunhua Jiang, and Jian Li. Learning gradient descent: Better generalization and longer horizons. In International Conference on Machine Learning (ICMI), pp. 2247-2255, 2017.
12

---

Under review as a conference paper at ICLR 2022
Song Mei, Yu Bai, and Andrea Montanari. The landscape of empirical risk for nonconvex losses.
The Annals of Statistics, 46(6A))2747-27744 2018.
Tristan Milne. Piecewise strong convexity of neural networks. In Advances in Neural Information Processing Systems (NeurIPS), volume 32, pp. 12973-12983, 2019.
Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Lariviere, Alina Beygelzimer, Florence d' Alché Buc, Emily Fox, and Hugo Larochelle. Improving reproducibility in machine learning research, Journal of Machine Learning Research, 22:1-20, 2021.
M. Riedmiller and H. Braun. A direct adaptive method for faster backpropagation learning: the rprop algorithm. IEEE International Conference on Neural Networks, pp. 586-591 vol.1, 1993. doi: 10.1099/CNNN11932229223
Herbert Robbins and Sutton Monro. A stochastic approximation method. The Annals of Mathemat- ical Statistics, 22(3):400-400 1951. ISSN 00034851.
Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv:1609.0423 047477 2016.
arXiv preprint
Itay Safran and Ohad Shamir. On the quality of the initial basin in overspecified neural networks. In International Conference on Machine Learning (ICML), pp. 774-782, 2016.
Jiayi Shen, Xiaohan Chen, Howard Heaton, Tianlong Chen, Jialin Liu, Wotao Yin, and Zhangyang Wang. Learning a minimax optimizer: A pilot study. In International Conference on Learning Representations (ICLR), 2021.
T. Tieleman and G. Hinton. Lecture 6.5 R.m-Rmsr Divide the gradient by a running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 2012.
Olga Wichrowska, Niru Maheswaranathan, Matthew W Hoffman, Sergio Gomez Colmenarejo, Misha Denil, Nando de Freitas, and Jascha Sohl-Dickstein. Learned optimizers that scale and generalize. In International Conference Machine Learning (ICML), 2017.
Ashia C Wilson, Rebecca Roelofs, Mitchell Stem, Nathan Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. In Advances in Neural In- formation Processing Systems (NeurlPS), 2017.
Yuanhao Xiong and Cho-Jui Hsieh. Improved adversarial training via learned optimizer, 2020.
Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W Mahoney. Pyhessian: Neural networks through the lens the Hessian. In 2020 IEEE International Conference on Big Data (Big Data), pp. 581-590. IEEE, 2020.
Yuichi Yoshida and Takeru Miyato. Spectral norm regularization for improving the generalizability of deep learning. arXiv preprint 1rXiv:170S.10944 2017.
Yuning You, Tianlong Chen, Zhangyang Wang, and Yang Shen. L2-gen: Layer-wise and learned efficient training of graph convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2127-2135, 2020.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Commun. ACM, 64(3):007-115, February 2021. ISSN 0001-0782. doi: 10. 145/3446776.
Pan Zhou, Jiashi Feng. Chao Ma, Caiming Xiong, Steven Chu Hong Hoi, et al. Towards theoretically understanding why SGD generalizes better than Adam in deep learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, 2020.
Yi Zhou, Yingbin Liang, and Huishuai Zhang. Generalization error bounds with probabilistic guar- antee for SGD in nonconvex optimization. arXiv preprint arXiv: /802. 06903, 2018a.
Yi Zhou, Junjie Yang, Huishuai Zhang, Yingbin Liang, and Vahid Tarokh. SGD converges global minimum in deep learning via stav-convex path. In International Conference on Learning Repre- sentations (ICIR), 2018b.
13

---

Under review as a conference paper at ICLR 2022
Difan Zou, Yuan Cao, Yuanzhi Li, and Quanquan Gu. Understanding the generalization of Adam in learning neural networks with proper regularization. arXiv preprint vrXiv:2108.11371, 2021.
14

---

Under review as a conference paper at ICLR 2022
Supplementary Materials
A ADDITIONAL EXPERIMENTAL RESULTS
A.1 RESNET20 EXPERIMENTS
In this section, we evaluate the performance of our trained optimizers on larger neural networks ResNet-20 on CIFAR-10 dataset. The training loss and testing accuracy are plotted in Figure 5. We can see that the Entropy regularizer able to outperform other methods in both training loss and testing accuracy, demonstrating its generalization ability large unseen models. Further note that although the Hessian regularizer may not be preferred in large neural networks, does perform better than the Entropy regularizer in small networks
have shown in Figure 3.
ResNet-2O-CIFAR
0.8
L20-DM-CL
L20-DM-CL + Hessian Trace
2.5
0.7
L20-DM-CL + Entropy
Testing Accuracy
Training Loss
1.5
L20-DM-CL
0.3
120-DM-CL Hessian Trace
L20-DM-CL. Entropy
0.2
0.0
2000
4000
6000
8000
10000
0
2000
4000
6000
8000
10000
Training Iterations
Training Iterations
Figure 5: Comparison the training loss/testing accuracy ResNet-20 trained using L20-DM-C (Chen et al., 2020a) with/without the proposed Hessian/Entropy regularization.
A.2 WALL CLOCK COMPARISON BETWEEN DIFFERENT ALGORITHMS
We further conduct optimizee training time comparison between methods and analytical optimizers, L.20-DM-CL, and Entropy-SGD (Chaudhari et al., 2017) Table 1. Note that 1.20- DM-CL+Hessiann and L.2-DM-CL+Enntee
share the same time train optimizee L20-DDM- CL. From Table 1, we can see that trained L20-DM-CL requires only ~ 1.5x time than analytical optimizers in terms of inference time, which is thus time efficient for practical usage. However, Entropy-SGD requires ~ 21x time than analytical optimizers to train optimizees. Such cost is because Entropy-SGD requires multiple Langevin dynamic steps per iteration estimate the local entropy.
Table 1: Empirical Time Cost Comparison per Iteration
Methods
SGD
ADAM L20rL20+Hessian, L20+Entropy)
Entropy- SGD
Time (secs)
0.045
0.045
0.067
0.958
A.3 ACCURACY COMPARISON BETWEEN DIFFERENT ALGORITHMS
We also compare the testing accuracy (%) of our proposed methods with Entropy-SGD (Chaudhari et al., 2017) and SGD with Hessian regularization. The Conv-MNIST results shown in Table 2 are evaluated on L20-DM-CL and the Conv-CIFAR results shown Table 3 are evaluated on L20- Scale. We adopt the same experimental setting in Section 5 for training except that the running
15

---

Under review as a conference paper at ICLR 2022
epochs are limited to 100 to investigate whether the performance of trained optimizers would persist in long term.
Table Additional Testing Accuracy Comparison on Conv-MNIST
Methods
L20
L20+Hessian
L20+Entropy
SGD Entropy-SGD
SGD+Hessian
Testing Accuracy
92.74
97.34
97.87
80.73
97.54
95.37
Table 3: Additional Testing Accuracy Comparison on Conv-CIFAR
Methods
L20+Hessian Entropy-SGD
SGD SGD+Hessian
Testing Accuracy
59.57
57.73
54.69
51.41
From these comparisons, achieve the best performance compared with regularized analytical optimizers. Specifically, in Conv-CIFAR setting shown in Table our algorithm L20+Hessian outperforms SGD+Hessian and Entropy-SGD. In Conv-MNIST setting as shown in Table 2, the performances of top three al- gorithms, i.e. L20++Enroppy Entropy- and LeO+aan are similar and much better than the performances of L20 and SGD+Hessian. Among the top three algorithms, the iteration running time for Entropy-SGD is 0.958 secs while L.20+Hessian and L.20+Entrppy only take 0.067 secs as shown in Table 1. Such wall clock comparison shows that L20+Hessian and L20+Entropy more time efficient than Entropy-SGD while achieving the high accuracy, which are preferred for practical usage.
that our proposed optimizers (L22On I.20+Entropy)
B PROOF OF THEOREM
B.1 RESTATEMENT OF ASSUMPTIONS
Assumption 5 (Restatement of Assumption 2). Lipschitz properties are assumed on functions ()) and (B).
a) L (0) function M M-Lipschitz, i.e., for any 01 and 02 ([,))((1) ((())())LL((())) (02)l s M| 0211.
b) V. (0) and (2 are L-Lipschitz ie., for any O1 and Oz. (Vo(( L (02) & L 02 1,2).
(0) and ()) p-Lipschitz i.e., for any 01 and 02. (V VOL (02) S p|10 02ll(i = 1,2). This assumption also holds for stochastic VOL (0) and ( (0).
(1)
Assumption 6 (Restatement of Assumption 3). Similarly as in Mei et al. (2018), we assume the loss gradient (0) {) is 7 iee. for any E RR, and 0 E D* where D" (* = {A RP, ||0 r).
(?2llel
Blexp((e, (F()(0:)) EIVF S exp
Meanwhile, we assume the loss Hessian is 7² ---sub-epponntial, i.e., for any E (0), and 0 e (O)
Ea. = (2.VV)V V2 (0:E0)) B{ exp ( Ee.0 BLe,ol } $2
and there exists a constant on such that L S T2th p S P'p
Assumption 7 (Restatement of Assumption 4). We assume functions T,(1 (0) (c, sttronlyyy Morse in D' (0), i.e., if ())]l > for [|0|]2 r and, for any E RP, [10|] < r, the
following holds:
VL(0)]22 S < min (1 2n
EEpp
16

---

Under review as a conference paper at ICLR 2022
where X₂ (O)) denotes the i-th eigenvalue of (8). We further make the assumption
()))))))) that the local basins (= (o))) = 1, of convergence points 04 (i = 2, are in D (0).
B.2 PROOF OF SUPPORTING LEMMAS
Lemma 1 (Restatement of Theorem 1(b) in Mei al. (2018)). We assume 0 corresponding to 0% in local basin. Based on Assumptions 2 and 3, there exists a universal constant Co. and we let
C Co log / 13. 2 Cplog p. then we have
Cplog N
[2) V2L(O)I S S
sup (EDP(r)
with probability at least 1 5. 5.
Lemma (Restatement of Theorem 2 in Mei al. (2018)). Based on Assumptions 2, 3 and we set C as in Lemma 1, assume that 0* is corresponding to 0y and let N 4Cp N/ where
n = ( (L?))1))) Then, for each corresponding N and 0*, we have
2T / 17
Cplog N
10% 1-
with probability at least 1-
Lemma 3. Suppose Assumptions 1 and 2 hold. Then, have
(L-
L
L L- T-T" (GD) - -0 (1)
1102 (0)) -0)) 5 V L+
[
(12)
where T is the minimum that after T" gradient descent updates, the updated optimizee parameter 0. (GD) locates into the local basin of 0 (Φ*).
Proof. Since the local basin is p-strongly convex and (x(1) is the optimal point of smooth function L (0) in local basin. Then, we have
I (1 ())) - L (O
H (10 122
2
2 Furthermore, we rearrange the terms and obtain
10 0 -
(1 (0f (o)
V
(9)
(o (GD)) LN (ON
1
68)
L
10 (GD) 0 1 12
10f (GD) - -
(G2)
1 T-T"
(0("GG)) - 0771111 |]
レ+
where (i) follows because o² arg min,, ( (o(") (Φ)) and 044 (GD) locates in the local basin of 020) (ii) follows from Assumptioin 2 and the fact that 0 arg ming L (0), and (iii) follows if we set step size of GD as L.L..
Lemma 4. Based on Assumptions 1, 2, and we let N 2 max p. 4Cp log N/12} where n² n¹
C = Co max(cr, 1, log( }), n2 min y2n th: p Co is universal constant. Then, with probability least 25 we have
17

---

Under review as a conference paper at ICLR 2022
[V Sp
2T
Cplog N
L
L 1 I
T-T'
10 - 1
1)
P
Cplog N
/
+
A #
+ -2,
Bitessian (N);
where 4y Mel4-722
N
v (1) ( (8) and T"' defined in Lemma 3.
Proof. Firstly, we buund 12 as following:
V ( - (1200)0)000000000000"01"11111110
V2LL IV3L (0)(1)) - - 12 + -
VOL. (0
(f)))
V2LN (07 (o"))l,
where (1(1) is corresponding to N in the same local basin of 0 (Ø*).
Based on the constrained problem formulation eq. (9), the optimal optimizer parameter o² is equivalent to the following:
Φ arg min L (0g) (Ø)) subject to VOL ( (0t)) (o)) s BHessian (A).
Thus, obtain V (0 (+)) Blessia (A). Furthermore, if we let N 2 Cplog where C' = Co m Ch. 1, logg( })} and Co is an universal constant, based on Lemmas 1, 2 and 3, and Assumptions 2, we have
| V Sp
[10-00 ( ON 1
(L--))) V L+
T-T" |10 (GD) - 02
Cplog N
Di 72,
Bitessian ()),
with probability at least
Furthermore, if we assume N > max(4Cplog N/ N/1ž, Cplogp} where n/ min based on Lemma 2, we have
2 x² Fr, FT.
I Sp
2T
Cplog N N
L
L 1
T-T'
|10(|) (GD) ex
V P
L+H
Cplog
+ BHessian (A) + Ai + v2
N
with probability at least 1 25.
Lemma 5. Based Assumptions 2, 3, and we let N 2 4Cplog N/n² where C and n/ are
defined in Lemma 2. Then, with probability at least 1 - 5, we have
( 0-(211 < D V 1
T-T
10 (GD) 020
2T
Cplog N
N + Ai.
T
71
where A = -1(0(11) (2) ||, A = 110 - -001) (b*)l and T" is defined in Lemma 3.
Proof. Based on triangle incquality, we obtain
10f=((") (0(2)
18

---

Under review as a conference paper at ICLR 2022
< - + 1100 (0* 0 + +110% - 00aa0 + 110 - 1
(6*)
( S A + 110 (6*) - NN + 10 11 + -10000) - 1113)|1
(0) A
- 1
T-T
100 (GD) 0N + 110 - 10(0 + A
(+
where (i) follows from definition of A7., (i) follows from Lemma 3 and definition of Ag. Based
on Lemma 2, if we let N 2 4Cplog N/ m/ Then, with probability at least - 5, we have
(0)) 0,02)
L
L 14
T-T
10 (GD)
ON
|| +
2T
Cplog N N
+ A
1
T.+1
"
Cloog N N
= A + Oww + +0(1
where
ي- L.pp'
B.3 PROOF OF THEOREM 1
Generalization loss is defined below:
22 (0)) (0)) (, (0(()))
@ (0 -0(2 (,010000000000000000000700101110 0212)
(
(20 -00
(22 )
(09
V 1 (f))))) - 0
2 100 0 19
+ 11141
(00)
50109 (0') 22 1 2 14
(0) -
マ/ (0) - 11 (0")
(1) + (0-2)))
where (i) follows from Taylor expansion and follows from the conditions that 110' - 000 (o")) S
[|0~(2) - /2 (d")) II and 1|0'- (2))) S 1|0-22) - of (dt)), (ii) follows because (2 (0^(2)) = O. and (mii follows from Assumption 2 and the fact that 100 - (~ || S (10-(2) - -00² (b))).
Based on Lemmas 4 and 5, if we let N 2 max(4Cp N/n Cplogp} where C' and r are defined in Lemma 4. Then, with probability at least 28, we have
210000 (1²) - (1 + (((22)
A L 4-H
H
T-T"
-
+
2T
CCllo N
+ +1b)
1 L+
17
N
Cplog N
'T
Cploo N
T-T"
[100) (GD) - -8%"11)
+p
N
17
N
V F
L
AH + B(A)
T.
T.
T-
[If) - - + (ApT + +2) v
Cp log N
= + 201 p
1
n
N
+ PDj + Aj + BHessian(A) (A)
Clog N N )
= =Biessiam (A) + pai + p + A1 + Ow + O(V
19

---

Under review as a conference paper at ICLR 2022
where L- Ltn Ai (1 VOLL) A =10001 1-(2)) A7
|00 (d)) o7 (d')].
Furthermore, We let A Blesssin (A) + p4 + ودم + A + O(w + +0(v clay log ) A2
A7 + A3 + +ww V
C Clog Ny
Then, we have
1.00 (b))) - ( (0)(2))
- 10,0(((0) (6)) - ((110(()))))0000))))))))))))"))) 10
1
2
1 2 12AA1,
with probability at least 25. Then, the proof is complete.
C PROOF OF THBOREM 2
C.1 PROOF OF SUPPORTING LEMMA
Lemma 6. Based on Assumptions and 2, in terms of entropy regularizer constraint Bentrop (A),
we have
BEmmpy (A) + mM P log(2r) zmm² + C(T.p, 2 log(det ( +g))
(o))
+ (1) (0 (o)).
where is constant, C(y, p, m) log So -01 exp (-31 0 8' do' and 0 R..
Proof.
We firstly split the integral area 6' E RP into two parts: (0 110 - 0| s m) and {0) :
|10' 0|| m]. Based the definition of (0:7), we have
G (0:7)
(IIN (8) 2.2 2110 - 07)3) dø'
= =log
cxp
(- (0') - 2110 0 00113) ao
-log
exp
00: 11-10'00]SSm
(-r.f) (0) _11000 100 01 do
+ log
exp
0':110'-01 2m
(2)
(-11) (0') - 2 10 10- 01 ao
< log
exp
0":] 00-011sm
2
(-2 0112) do
+ loog
exp
0':110' 10000-011 >m
(2z) log
(-12)(0) (0') 2 10 0113) a0' C((.p,)
exp
2
81: - 111m
(15z)
1- (0) - (0' - VLL (0) 1/ -
log
exp
0/ -01lsm
2 10 - 0/112 d0 + C(y,p,m),
where (i) follows from the fact that ( (0') (0') is non-negative, (ii) follows from the fact that 0
RPP and the definiton of C(y, P. m). (ii)) follows from Taylor expansion. Note that H" satisfies
||0" A|| < ||0 0'll and ||@" 0 < ||0 l'1l.
Based on Assumption 2, -(0' - 0) V N (0) & ml VIL ( (0) S m.M Then, we obtain
@ (0:7)
20

---

Under review as a conference paper at ICLR 2022
S - (2) + mM + +log [
- (-0000002220000000
exp (- - 1
00:10'-011sm
21100 112 do' + C(7.p.
2
=-L (0) + mM + log _0 00
( 1 2 - (V (0") + 71) -01 - a0
exp
+ C(7,D,
=- =- + mM + m)
+ log {
- 1 2
10' -0) -)0(((2 - - --" -
exp
-01:10r-011sm
+ 71) 777)(0) (0' - 0) ))000 dH
( LN (0) + mMM + m)) + 1
+22m2 pm²
exp ( (- 1
+ -log
2 (0' -0)" (0"-
(V2L (0) + (0)) (0' - -0)
d0'
-
(8)
S 1 (2) (0) + mM + 2 pmm + log / exp
2 (8' - 0)²² (V'I (0) + +11) (0' 00) do
+C(7,p,
where (i) follows from Assumption 2 and the fact that ||0" - ||0 < ||0 00 and (ii) follows because
exp(-1 - 0 ( (0) + +0 0)) 2 0.
(1) (0 (Ø)) + 7/) is a symmetric and positive- have the fact that ( definite matrix. Hence, we obtain
Based on Assumption
Φ (of (0)
-L (0 (0)) [og(de (1) (o
(6)) + +l) + C(..p.m
+ mM + 2 pm3 + P log(2x).
2
We rearrange the terms and get
(6)) -G (ot)) (0): v) 2 (2) (0 (Ø)) + 1 (o)) +n()) C(r.p.
-mM- L 2 (og)(2)= 2 pm3
1
Since -G (0):7) & BEmtoPy (A). Then, obtain
Banropy (A) + mM + 2 log(2T)) + 5 pm + CC7,p, 2 log(det( V22 (of (0) +T1)
1
+ L (8f (6)).
C.2 PROOF OF THEOREM 2
Based on Lemma 6, we have
(b)) Bentropy (A) + m. + log(2r) + 2 pm + C().p,m) 2 log(det( (2 (0 (0)) +77))
2
+ + (0f) (0)).
Since v (2 + 7I is positive definite and \ (0 2 (b) + 71) 2 7 + p for
any i ,. Then, based the definition of Matrix norm /
(1) (0 (0) 7l|
(f)(()))
21

---

Under review as a conference paper at ICLR 2022
Amax N
(1)) (()) 7/). have
(1 (0) + +*I|l \ 1 (6)) 11 P det ( (0 (0)) + 2 (7 + | N
Note that we use X₂ H) to denote the i-th eigenvalue of matrix H. Then,
log(dct( V N (8 (0)) +>D)) 2(p 1) log(> + + + log | (8) (6)) +
=(p 1) log(7 p) log V 1 (0 (0)) (0) +7).
Then, obtain
BeauoPy (A)+MM + P (2T) + pm + C(y. p. m) LW (89) (6)) + (p 1) log((> + 14) 2
+ (1) (04)) (o)) + 7)
Hence, we can get new function (x)) that
(+)iI V (0 (+)iI S D DB" (A)),
where D(x) = LN (0 (o)) + (p 1) log(7 + 11) mm - & 1og((T)) - pmmp - p) +
log(z + +7). Then, the proof is complete.
22


Reviews:
[{"review_id": "og0G_OvX_D", "reviewer": "Reviewer_6eeZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n1. This work applies the flatness-aware regularizer to the learning to optimize objective to improve the generalization of the learned optimizers, which is a novel application of the existing technique. \n\n2. The authors performed both theoretical and empirical analysis of the hessian and entropy based regularizers, and demonstrated its advantages. The improvement over the baseline (without regularizer) seems quite large. \n\nWeakness:\n\n1. Similar to existing L2O works, the evaluation is only performed on simple CNN and MLP models and CIFAR/MNIST. Given the sizable improvement over the other L2O method, I wonder if this method can be applied to more realistic models such as Wide ResNet or ResNet-50 for ImageNet since the learned optimizer seems to generalize over different architectures and datasets as is discussed in section 5. If not, it would help to add some discussion of the limitations and bottlenecks so that future works can be better informed. \n\n2. Given the large variance, it would be helpful to add some confidence interval to the curves in the figures. Or it helps to add some tables with mean and standard deviation for different methods for comparisons, which can complement the figures. \n\n3. It would be helpful to get more insight of the learned optimizer. One suggestion is to add a comparison with a baseline that applies flatness-aware regularizer to the analytical optimizer, which could help understand where the gain over the analytical optimizer is from, i.e., whether the learned optimizer learns to act like a flatness-aware regularized optimizer or there is more than that. \n\nThe paper claims that the learned optimizer \"favors generalization and requires no more time calculating Hessian or Entropy information while in use\". It would require some evidence to support this claim. For example, it would help to compare the Hessian spectrum or entropy of the minima found by learned optimizer with regularizations with the ones found by the baseline learned optimizer to show that it is indeed finding a flatter minima. ", "summary_of_the_paper": "This work proposes to apply flatness-aware regularizations to learning to optimize methods. Specifically, the approximate hessian spectrum and entropy based regularizers are added to the L2O objective. They performed a theoretical analysis of the regularized L2O method and evaluated different variants of the regularizer on simple CNN and MLP models trained on CIFAR and MNIST. Similar to the effect of flatness-aware regularization on the ordinary training, the regularizer improved the learned optimizers to find minima that generalizes better. \n", "summary_of_the_review": "This paper introduces flatness-aware regularizer to L2O objective to improve the learned optimizers, which is a novel contribution. The empirical result looks promising and it could be improved by adding more insights of the learned optimizer, some discussions of the limitations (or some results applying to a more realistic model) and confidence intervals to the results.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "eBWt4DN1QEs", "reviewer": "Reviewer_wp2u", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nTo the best of my knowledge, this is the first paper to add such regularizers to the L2O objective, and prove a bound in the generalization error. The paper is clear and easy to follow.\n\nWeaknesses:\n- This paper doesn’t show enough evidence of the practicality of the proposed method. The experiments are done on small networks and on datasets (MNIST and CIFAR) that are more or less “solved”. Instead of spending extra resources doing meta-learning to train an optimizer, one could tune an off-the-shelf optimizer with a more expressive model. We do not know whether this idea scales and generalizes to more realistic settings.\n- The performance of the baselines (SGD and Adam) are hard to believe. On MNIST, SGD and Adam barely reach 90% accuracy. On CIFAR, they both reach below 50%. These are with 10k steps. I’m suspicious that the hyperparameters for SGD and Adam are chosen poorly, because given the same architecture, the learned optimizer is able to achieve much better performance.\n- The relationship between “flatness” and the generalization ability of a solution are tenuous-- sharpness measures can be varied arbitrarily without changing the output of the neural network (Dinh et al., 2017). The generalization bounds do take into account flatness, but I’m not sure how tight this bound is, or how realistic the assumptions are.\n\nDinh, Laurent, et al. \"Sharp minima can generalize for deep nets.\" International Conference on Machine Learning. PMLR, 2017.\n\n------\nUpdate:\nI have read all the reviews and the author rebuttals. I am increasing my score (3-> 5) because idea is novel and it improves over L2O methods. However, I'm still not convinced that the paper should be accepted given that it wasn't tested on settings that are able to achieve competitive results on the chosen datasets, and because the experiments section could be polished and more organized.\n", "summary_of_the_paper": "This paper aims to solve the poor generalization performance of Learning to Optimize (L2O) methods by introducing flatness-aware regularizers. The idea is to add a regularization term to the meta-training objective that encourages the final iterate obtained by applying the learned optimizer, to lie in a flat region (measured by the spectral norm of the Hessian, or the local entropy function). The authors prove a bound for the generalization error, and show empirically that adding flatness-aware regularizers to existing L2O methods, can improve generalization or both the learned optimizer (across unseen tasks) and the solution (across dataset splits).", "summary_of_the_review": "Given this paper’s lack of practicality and questionable experiment results, I recommend rejection. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "79-8NKeEb7y", "reviewer": "Reviewer_ZY6B", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Pros**\n\n- The proposed method demonstrates large performance over the baseline L2O approaches.\n- The presentation of the method is clear and the paper is well-written.\n\n**Cons**\n\n- The method seems difficult to scale to larger models, considering the complexity of training the learned optimizer. While this may be ok when compared to other L2O (which are equally inefficient), it makes less sense when there already exists a much better regularizer [1] which not only applies to truly state-of-the-art models but also does not require an expensive meta training phase for the optimizer.\n- In a similar vein, it’s unclear how or can the proposed model generalize to larger and more realistic models. “How” as in whether the optimizer learned on the smaller model will find wide minima for a larger model and “can” as in whether there exists reasonable hardware that allows for this.\n- While the method doesn’t require explicit sharpness computation, there is extra complexity when using the learned optimizer. I would like to see a wall clock comparison between the proposed method and [1,2]\n- In `Conv-Large-MNIST` of figure 5, the training loss is completely gone from the plot while the test accuracy is really good. This doesn’t really feel like an “exception” but rather a bug. I would like to see some analysis of this phenomenon.\n\n**Question / Comments**\n\n- I am confused by the introduction of $L_M$ in eq 2. If I understand correctly, unlike regular meta-learning, the method is actually not optimizing for the validation loss but rather just the geometry on the training data. $L_M$ is not used at all after this introduction.\n- [1] should be cited and discussed.\n- The paper does have an interesting idea which is that you can specify properties of desirable optima at meta training time and learn an optimizer that implicitly looks for solutions with the desired properties; however, I feel like sharpness may not be the most convincing application for this technique due to aforementioned reasons and I believe there could be more compelling use cases.\n\n**Reference**\n\n[1] Sharpness-aware Minimization for Efficiently Improving Generalization. Foret et al.\n\n[2] Entropy-SGD: Biasing Gradient Descent Into Wide Valleys. Chaudhari et al.\n\n------------------------------\n\n**Update**\n\nThank you for the response and additional experiments.\n\nI am now convinced that you can train L2O on larger model but the performance of these models are much worse than standard SGD (i.e., < 80% test accuracy) so I am not sure if the proposed method is significant for real models. The claim about L2O doing better than SGD and Adam is also somewhat questionable since it's only done in the small convnet setting but the performance of this convnet is pretty bad on cifar10 and mnist. In particular, it achieves 80% test accuracy on mnist. In my opinion, this number is unusually bad since one can easily write an MLP that does > 90% accuracy on mnist by just following a tutorial. Is there no momentum? If so, why not?\n\nOverall, the new experiments and rebuttal have not changed my opinion of the paper and therefore I am keeping my current evaluation.", "summary_of_the_paper": "The paper proposes to use meta-learning to learn an optimizer that automatically seeks wide local minima without the need to explicitly compute the sharpness measure on the fly. When training the optimizer, the parameters of the optimizer are updated in such a way that the training loss and the sharpness measure at the end of the updated trajectory are minimized. Theoretical results were provided for the generalization guarantees of the learned optimizer under some technical conditions. Finally, the experimental results demonstrate that the sharpness-aware optimizer outperforms baselines on simple models.", "summary_of_the_review": "While the paper proposes an interesting idea with solid improvement over the baseline, I do not see the practicality of using a learned optimizer that seeks a wide solution when there exist much better alternatives that don't require meta-learning. I believe that a similar idea could be useful for something other than sharpness but in the current form, I don’t feel the paper is good enough for publication.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2Ka_v1SOSBj", "reviewer": "Reviewer_qg5Z", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The generalization issue in L2O has been a significant problem preventing the applications of learned optimizers in the real-world tasks, and it is good to develop an algorithm to improve the generalization.\n- Theoretical analysis and comprehensive experiments are conducted to support the proposed flatness-aware regularizers.\n\nWeaknesses:\n- The experimental part lacks some important information. As the authors claimed in the checklist, no running time and variance is reported. Since training the neural optimizer with those two proposed regularizers~(Hessian and Entropy) is expensive and time-consuming, reporting the running time is important for the audience to evaluate the algorithm. Besides, in each independent run, the final result might vary significantly and it is necessary to give the statistics accounting for the variance.\n- It seems weird to keep both Figure 3 and Figure 4 in the main paper, since Figure 4 only adds the plot for L2O-DM-CL + Entropy. Why not just use Figure 4 for illustration?\n- In the abstract, the authors mentioned two types of generalization, optimizer generalization and optimizee generalization. However, in the whole paper, I think the authors mainly focused on improving the optimizee generalization using two flatness-aware regularizers. On the other hand, in Section 5.2, it was also claimed that entropy regularizer \"boosts both optimizee and optimizer generalizations of L2O in most cases, as shown in Figure 4\". I did not see any explanations why these regularizers can improve the optimizer generalization. \n- Why did the authors proposed two different regularizers? There is no clear description or conclusion that under certain scenarios, one of them will be preferred, except for some observations in four empirical settings. I think this is an important problem to be investigated, otherwise it seems that the authors just finds two irrelevant regularizers for the flatness and puts them together in one paper without further thoughts. Another interesting attempt could be combining the two regularizers in one training objective to see whether there will be further improvement.\n- Instead of Hessian spectrum, directly using spectral normalization on parameters of the optimizee can also benefit the generalization, as demonstrated in [1]. Did the authors try this simpler and efficient method compared with Hessian spectrum?\n- It is worth carrying out some experiments to incorporate the flatness-aware regularizers in the meta-testing stage for both learned optimizers and traditional analytical ones. It would be good if the performance trained with the learned optimizer (trained with flatness-aware regularizers) by the vanilla classification loss can outperform the performance trained with analytical ones by the regularized training objective.\n\n[1] Yoshida, Yuichi, and Takeru Miyato. \"Spectral norm regularization for improving the generalizability of deep learning.\" arXiv preprint arXiv:1705.10941 (2017).", "summary_of_the_paper": "The paper aims at improving the generalization of the learned optimizers. Some regularizers to induce the flatness of local minimas including Hessian spectrum and the entropy function are proposed. Both theoretical analysis and empirical results are presented to demonstrate that flatness-aware regularizers can enhance the generalization ability of optimizees.", "summary_of_the_review": "The current version needs some modifications on the experimental part to include more important information like running time and variance, as well as more experiments comparing the proposed algorithm with traditional optimizers. Also, it lacks the description of connections between two different regularizers. Thus, I think the paper is slightly below the acceptance criterion.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a learning-to-optimize approach that is "flatness-aware", i.e. it tries to find flatter minima in the loss landscape. The idea is accompanied by both theoretical and empirical verification. However, in the current state, the paper did not convince the reviewers about its potential impact. In particular, reviewer ZY6B points out that comparison with "sharpness-aware" minimization (SAM), which is a non-learned optimizer for seeking flat minima, is an important comparison that is currently missing. Another issue mentioned by the reviewers  ZY6B and wp2U relates to the lower performance of the learned optimizer compared to standard SGD in large models (lower than 80% test accuracy). These reviewers are interested in the question of whether this method can still produce good results in the competitive setting (e.g. > 90% accuracy on CIFAR). Considering the performance/baseline issue even on small datasets such as MNIST and CIFAR, the impact of the proposed method is unclear. I encourage authors to adopt more conventional baselines to better indicate the potential impact of their method, and do consider comparison with optimizers that directly aim to improve flatness, such as SAM.
Decision: Reject