Implicit regularization in Heavy-ball momentum accelerated stochastic gradient descent
Authors: Avrajit Ghosh
He Lyu
Xitong Zhang
Rongrong Wang
Venue: ICLR 2023 notable top 25% (2023)

Abstract: It is well known that the finite step-size ($h$) in Gradient descent (GD) implicitly regularizes solutions to flatter minimas. A natural question to ask is \textit{Does the momentum parameter $\beta$ (say) play a role in implicit regularization in Heavy-ball (H.B) momentum accelerated gradient descent (GD+M)?}. To answer this question, first, we show that  the trajectory traced by discrete H.B momentum update (GD+M) is $O(h^2)$ close to a continuous trajectory induced by a modified loss, which consists of an original loss and an implicit regularizer. This implicit regularizer for (GD+M) is indeed stronger than that of (GD) by factor of $(\frac{1+\beta}{1-\beta})$, thus explaining why (GD+M) shows better generalization performance and higher test accuracy than (GD). Furthermore, we extend our analysis to stochastic version of gradient descent with momentum (SGD+M) and propose a deterministic continuous trajectory that is $O(h^2)$ close to the discrete update of (SGD+M) in a strong approximation sense. We explore the implicit regularization in (SGD+M) and (GD+M) through a series of experiments validating our theory. 

Published as a conference paper at ICLR 2023
IMPLICIT REGULARIZATION IN HEAVY-BALL MOMEN- TUM ACCELERATED STOCHASTIC GRADIENT DESCENT
Avrajit Ghosh" He Lyu
Xitong Zhang Rongrong Wang
Department of Computational Mathematics, Science and Engineering (CMSE)
Michigan State University Correspondance to: ghoshavr@msu.err edu
ABSTRACT
It is well known that the finite step-size (h) in Gradient Descent (GD) implic- itly regularizes solutions to flatter minima. A natural question to ask "Does the momentum parameter 3 play role in implicit regularization in Heavy-ball (H.B) momentum accelerated gradient descent (GD+M)?" To answer this ques- tion, first, show that the discrete H.B momentum update (GD+M) follows continuous trajectory induced by a modified loss, which consists of an original loss and an implicit regularizer. Then, we show that this implicit regularizer for (GD+M) stronger than that of (GD) by factor of (+=)). thus explaining why (GD+M) shows better generalization performance and higher test accuracy than (GD). Furthermore, we extend our analysis to the stochastic version of gradient descent with momentum (SGD+M) and characterize the continuous trajectory of the update of (SGD+M) in pointwise sense. We explore the implicit regular- ization in (SGD+M) and (GD+M) through a series of experiments validating our theory.
INTRODUCTION
Deep neural networks (NN) have led huge empirical successes in recent years across wide va- riety of tasks, ranging from computer vision, natural language processing, autonomous driving to medical imaging, astronomy and physics (Bengio & LeCun 2007 Hinton et al. 2006 Goodfellow et al., 2016). Most deep learning problems are in essence solving an over-paraman werrparamettrrized, large- scale non-convex optimization problem. A mysterious phenomenon about NN that attracted much attention in the past few years is why NN generalizes so well. Indeed, even with extremely over- parametrized model, NNs rarely show a sign of over-fitting (Neyshabur (017)). Thus far, studies along this line have successfully revealed many forms of implicit regularization that potentially lead to good generalization when gradient descent (GD) or stochastic gradient descent (SGD) algorithms used for training, including norm penalty (Soudry et al.) 2018), implicit gradient regularization
(Barrett & Dherin 2020), and implicit Hessian regularization (Orvieto et al. 2022a b) through noise injection.
In contrast, the family of momentum accelerated gradient descent methods including Polyak's Heavy-ball momentum (Polyak) 1964), Nesterov's momentum (Sutskever et al.) 2013), RMSProp (Tieleman al.) 2012), and Adam (Kingma & Bal [0144), albeit being powerful alternatives to SGD with faster convergence rates, are far from undl-understood in the aspect of implicit regularization. In this paper. analyze the implicit gradient regularization in the Heavy- momentum acceler- ated SGD (SGD+M) algorithm with the goal of gaining more theoretical insights on how momentum affects the generalization performance of SGD, and why it tends to introduce variance reduction effect whose strength increases with the momentum parameter.
RELATED LITERATURE
has been well studied that gradient based optimization implicitly biases solutions towards models of lower complexity which encourages better generalization. For example, in an over-parameterizzd quadratic model, gradient descent with a near-zero initialization implicitly biases solutions towards
denotes equal contribution

---

Published as a conference paper at ICLR 2023
having a small nuclear norm (Arora et al. 2019; Gunasekar et al. 2017: Razin & Cohen 2020
in a least-squares regression problem. gradient descent solutions with initial guess are biased towards having a minimum l norm (Soudry al. 2018 Neyshabur et al., 2014 Ji & Telgarsky
2019 Poggio et al.] 2020). Similarly, in a linear classification problem with separable data, the solution of gradient descent is biased towards the max-margin (i.e., the minimum f2 norm) solution (Soudry et al.) 2018). However in (Vardi & Shamir [02I), the authors showed that these norm-based regularization results proved on simple settings might not extend to non-linear neural networks.
The first general implicit regularization for GD discovered for all non-linear models (including neu- ral networks) is the Implicit Gradient Regularization (IGR) (Barrett & Dherin 2020). It is shown that the learning rate in gradient descent (GD) penalizes the second moment of the loss gradients, hence encouraging discovery of flatter optima. Flatter optima usually give higher test-accuracy and more robust parameter perturbations (Barrett & Dherin 2020).
Implicit Gradient Regularization was also discovered for Stochastic Gradient Descent (SGD) (Smith) al. 2021 Li et al. (2019). as one (but perhaps not the only one) reason for its good generalization. SGD is believed to also benefit from its stochasticity, which might act type of noise injection to enhance the performance. Indeed, it is shown in (Wu et al. 2020) that, by injecting noise to the gradients, full-batch gradient descent will be able to match the performance of SGD with small batch sizes. Besides injecting noise to the gradients, many other ways of noise injections have been discovered have an implicit regularization effect on the model parameters, including noise injection to the model space (Orvieto et al. (2022)) and those to the network activations (Camuto al. 2020). However, how these different types of regularization cooperatively affect generalization still quite unclear.
The effect of generalization in momentum accelerated gradient descent has been studied much less. Li al. (2019) analyzed the trajectory of SGD+M and found that it can be weakly approximated by solutions of certain Ito stochastic differential equations, which hinted the existence of IGR in (SGD+M). However, both the explicit formula of IGR and its relation to generalization remain un- known. Recently, in (Wang et al.) 2021), the authors analyzed the implicit regularization in momen- tum (GD+M) based on a linear classification problem with separable data and show that (GD+M) converges to the L2 max-margin solution. Although this is one of the first proposed forms of im- plicit regularization for momentum based methods, it fails to provide an insight on the implicit regularization for momentum in non-linear neural networks.
Recently, (eelasi & Li 2022) has shown that the (GD+M) increases the generalization capacity of networks in some special settings (i.e., simple binary classification problem with two layer network and part of the input features are much weaker than the rest), but it is unclear to which extent the insight obtained from this special setting can be extended practical NN models.
To the best of our knowledge, no prior work has derived implicit regularization for (SGD+M) for general non- con-linear neural networks.
IMPLICIT GRADIENT REGULARIZATION FOR GRADIENT DESCENT AND ITS RELATION TO GENERALIZATION
We briefly review the IGR defined for GD (Barrett Dherin 2020) which our analysis will be based Let E(x) be the loss function defined over the parameters space E R" of the neural network. Gradient descent iterates take a discrete step (h) opposite to the gradient of the loss at the current iterate
x *+1 = xx Ex**)
(1)
With an infinitesimal step-size (h 0), the trajectory of GD converges to that of the first order ODE
x((1) = - (x(()))
(2)
known as the gradient flow. But for a finite (albeit small) step size h. the updates of GD steps off the path of gradient flow and follow more closely the path of a modified flow:
where O(x) = E(x) + h []E(()))
x' Ex((())),
(3)
ii

---

Published as a conference paper at ICLR 2023
It is shown (Barrett & Dherin 2020 via the so-called classical backward analysis that when GD and the two gradient flows 2 and all set off from the same point x", the next gradient update x*+1 is (hh)) close to the original gradient flow (2) evaluated at the next time point f h + ta. but
(hh) close to the modified flow (3 evaluated at 1-++1 So, locally, the modified flow tracks the gradient descent trajectory more closely than the original flow. To discuss the global behaviour of GD, we need the following definition of closeness between two trajectories.
Definition 3.1 (O(ha) (O(h"")) cllosensss in the strong sense), Fix some I O, we call the trajectory of the discrete GD-update x* and a continuous flow x(tx) to be O ) in the strong sense
T
max
|xx* x(xx)]l2 S ch REK.
K
h
where t = hk, and c is some constant independent of h and k
Definition 3.1 quantifies the global closeness of discrete trajectory and continuous one. By accumulating the local error, one can show that setting off from the same location x", the original gradient flow equation 2 is O(h)-close to the GD trajectory while that of the modified flow 3 is clooee Based on this observation, the authors defined the O(h) term & VIV in the modified flow as the IGR term and concluded by stating that it guides the solutions to flatter minima in highly non-convex landscape.
To justify why minimizing // F12) is good idea and why it encourages flat minimizers (which seems to be missing from the original paper), we borrow an argument from (Foret et al., 2020) that was originally developed for different purpose.
Due to the PAC-Bayes analysis (Neyshabur 2017). a simplified generalization bound for NN derived under some technical conditions can be stated (Foret al., 2020)
[xx1²²
|| (x(x)= < max Ls(x +c)
for any > 0,
where LD is the population loss (i.e., the generalization error), Ls is the empirical/training loss, and h R4 + R4 is some strictly increasing function. One try to minimize this upper bound in order to minimize the generalization error Lp. The term h [|x]] in the upper bound can be controlled by activating a weight decay penalty during training, and the first term in the bound is usually written into
max Cs(x+e) max (Cs(x + Ls(x)) (x(x) S5
training
sharpness
which consists of the training loss and an extra term called sharpness, minimizing which will help with generalization. Since directly minimizing the sharpness is difficult, one can then use the fol- lowing first-order Taylor approximation
max (xs(x c) Ls(x)) max (VLss( pll (x)))l |e||Sp
Se
Using our notation, VCs(x) is E so the sharpness approximately proportional to |VE which is the square root of the IGR term.
4 IMPLICIT GRADIENT REGULARIZATION FOR HEAVY BALL ACCELERATED GRADIENT DESCENT
The main mathematical challenge in studying the IGR momentum updates is that we now need to perform global error analysis instead of the local backward analysis, as the momentum updates utilizes the entire update history.
Weak sense approximation as studied in et al. 2019) only requires distributions of the sample processes x and x to be close, whereas our strong-senss sense approximation requires each instance of x and & to be close. The strong sense IGR found using the latter is valid for trainings with any fixed random batch sequence and any fixed initialization, while the former only characterizes the mean trajectory taking expectation over many different trainings (each with random hatch-sequence and initialization).
ii

---

Published as a conference paper at ICLR 2023
The IGR for Heavy-ball momentum was previously analyzed in (Kovachki & Stuart 2021 studying its relationship with the damped second order Hamiltonian dynamic
through
mx (1) + 7x" (1) + VE(((()) =0
which has been well-known as the underlying ODE for the momentum updates. However, only O(h) closeness is proven between the momentum updates and this ODE trajectory under general step size assumptions 2 which is not enough since the implicit regularization term ' || El| itself of order Ohh). addition, this approach is difficult to be applied the stochastic setting.
In this paper, we circumvent the use of the second order ODE (as it only gives (h) closeness) and directly obtain a continuous path that is O(h")-close to the momentum update for both GD and SGD. This is achieved by linking the momentum updates with first order piecewise ODE, proving that the ODE has a piece-wise wise differentiable trajectory that is O(h2)-close to the momentum updates, and then using its trajectory study the IGR. The detailed argument can be found in the appendix. Here we provide the final mathematical formula for the implicit regularization of (Heavy-Ball) momentum based gradient descent method (IGR-M).
Theorem 4.1. (IGR-M); Let the loss for the full-batch gradient E be smooth and 4-times differen- tiable, then the (GD+M) updates
x k+1
x
(x + 3(x* x
Ak = 1,2,...,
O close to the flow of the continuous trajectory of the piecewise first-order firstooref ODE
gk+1
hy(1+ + EEE(())) (x41]
x1)) 1 -
VE(X(())
(4)
2(1 - 3)3
where ly = kh and
++1 (1 (1-3) B)
=1 B2k+2) ) 4(k + 1)t
(1+ B)
Since 3h quickly decays to 0 as grows, for a sufficiently large iteration k, equation reduces to:
1
x (t)
VE(((t)) t [0,T]
(5)
driven by the modified loss E(x(()) E(X(()) fixed time T, there exists a constant C. such that for any learning rate 0 < h < T, we have
(E((( More specifically, for I(-
(L+D)A
T
[x(tx) x*1 ll2 S Ch2 tx kh,
k= 1,2, ....
(6)
Comparing the continuous trajectory traced with 5 and the one without momentum immedi- ately have a few important observations:
Remark 4.1. Ignoring the (h) terms in 5 and we see that the momentum trajectory is driven by a force that is 1-6 times as large as that for GD. Therefore, using the same learning rate, (GD+M) expected to converge I'z times fast GD. Alternatively, (GD+M) with learning rate h has From now on, we call h I-3 the
h roughly the same convergence rate GD with learning rate T-3- effective learning rate of (GD+M).
Remark 4.2. terms of the IGR, we can see that adding the momentum amplifies the strength of the IGR (the coefficient front of the IGR term increased from the \/2 in (GD) the 1 (1- in
1+3
(GD+M). Even when letting the effective learning rates of the two methods equal (i.e., hCD
1-9
hGD),, the implicit regularization in (GD+M) is still I-2 times stronger.
Remark 4.3. The IGR for (GD+M) reduces to the IGR for GD when 3 0.
With an additional momentum parameter 3, the strength of the implicit regularizer increases by factor of 1+3 Hence for increasing values of momentum parameter B3, the strength of the regular- ization increases, thus preferring more flatter trajectories through its descent.
2 An ) closeness is proven under very stringent conditions on the learning rate, which excludes the interesting regime where momentum has advantage over plain GD in terms of the convergence rate and stability
iv

---

Published as a conference paper at ICLR 2023
4.1 IGR-M IN A 2D LINEAR MODEL
We first show the impact of IGR-M in a very simple setting that minimizes a loss function with two learnable parameters, i.e., (u11, viz) arg min w2 F(w1, w2) where E(w1, w2)
(V wi w22))² Here æ, y, Wi, w2 are scalars and mimics simple scalar linear two-layer net- work. For a given scalar (I, v). the global minima of E(wi, w2) are all the points the curve W1 = The continuous gradient flow of the parameters are w/ (t) Bun The IGR for this problem is AGR IVEE
aE and w/s(t aE
Dws
4 (q + w3)E. E. which will regularize the trajectory find minima with a smaller value (wf w (towards the cross) among all the global minima lying on W2 = y. We inten- tionally chose the same experiment as (Barrett & Dherin (2200) to compare the effect of implicit regularization for (GD) and (GD+M).
Initialization: w1=2.8, w2=3.4, affactive 1r=1e-2
20
2.0
dlaal mnnn
@D+Maata-d
-Mbota-0.
For a fair comparison between (GD) and (GD+M), we set the effective learning ACD+M rates to be the same, i.e, MGD = 1- as Remark 4.1 With the same initial- ization (wf, wg) (2.8,3.4
10
explore and track the path of four trajectories with (hGD+M, B) being (5 10 0.5). (2 10 10-3, 0.8), (10-3 0.9) and In all the four cases, the effective learn- ing rates are the same, i.e, hpp (1-3)
0.0
0.5
2.0
0.0)..
Figure 1: Implicit regularization for (GD+M) is stronger than that of (GD) for the same effective learn- h ing rate (1-35- As 3 increases the optima seems to find solution with lower norm. This confirms Remark 4.2 that the strength of implicit regularization increases with B. The background color denotes the magnitude of the norm of the gradient, i.e., |V
hGD = 10-2 We make the following observations: a) For all the four trajec- tories, the converged weight parameters w" = (wf, wz) lic on the global minima curve. b) With increasing value of 3, the converged solutions have decreasing value of &a norm (or increasing strength of im- plicit regularization), i.e, || 1111--,00.9) -,00.))l22 1|2 <
llz < [ 10-3 -0.y. < | l1a- This observation supports Remark (.2 that the strength of implicit regularization increases with 3, even with the effective learning rate.
5 IMPLICIT REGULARIZATION IN SGD WITH MOMENTUM
In SGD, the full-batch gradient is replaced by it's sampled unbiased estimator. Assume that the loss function E(x) has the following form
1 M Σ (x)
E(x)
where Egy is the jth mini-batch loss. the 1tth iteration, we randomly pick a mini-batch, whose loss is denoted by Ex, and update the parameters accordingly. The heavy-ball accelerated SGD iterates as
x k+1 = x* - -x* + 3(x* -x-11 A
x x - (xx
(8)
0 -
For each iteration k; the update driven by the current mini-batch loss Eg. Its continuous approxi- mation is
x((1= ((0==vEa(xi) -VE(((()) + 3 (x((x) x(lx-1)) for la < l < 1641
(9)

---

Published as a conference paper at ICLR 2023
during the time [tg, f As result, the trajectory of x is continuous but piece-wise differentiable as it is easy to see that the left and right-side derivatives are not equal at a transit point tę from one batch to another, x' <((t)) * x' x(t)). ). Therefore, expect the implicit regularization term to also have discontinuous derivatives on different intervals. Below we present the mathematical formula for IGR-M in the stochastic setting.
Theorem 5.1. [IGR-M stochastic version (IGRM,)) 3) Let the loss for each mini-batch Eg be 4-times differentiable, then the Heavy Ball momentum updates 8 are O(h)) close to the trajectory of the gradient flow with the modified loss,
(1) -Vbxx)))(( for to St < tx+1
k-1
where,
Ex(x) = Gx((X) (IVG) +
8 VV
(10)
force
r=0
IGRM,
with G1 (x(1)) = Er=o gh E., (x(1)) Specifically, there exists constant C such that
[(((ta) - xx11xx1225 Ch2 k =11.2....
The theorem tells us that the IGR for momentum is in the form of E norms of VG which is a weighted sum of the historical gradients V.R, k = O..... n. by powers of 3 and evaluated at the current location x(). In addition, the stochastic IGR-M has different expressions on different intervals, caused by the abrupt changes between adjacent batches. Some further remarks about the (SGD+M) algorithm are listed below.
Remark 5.1. The flow of the continuous trajectory is governed by the driving-force - VV&((x()) and the negative gradient of an implicit regularizer IGRM,(() = <() +
> 8 ( which depends on both the learning rate h and momentum 8.
Remark 5.2. When 0. 10 reduces to
h
x
Ex((())
(VE
t E [t,, tx+1)
(11)
force
IGR
which is the modified loss for SGD.
Remark 5.3. Taking expectation over the random selections of batches, we get E(IGRM,) = A(1+-9)
V F3| - F, where F E|| F,g F212 (appendix Th 3.1). In comparison, the IGR term in SGD after taking expectation is 1/2 (|IV E 2| + F) (Smith al. 2021 which is much weaker. Even with the adjusted learning rate, the IGR in (SGD+M) IS still about I/(1 3) times stronger than SGD.
4(--)>
Remark 5.4 (Variance reduction). We notice that momentum has variance reduction effect. Ex- plicitly, suppose the effective learning rate (Remark 4.1 is used so that the force terms in (SGD) and (SGD+M) have the same expectation, and then
compare their variance. Let the covari- ance matrix of V Es at a fixed point x be C' == E(V Es (x) - V Ex)) (V Ex(x) V E()))" Here V Ex) denotes the full-batch gradient. Then the covariance matrix of the force - EE driving (SGD) is exactly C, while that of the force - G driving (SGD+M) is only (1-3) C (appendix Th 4.1), T+3
which is about (1-A)
[1+B] times smaller.
Remark 5.5. is observed and confirmed by many literature that a larger variance of SGD iterations (caused by either small batch size or larger learning rate (Smith & Le. 2017 Li et al. 2017)) increases the generalization power. Larger variance mini-batch gradients increases the escape efficiency of SGD from bad local minimas (Ibayashi & Imaizumil 2022) ISee Appendix section-7 for detailed discussion], hence increasing generalization power. Therefore, the variance reduction effect of (SGD+M) hurts generalization. On the other hand, the fact that (SGD+M) has stronger IGR (Remark (.3) and that (SGD+M) is more stable than (SGD) to the use of large effective learning rates (see e.g., (Cohen al. 2021 tend to help with its generalization. This explains why in practice we usually observe that (SGD+M) is only slightly better than (SGD).
vi

---

Published as a conference paper at ICLR 2023
1.00
1.00
0.9
0.90
0.0
0.95
0.50
0.40
20.44
00.30
0.30
0.93
0.40
0.40
0.30
100
200
400 Epocha
500
650
200
0.02
100
220
350
500
620
800
0.00
Esochu
(a) Accuracy
(b) Loss
Figure 2: (GD): Classification results of ResNet- 18 on MNIST datasct performed with full-batch gradient descent. Figure shows the effect of implicit regularization due to the finite learning rate h. Test accuracy improves with higher learning rate till h 0.2. For h 0.2, the best test-accuracy is achieved at 95.72%
1.00
2.00
2.50
0.35
0.98
1.00
Aov2 Uine
3.50
0.90
0.96
Soach
0.4
0.04
5.92
3.25
Leer
0.20
0.00
0.15
0.15
0.42
3
+040
0.10
130
200
300
800
200
500
600
(a) Accuracy
(b) Loss
Figure 3: (GD+M): Classification results of ResNet-18 MNIST dataset trained with various values of momentum parameter for full-batch gradient descent. The test-accuracy reported 96.22%
6 NUMERICAL EXPERIMENTS
Our first experiment is to compare the full-batch (GD) with (GD+M). For linear least-squares problem with a Hessian matrix bounded by L the spectral norm, it is well-known that (e.g.. (Cohen 2+28 and GD stable as long as S Z. This means, et al. 2021) (GD+M) is stable as long as < -
the maximum achievable effective learning rate by (GD) is 2. while that by (GD+M) can be as large 2(1-8)
as L(1-3]. Since larger effective learning rates means a stronger IGR, (GD+M) clearly benefits from its large stability region. To confirm this, ResNet-18 is used to classify a uniformly sub-sampled
MNIST dataset with 1000 training images. All external regularization schemes except learning rate decay and batch normalization have been turned off. We perform the following experiments 1) Full-batch gradient descent with 3 0 (i.c.. GD) for various learning rate h and the best test accuracy is noted (in Figure 2 to be 95.72%. 2) Full-batch gradient descent with momentum (GD+M) performed for various 3 with a fixed step- size h = 0.1 and the best test-accuracy is noted (in Figure 3) to be 96.22%. Our observation is that the best performance of GD (across all learning rates) is worse than the best performance of (GD+M) (across all 3's). This observation failed to be explained by the known theory of edge of stability but can be well-explained by our implicit regularization theory for (GD+M) as adding momentum increases the strength of the IGR.
To study the effect of implicit regularization (SGD+M), series of experiments have been per- formed on an image classification task. Four well-known and popular network architectures
"edgge stability" (EOS) Cohen et al. 2021 is a phenomenon that shows during network training by the full batch gradient descent, the sharpness E tends to progressively increase until it reaches the threshold
and then hovers around it. For GD+M, the sharpness will hover around a large value 20+0) Since larger sharpness usually means worse generalization, the EOS theory then predicts that adding momentum hurts the generalization.
vii

---

Published as a conference paper at ICLR 2023
0.9
6.9
0.8
0.8
0.7
0.6
0.6
6.6
0.6
0.5
200
0.5
Satund and
Entued baed
0.4
Enteodh and E1//se
0.4
Intoodt bedf B1/ccl
Eddantia and Ri/aum
0.3-
Intuetia and B1/om
50
100
155
200
250
100
150
200
230
Epoochs
Spochss
(a) WideresNet-1644
(b) ResNet-18 18
1.0-
3.0
1.0
3.0
0.0
0.9-
0.8
0.5
0.6
Enteed pnd 67/mm
0.4
0.4
Estund $1/mm
03-
and (5)m
0.3
0.3
kud 2%
0.8
Edds cas Bi/aa
6.2
Bitb cul B1/mc
0.2
200
230
Spochs
(c) ResNet-50
(d) DenseNet- 121
Figure 4: Classification results for CIFAR-IO dataset with various network architectures with combinations of (h, () chosen such that the effective learning rate (1-Øy remains samc. In all of the experiments, external regu- larization weight-decay, scheduler, The results have heen averaged over 3 random seeds having different initializations. (SGD+M) has a) higher test accuracy for increasing than (SGD) confirming Remark 5,4 Less variance for test accuracy confirming Remark 5.3
h
are kept off (except Batch-normalization))
namely DenseNet (landola et al. 2014). ResNet-18, ResNet-50 (He et al. 2016) and WideResNet (Zagoruyko Komodakis] 2016) are trained classify images from the CIFAR- and CIFAR-100 datasets. We are interested to know how well training these networks with (SGD) and (SGD+M) respectively can generalize well onto the test dataset. To solely observe the effects of the momen- tum parameter 3 and learning rate h in generalization,
turn off all the external regularization like dropout, weight-decay and label-smoothing. We fix the batch-size to 640 all our experiments.
CIFAR-10
CIFAR- 100
3th DN-121 RN-18
RN-50
WRN-16-8
DN-121
RN-18
RN-50
WRN-16-5
0.0/0.10 10 84.0.1..0
79.745.66
79.3+2.55
65.3+18.1
60.4
53.1::0.6
47.412..1
38.6
0.3/0.07
85.145.4 78.749.3
8..0+
72.547.44
60.048.6
52,7+110
48.9+222
37.0
0.5/0.05
87.611.2 81.5+0.9 80.740.7
71.849.8
63.2+2.4
53.341.1
50.311.0
39.413.9
0.8/0.02
88.6.0.7 82.410.4 82.410.7
75.412.8
64.7100.8
54.310.. 49.610.7 40.6
Table 1: Testing accuracy of CIFAR-10 and CIFAR-100 with different momentum and learning rates h, the same effective learning rate & (I-3 = 0.1. The best performance of different models is highlighted. The mean and standard deviation of test accuracy is calculated over last 5 epochs and three random seed initializations. DN-121 is the Densenet-121, RN-18 and RN-50 denote the Resnet-18 and Resnet-50, WRN- 16-8 represents the WideResnet with depth 16 and width-factor 8.
In the first experiment, we showed (GD+M) has a larger stability region than (GD) and hence allows for the use of larger effective learning rate. The same conclusion holds for (SGD+M) and (SGD). However, here we want to show that even in the region where both algorithms are stable, (SGD+M) still not just scaled version of (SGD). For this purpose, we pick a small learning rate to ensure stability of both algorithms, and keep the effective learning rate ाা- for (SGD+M) to be the same the learning rate for (SGD) (both equal 0.1). We observe from Table T (also Figure 4) that the
viii

---

Published as a conference paper at ICLR 2023
maximum test accuracy is almost always achieved at the highest value for 3. This observation is consistent with Remark-5.3 where we showed that the implicit regularization in (SGD+M) is indeed stronger than (SGD), even after the learning rate adjustment.
The standard deviation of test accuracy in Table T is calculated over the last 5 epochs and three ran- dom seed initialization. Lower standard deviation indicates a smoother test accuracy curve meaning less variation of test accuracy within epoch interval. We observe that the lowest standard devi- ation is achieved at the highest value of 3. Hence the observation that variance reduction effect is more prominent with higher is consistent with Remark 5.4
7 COMBINED EFFECTS OF IGR AND NOISE INJECTION
Despite its close relation sharpness (Section 3), the IGR term V gets very weak and inrele- vant as x" approaches a local minimizer, since VE - 0. However, we find that this would not be the case if there noise injection, which can help the IGR term retain its power minima. More specifically, as studied previous literature (Orvieto et al., [022b]] [Camuto et al.) 2020), the algorithm resulting from injecting noise each iteration of GD usually called PGD (Perturbed gradient descent) that essentially minimizes an averaged objective function
local
R(x) == -0. 17).
For small values of σ, we can expand R(x) into
R(x) E(x) ( (x V''' E(x)) E(x) E(x)+ 1 --
where Tr denotes the trace operator. Thus minimizing R(x) regularizes the trace Hessian of E. When minimizing R(x) using SGD) type of update, the iterations would be
++1 x
EV + m)k,, where 7/s N(0,001)
which is known as form of PGD. Because of the finite learning rate, the updates would follow the modified flow with an IGR term, which in this case is
E(x + nh) + IVE( + 11)
In expectation, the modified loss is
+ h VE(x 7hk 11))
E
E(x + mk)
4
Ey
E(x) ni [ + 2
(nT)) /' 22 E()
+
Eyy (x + F FXx mxl11 + O(03)
=E(x) + T'TT(("E(xx)) + [ [(x) IVE(() + V Exx) Me. mell + O
We see that now there is Hessian regularization term E(x coming out of IGR which does vanish even around local minimizers, and it's strength is proportional the learning rate. We expect this new regularization term to get stronger when momentum is added, as momentum amplifies the power of IGR (Remark 4.2 (.3). This observation suggests that IGR and noise injec- tion two different types of implicit regularization might be able to reinforce each other when used collaboratively.
CONCLUSION
This work studies the generalization of momentum driven gradient descent approach through the lens of implicit regularization (IR) with both theoretical analysis and experimental validation provided. We examined the similarities and differences between (SGD) and (SGD+M) and find that (SGD+M) with suitable parameters outperforms (SGD) in almost all settings. Moreover, we found that in addition to momentum, IGR may also be magnified by noise injection, which a topic we want to further explore in the future.
ix

---

Published as a conference paper at ICLR 2023
REFERENCES
Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. Advances Neural Information Processing Systems, 32, 2019.
David GT Barrett and Benoit Dherin. Implicit gradient regularization. arXiv preprint atXiv:2009.11162 2020.
Yoshua Bengio and Yann LeCun. Scaling learning algorithms towards Al. In Large Scale Kernel Machines. MIT Press, 2007.
Alexander Camuto, Matthew Willetts, Umut Simsekli, Stephen J Roberts, and Chris C Holmes. Explicit regularisation in gaussian noise injections. Advances in Neural Information Processing Systems, 13:16663-1661, 2020.
Jeremy M Cohen, Simran Kaur, Yuanzhi Li, J Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. arXiv preprint 10X0::2003.00065, 2021.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimiza- tion for efficiently improving generalization. arXiv preprint arXiv:2010.01412, 2020.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT Press, 2016.
Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Sre- bro. Implicit regularization matrix factorization. Advances in Neural Information Processing Systems, 30, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Geoffrey E. Hinton, Simon Osindero, and Yee Whye Teh. A fast learning algorithm for deep belief nets. Neural Computation, 18255-1155 2006.
Forrest landola, Matt Moskewicz, Sergey Karayev, Ross Girshick, Trevor Darrell, and Kurt Keutzer. Densenet: Implementing efficient convnet descriptor pyramids. arXiv preprint arXiv: 1804.1869, 2014.
Hikaru Ibayashi and Masaaki Imaizumi. buasi-potentiil theory for escape problem: Quantitative sharpness effect on SGD's escape from local minima, 2022. URL httttpss:://peeeeeviee
net/Eorum?idd-lnne
1F3
Samy Jelassi and Yuanzhi Li. Towards understanding how momentum improves generalization in deep learning. In International Conference on Machine Learning, pp. 9965-10040. PMLR, 2022.
Ziwei Ji and Matus Telgarsky. The implicit bias of gradient descent nonseparable data. In Conference on Learning Theory, pp. 1772-1798. PMLR, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Nikola B Kovachki and Andrew M Stuart. Continuous time analysis of momentum methods. Journal of Machine Learning Research, 22(17):1 2021.
Qianxiao Li, Cheng Tai, and Weinan. Stochastic modified equations and adaptive stochastic gradient algorithms. In International Conference Machine Learning, pp. 2101-2110. PMLR, 2017.
Qianxiao Li, Cheng Tai, and Weinan E. Stochastie modified equations and dynamics of stochastic gradient algorithms i: Mathematical foundations. Journal of Machine Learning Research, 20(40); 1-47, 2019. URL (Rtptttp: / //3ml r. http:///rrrrppppa////sss/e/e/e//
/tteeerssvvv//e v200 52.6. html]

---

Published as a conference paper at ICLR 2023
Behnam Neyshabur. Implicit regularization in deep learning. arXiv preprint arXiv:1709.0199 2017.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. In search of the real inductive bias: On the role of implicit regularization in deep learning. arXiv preprint arXiv:1412.660 1412.6614, 2014.
Antonio Orvieto, Hans Kersting, Frank Proske, Francis Bach, and Aurelien Lucchi. Anticorrelated noise injection for improved generalization. arXiv preprint arXiv:220.008831 2022a.
Antonio Orvicto, Anant Raj. Hans Kersting, and Francis Bach. Explicit regularization in over- parametrized models via noise injection. arXiv preprint arXiv:2206.04613 2022b.
Tomaso Poggio, Andrzej Banburski, and Qianli Liao. Theoretical issues in deep networks. Proceedings of the National Academy of Sciences, 11744))30039-30445 2020.
Boris T Polyak. Some methods of speeding up the convergence of iteration methods. Ussr
computational mathematics and mathematical physics, 4(5):1-17, 1964.
Noam Razin and Nadav Cohen. Implicit regularization in deep learning may not be explainable by norms. Advances in neural information processing systems, 33:21174-21187, 2020.
Samuel I. Smith and Quoe V Le. A bayesian perspective on generalization and stochastic gradient descent. arXiv preprint arXiv:1710.06451, 2017.
Samuel L Smith, Benoit Dherin, David GT Barrett, and Soham De. On the origin of implicit regu- larization in stochastic gradient descent. arXiv preprint arXiv:2101. 12176, 2021.
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The im- plicit bias of gradient descent on separable data. The Journal of Machine Learning Research, 19 (1):2822- 2878, 2018.
Ily Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initial- ization and momentum in deep learning. In International conference on machine learning, pp. 1139-1147. PMLR, 2013.
Tijmen Tieleman, Geoffrey Hinton, et al. Lecture 6.5-rmsprop: Divide the gradient by running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):26-31, 2012.
Gal Vardi and Ohad Shamir. Implicit regularization relu networks with the square loss. In Conference Learning Theory, pp. 4224 4258. PMLR, 2021.
Bohan Wang, Qi Meng, Huishuai Zhang, Ruoyu Sun, Wei Chen, Zhi-Ming Ma, and Tie- Yan Liu. Does momentum change the implicit regularization on separable data?, 2021. URL https:
Jingfeng Wu, Wenqing Hu, Haoyi Xiong, Jun Huan, Vladimir Braverman, and Zhanxing Zhu. On the noisy gradient descent that generalizes as sgd. In International Conference on Machine Learning, pp. 10367-10376. PMLR, 2020.
Sergey Zagoruyko and Nikos Komodakis. arXiv.160..07146, 2016.
Wide residual networks.
arXiv preprint
xi


Reviews:
[{"review_id": "mMaaHPaxWP", "reviewer": "Reviewer_DtPB", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides a new analysis for implicit regularization for the heavy ball momentum gradient descent. The paper shows that the momentum updates for gradient descent and its stochastic counterpart are closer to the modified regularized gradient flow than previously known. Their analysis is based on studying a piecewise first order ODE, as opposed to a second order ODE analysis that was undertaken by a previous work which makes the crucial difference in the final results. ", "summary_of_the_review": "The paper makes a non-trivial contribution to the field and should be of interest to the community. The result demonstrating the regularizing effect of heavy ball in both the classical gradient descent and the stochastic settings is interesting, and so is the presented analysis. ", "strength_and_weaknesses": "Strengths:\n\n- The paper explains the setup well, explains the implicit regularization effect due to the closeness to the modified flow, and also why the modified flow is suited for better generalization which helps in a lot in getting a complete picture of the impact of this work.\n- The variance reduction effect of momentum in SGD is one of the most interesting aspects of the presented analysis. The authors have made relevant comments how the overall generalization is likely due to a combination of various factors including variance, stability and the contributing implicit regularization. The effect of variance, for example, still requires further imho even though there are a couple of studies cited in this work that talk about how smaller variance can lead to bad local minima. \n- The relevant literature is well-cited and well discussed, mentioning key contributions and how they relate to this paper. \n- The empirical section is largely sufficient and complements the theoretical contributions well. \n\nWeaknesses: \n- I had a hard time reading the paper the first time mostly because I had to wait till eq 6 on page 4 before finally understanding and appreciating what O(h^2) closeness means and why that is important, even though the authors use it freely starting from right in the abstract. I suggest the authors consider re-wording this. I have not read all the related works cited in this paper, so I am not sure if it is a common jargon used in this community, but nevertheless it makes the paper a lot harder to read for more general audience. \n- Is there a reason why the authors used only the simple linear 2layer model to illustrate the IGR effect in Sec 4.1, especially considering that they have mentioned earlier in the paper about jelassi and Li 2022 not applicable to more general NNs? I understand they reproduced the setting of Barrett and Dherin 2020 but it would have been interesting to see a non-linear activation. Nevertheless, this is not a deal-breaker since the stability and full-batch experiments complement the theoretical contributions well. \n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RoMZwjskdx", "reviewer": "Reviewer_Yonc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper uses a classical tool in numerical analysis, namely modified equation, to account for the effect of small but finite learning rate used in (stochastic) gradient descent with momentum. As a result of this analysis and as a standard treatment, the continuous time limit of the optimizer, which is an ODE, will have additional O(h) term(s), which can then be employed to characterize the optimizer’s implicit bias. This paper investigates in such correction term(s) and shows that the momentum parameter also affects the strength of the implicit regularization. Empirical discussions and experiments are also provided.", "summary_of_the_review": "This paper is considering an interesting problem from a nice quantitative perspective. However, before I can better understand the innovation, I’m afraid I cannot give a very positive recommendation.", "strength_and_weaknesses": "Strength:\n\n(1) the problem considered is of critical importance to machine learning.\n\n(2) working out the algebra to quantify the role of momentum variable is nice.\n\n(2) the empirical discussions and experiments are interesting.\n\nWeakness:\n\n(1) Modified equation for SGD with momentum, unlike suggested, has already been studied in details. See for instance [Li et al. Stochastic modified equations and adaptive stochastic gradient algorithms. 2017] for a short version and [Li et al. Stochastic modified equations and dynamics of stochastic gradient algorithms i: Mathematical foundations. 2019] for a long version where the validity of modified equation, which is otherwise just a formal result, is actually rigorously quantified. \n\n(2) In addition, unlike the aforementioned results, the derivation of modified equation in this paper is only formal. In fact, the formal asymptotic expansion used for deriving modified equation will blow up once h becomes not that small. This problem is already starting to be addressed by the more recent literature, such as [Wang et al. Large Learning Rate Tames Homogeneity. 2021].\n\n(3) The main results, Theorem 4.1  and 5.1, are just short time results, meaning $T$ is fixed in Theorem 4.1 and $n \\leq T/h$ for fixed $T$ in Theorem 5.1. The derivation is based on standard (but still nice) numerical analysis, which in fact requires $T$ to be O(1) (i.e. small) in order for the hidden constants in the error bounds to be not exponentially large. This limits the applicability of the results to practical training, unless the setup is more specific so that early stopping becomes reasonable or necessary.\n\nIn terms of actions, it would be great if the above are at least fairly discussed in the paper.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RYa0IYnR1Z", "reviewer": "Reviewer_Cu2v", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work studies the generalization performance of momentum methods (GD+M, SGD+M) through the lens of implicit gradient regularization (Barrett & Dherin, 2020). The authors show that the effect of momentum can be understood as stronger implicit gradient regularization + stronger variance reduction (compared with GD/SGD using the same effective learning rate), and verify their claim with numerical justifications. These results provide a deeper understanding of the mechanism of momentum and some promising future directions (such as combining implicit gradient regularization with noise injection).", "summary_of_the_review": "This work explains the effect of momentum using the idea of implicit gradient regularization (IGR). The theorems nicely explain several empirical differences between GD/SGD and GD+M/SGD+M. I believe that the IGR has the potential to explain more NN optimizers, and thus the authors are heading in the correct direction. I vote for acceptance.", "strength_and_weaknesses": "This is a very interesting work, which extends the idea of implicit gradient regularization (IGR) (Barrett & Dherin, 2020) to explain the effect of momentum in GD/SGD. The theorems explain the empirical observation that with the same effective learning rate, the convergence of GD/SGD is more unstable (has larger variance) than GD+M/SGD+M. And also the experiments in (Ma, J., & Yarats, D., 2018) which fixed the effective learning rate, although they used Nesterov's momentum (maybe IGR can also be used to study Nesterov's momentum? or even QHM?). The authors give very detailed discussion and insights about the theorems, which I found very nice. I didn't check all the proofs, but they looks correct. \n\nSection 7 points out another very interesting topic, though without empirical verifications. From my perspective, the current contribution is sufficient for acceptance. I think it would be interesting to also discuss the effect of learning rate schedulers. The authors mentioned that the IGR term will get very weak when converging, and thus the generalization performance of GD/SGD and GD+M/SGD+M might be similar in this case. This can be observed in (Ma, J., & Yarats, D., 2018) for NAG and SGD after learning rate reductions. I think that the IGR is a powerful tool, and that the authors are moving in the correct direction using it to understand NN optimizers.\n\n\n\n(Ma, J., & Yarats, D., 2018)  Ma, J., & Yarats, D. (2018). Quasi-hyperbolic momentum and Adam for deep learning. In International Conference on Learning Representations.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2yPgrj03N3v", "reviewer": "Reviewer_5gUB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the implicit regularization effect in GD and SGD methods with momentum. It quantifies the regularization effect when algorithms take heavy-ball momentum step with a parameter beta, which partially explains the generalization ability and why momentum helps GD and AGD to find flatter minima. The authors also corroborate their findings with some empirical results.", "summary_of_the_review": "I like the problem it studies and the perspective it takes. With some efforts the exposition can be made better and I'd be happy to re-evaluate once my concerns have been properly addressed.", "strength_and_weaknesses": "The paper has a strong motivation, clean exposition and nice writing. I think it answers the very interesting problem of implicit regularization effect of momentum steps from one perspective, which may have positive effects on future work toward a more complete understanding of implicit regularization and acceleration. The following parts could be improved a bit more:\n\n1) The theoretical part seems to suggest the parameter beta of heavy-ball momentum shall be set to as high as possible, since then it leads to both faster convergence and better generalization. This I feel may be counter-intuitive to observations in practice. While the authors did try to touch upon such concerns in last sentences of Section 4, it may be helpful to particularly touch upon that point and how the theory guide practical choice of beta in more detail.\n\n2) The convergence rate difference of 1/(1-beta) factor seems to be somewhat different from the classic square root speedup for Polyak's momentum method in convex optimization. Is there a good intuition explaining why this is true?\n\n3) The sharpness and \"edge of stability\" explanation on middle of page 7 seems a bit hand-wavy. It might be helpful to elaborate in a bit more detail why this means (GD+M) shall potentially have worse test accuracy.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%