How Normalization and Weight Decay Can Affect SGD? Insights from a Simple Normalized Model
Authors: Ruosi Wan
Qiaosen Wang
Xiangyu Zhang
Yu-Wing Tai
Jian Sun
Chi-Keung Tang
Venue: Submitted to ICLR 2023 (2023)
Keywords: normalization
stochastic gradient descent
optimization

Abstract: Recent works(Li et al., 2020, Wan et al., 2021) characterize an important mechanism of normalized model trained with SGD and WD (Weight Decay), called Spherical Motion Dynamics (SMD), confirming its widespread effects in practice. However, no theoretical study is available on the influence of SMD on the training process of normalized models in literature. In this work, we seek to understand the effect of SMD by theoretically analyzing a simple normalized model, named as Noisy Rayleigh Quotient (NRQ). On NRQ, We theoretically prove SMD can dominate the whole training process via controlling the evolution of angular update (AU), an essential feature of SMD. Specifically, we show: 1) within equilibrium state of SMD, the convergence rate and limiting risk of NRQ are mainly determined by the theoretical value of AU; and 2) beyond equilibrium state, the evolution of AU can interfere the optimization trajectory, causing odd phenomena such as ``escape'' behavior. We further show the insights drawn from NRQ is consistent with empirical observations in experiments on real datasets. We believe our theoretical results shed new light on the role of normalization techniques during the training of modern deep learning models.

Under review as a conference paper at ICLR 2023
How NORMALIZATION AND WEIGHT DECAY CAN AF- FECT SGD? INSIGHTS FROM A SIMPLE NORMALIZED MODEL
Anonymous authors Paper under double-bling review
ABSTRACT
Recent works (Li al., 2020 Wan al. 2021 characterize an important mech- anism of normalized model trained with SGD and WD (Weight Decay), called Spherical Motion Dynamics (SMD), confirming its widespread effects prac- tice. However, no theoretical study is available on the influence of SMD on the evolution of the loss of normalized models in literature. In this work, we seek to understand the effect of SMD by theoretically analyzing a simple normalized model, named as Noisy Rayleigh Quotient (NRQ). On NRQ. We theoretically prove SMD can dominate the whole training process via controlling the evolution angular update (AU), an essential feature of SMD. Specifically, we show: I) within equilibrium state of SMD, the convergence rate and limiting risk of NRQ are mainly determined by the theoretical value of AU; and 2) beyond equilibrium state, the evolution of AU interfere the optimization trajectory, causing odd phenomena such as "escape" behavior. We further show the insights drawn from NRQ is consistent with empirical observations in experiments on real datasets. We believe our theoretical results shed new light on the role of normalization techniques during the training of modern deep learning models.
INTRODUCTION
Normalization (loffe & Szegedy 2015 Wu & He 2018) one of the most widely used deep learning
techniques, and has become an indispensable part in almost all popular architectures of deep neural networks. Though the success of normalization techniques indubitable, its underlying mechanism still remains mysterious, and has become a hot topic in the realm of deep learning.
Many works have contributed in figuring out the mechanism of normalization from different as- pects. While some works (loffe & Szegedy, 2015) (Snnturar et al.) 2018: |offer et al. 2018 Bjorck al., 2018 [Smmmers Dinneen 2019 De Smith (2000) focus on intuitive reasoning or empirical study, others (Dukler et al. 2020, Kohler et al.] 2019 Cai et al., 2019 Arora et al., 2018 Yang al. 20188] Wu et al. (2200) focus on establishing theoretical foundation. A series of
Chiley et al. 2019 Kunin et al. 2021 Li et al. 2020; Wan et al.) 2021 Lobacheva et al. 2021 Li & Arora 2019) have noted that, in practical implementation, the gradient of normalized models is usually computed in straightforward manner which results in its scale-invariant property during training. The gradient of a invaraan weight is always orthogo- nal the weight, and thus makes the training trajectory behave as motion sphere. Besides, in practice, many models are trained using SGD with Weight Decay (WD), hence normalization and WD in SGD can cause a so-called "equilibrium" state, in which the effect of gradient and WD on weight norm cancel out (see Fig. I(a)..
works (Van Laarhoven 2017:
It has been long time since the concept of equilibrium first proposed (Van Laarhoven 2017) while either theoretical justification or experimental evidence had still been lacking until recently. Recent works (Li al.) 20200] Wan al. (021) theoretically justify the existence of equilibrium in both theoretical and empirical aspects, and characterize the underlying mechanism that yields equilibrium, named as "Spherical Motion Dynamics". In Wan al. (2021) the authors further show
SMD exists in a wide range of computer vision tasks, including ImageNet Deng et al. (2009) and MSCOCO (Lin et al. 2014). More detailed review can be seen in appendix A.

---

Under review as a conference paper at ICLR 2023
Though the existence of SMD has been confirmed both theoretically and empiri- cally, as well some of its characteris- ties, we notice that so far no previous work has theoretically justified how SMD can affect the evolution of the loss of nor- malized models. Although some attempts have been made in Li et al. (2020): Wan al. (202I) to explore the role of SMD in the training by conjectures and empiri- cal studies, they still lack theoretical justi- fication their findings. In hindsight, the main challenge to theoretically analyze the effect of SMD is that SMD comes from the joint effect of normalization and WD which significantly distort the loss landscape (see Figure [(b),, and thus dramatically weaken some commonly used assumptions such as (locally) convexity, Lipschitz continuity, etc. Explor- ing the optimization trajectory such distorted loss landscape very challenging, much less that taking in addition SMD into account in the consideration.
Wn
nn
Aw
An
Wn-
(a)
(b)
Figure 1: (a) Illustration of Spherical Motion Dynam-
ics; (b) Loss landscape of Rayleigh Quotient with
WD (I2 regularization): x 2y²² (x +y (x² +z²)
In this paper, as the first significant attempt to overcome the challenge on studying the effect of SMD towards evolution of the loss, we propose a simple yet representative normalized model, and theoretically analyze how SMD influences the optimization trajectory. We adopt the SDE framework of Li et al. (020) to derive the analytical results on the evolution of NRQ, and concepts of Wan et al.) (2021 to interpret the theoretical results we obtain in this paper. Our contributions are
We design simple normalized model, named as Noisy Rayleigh Quotient (NRQ). NRQ possesses all the necessary properties to induce SMD, consistent with those in real neural networks. NRQ contributes powerful tool for analyzing how normalization affects first- order optimization algorithms;
We derive the analytical results on the limiting dynamics and the stationary distribution of NRQ. Our results show the influence of SMD mainly reflected on how angular update (AU), a crucial feature of SMD, affects the convergence rate and limiting risk. We discuss the influence of AU within equilibrium and beyond equilibrium respectively, figuring out the association between the evolution of AU and the evolution of the optimization trajectory in NRQ:
We show that the insights drawn from the theoretical results on NRQ can adequately in- terpret typical observations in deep learning experiments. Specifically, confirm the role of learning rate and WD is equivalent to that of scale-invariant weight in SGD. We show the Gaussian type initialization strategy can affect the training process only because it can change the evolution of AU at the beginning. We also confirm that under certain condi- tion, SMD may induce "escape" behavior of optimization trajectory, resulting "pseudo overfitting" phenomenon in practice.
2 NOISY RAYLEIGH QUOTIENT
2.1 PROBLEM SET UP
We use Rayleigh Quotient Horn & Johnson (012 as the objective function, defined as
((X) X AX
(1)
2XTX
where X E (O). A E RPxP is positive semi-definite. Based on its form, Rayleigh Quotient is equivalent to quadratic function using weight normalization (Salimans & Kingma 2016).
Now considering the following optimization problem
min X XERP\(()) (0)
L(X),
(2)
2

---

Under review as a conference paper at ICLR 2023
Obviously the solutions to equation 2 are X favla E R -S-1 Ay is the smallest eigen value of A}. Consider solving Eq equation 2 by Stochastic Gradient Descent (SGD) with constant learning rate (LR) 17 0 and Weight Decay (WD), the update rule at step is
Xn+1 = X 79,, - X
(3)
where Xn the weight decay part with a positive factor X; 9, is the stochastic gradient of equation M at step Inspired by (Zhang et al. (2019), the gradient noise is constructed as Gaussian noise to simulate the "mini-batee" training". Specifically, 9n. is constructed as
Pr(E) En PAAX n + |Xal|2
gn ||x,||2
where X, A X, 2: P. _ (I X, Σ E TRexP a positive definite matrix; En (0, T). Then we have
Egg vx),,,,, Cov(gn)
PM
(5)
X
In defined as equation 4 simulate the mini-batch stochastic gradient of scale-invariant loss because
(g.. X.) O, In -m
Vk N=00
which are necessary conditions to let SMD occur during the evolution SGD (Li et al. 2020 Wan al. 2021). We call the process Noisy Rayleigh Quotient (NRQ) which optimizes the Rayleigh Quotient equation 3 Jusing stochastic gradient Eq equation 1 as
Remark 1. The form of stochastic gradient of NRQ can be regarded as the normalized form of Noisy Quadratic Model (NQM) (Zhang et al. 2 in which the objective function
1 x" AX
(
2
and the stochastic gradient defined as
gn = AX,, + +/122
(8)
But the dynamics of NQM and NRQ are quite different: NQM is basically convex model and has only one optimal solution O, while NRQ is nonconvex problem and has an infinite number of solutions (X'*,, thus making much more difficult to analyze comparing with NQM.
2.2 APPROXIMATE SGD AS STOCHASTIC DIFFERENTIAL. EQUATION
Though the thorough amalysis the characteristics of SMD is established on the discrete form (Wan cct al. 2021). directly analyzing evolution dynamics of equation taking SMD into account in discrete form is still too complex. On the other hand, we can tackle the problem using the SDE approximation introduced in Li et al. (2020). which has established the continuous form of SMD.
Using SDE approximation, the evolution dynamics of equation 3 can be approximated by
P.AX. + AX,) 1) P/ ($)2222 dB, dX, = - X2,12
1
X,
where B, is a p-dimensional Brownian motion. Here we follow the form of SDE used in Li et al. (2000) instead of the commonly used form proposed in Li et al. (2017) by extracting a LR factor from the differential time dt. The extracted factor is useful in connecting the characteristics of SMD in discrete setting and continuous setting.
Due to the scale-invariant property, Xella cannot influence the Rayleigh Quotient L(X,) all, the intrinsic domain of NRQ is a unit sphere SP-1 (Li et al. 2020). But |X may affect the evolution dynamics of NRQ since it is involved in the system equation | To decouple the evolution of X, on its intrinsic domain (i.e. the evolution of X,), and the evolution of |X according Li al.
12220 let M, & || then equation
9 can be rewritten as the following two SDEs:
dx, -
"1 Mr
P,A,,
1/2
" P.SiBB
(10)
2M dM, -1- + Tr Tr(Pi))ddtt
M,
n² M,
(11)
3

---

Under review as a conference paper at ICLR 2023
Note the diffusion part is missing in Eq equation I1 of Eq equation 11 computed as
so it is possible to derive the explicit solution
4 My + 21/ ( c--aaat- Tr(P, E)dr.
M2 e 4Ant
(12)
2.3 CHARACTERISTICS OF SMD IN NRQ
Previous works (Van Laarhoven 2017 Chiley et al. 2019 Kunin et al., 2021 Li ei al. 2020p usually regard the convergence of weight norm as the sign of equilibrium state in SMD. However, Wan ctt al. (222)) argues that cquilibrium of SMD practice actually a dynamic state, where the convergence of weight norm may not hold when the variance of gradient noise on intrinsic domain varies dramatically during the whole training process. Notwithstanding, Wann et al. (2021) reveals another essential characteristic of equilibrium regardless of the convergence of the norm: the AU, defined as An = 2(X,, Xn+1). When equilibrium of SMD is reached, AU will satisfy EA, v22m In NRQ, the (discrete) AU can be computed by
9n. X
1 117
A, 4Xx,s X arctann
(13)
|x,
Then have
EA = || x
112 Tr(P,
(14)
|x
Though AU has specific geometric meaning in discrete form, as it represents the geodesic distance between X,, and Xx+1 unit sphere SP-1, its definition cannot be applied directly in continuous setting. To connect the concept of SMD in discrete setting and continuous setting, a continuous version of AU in NRQ is defined as
Definition (AU). X X.112 is differentiable on it. too), then AU at is defined as
E, X x,1
A
(15)
The definition of AU in continuous setting is inspired from the concept of "aagularr velocity" used in Kunin et al. (2021), in which the author formulated the equilibrium SMD using gradient flow.
Remark 2.
This definition relies on the fact that E X, - x,112 is differentiable on t. By the definition we can derive the following properties of AU in NRQ:
Lemma 1. In the evolution of equation
10 equation
11 we have A?
Tr
IFTr(P,E) is Mf
constant, then lim A = 22Ar..
The proof is in Appendix B. Comparing with equation 14 and lemmal 1 the theoretical value EA in discrete form is similar to its continuous form except for additional term V V in
the top of fraction, denoting the full gradient norm. Note when X,, close its optimal point,
this term is usually close to zero, hence can be ignored comparing with the magnitude of gradient noise Tr( P,, D.). This is called noisy dominated regime (Zhang al. 2019 Smith et al. 2020), which commonly holds in practice especially in large-scale dataset tasks (Smith et al. [020]] Wan al. 202T). and happen to be the case where discretization error can be controlled (Li al. 2021).
Besides, the limit of A, is /2xn,, exactly same as the theoretical value of AU in discrete form.
In summary, SMD in continuous form is fundamentally equivalent to SMD in the discrete form in noisy dominated regime, where they share the same characteristics on AU. In the following context, we adopt the unifying concept of SMD and AU, without distinguishing the discrete and continuous form.
3 ANALYTICAL RESULTS ON EVOLUTION OF NRQ
First of all, the following two assumptions are introduced to simplify the derivation and highlight the insights of the analytical results:
4

---

Under review as a conference paper at ICLR 2023
Assumption 1. A is diagonal matrix with diagonal elements in ascending order, i.. A diag((1. .2.....ap. a1 02 I 03 S ap.
Assumption 2. 3o > 0, I
In assumption 1 is diagonalized simplify derivation, same as Zhang al. (2019) does in the quadratic model. We further assume d1 < d2 to ensure NRQ has at most solutions fei, where el" 1, e 0,2 I i is 1. Assumption 2 is used ensure Tr (P.E) is constant during the whole process. This constant variance gradient noise assumption are also used in Zhu et al. (2019): al. (2200).
Note even under assumption NRQ still has two different global optimal solutions the on 85-1, so directly analyzing the convergence behavior by distance the optimal points inap- propriate. Therefore, to properly track the optimization trajectory, we analyze the evolution of ft A x2 (X where x denotes the first element of X. Note that is ideal index to reflect the optimization trajectory because J+ = and only if X = tee- Besides, fo can (roughly) bound the evolution of the loss L by
a1 (a2 a1)( fi) S T. S 01 + (ap - a- )1 ft)-
Using Itô Lemma, the SDE of fe can be derived from equation T0 and equation
written
dft = 7 T, a1)
02
2n1
Mc fet
- [-ppe]]dt+ +
fe((1 ft)dB,,
(16)
M2
Mr
(p (0-1)0²m
(p 1)02n
M3 =e Adnt 4M
(17)
2A
2\
Remark is possible to directly explore the evolution of the loss Lg by deriving the SDE of L using Itô's lemma. But some terms SDE of L, is hard to handle comparing with the SDE of fc.
Now we can define the risk of NRQ as r & 1 - Eft, our first theorem depicts the convergence behavior of NRQ by giving the bounds of the risk;
Theorem 1. The solution of Eqs equation 16 and equation 17 satisfies
T1 (1(t) - so / A e3e
(18)
where
G1(1) & [/' (a, 01)M
vp To -/ A, + P A2]
(19)
p --1
Furthermore, given & E (0, 1), define E(t) = P(fe < {) as the tail probability of ft dynamics. Then for any t 2 0, we have
< e 01(0) fot f (a2
f:(102 (122-a))nee (1)ne(()) A, + +,?) 6.
-2))--0.d1) d7].
(20)
vp-
where
G1(1) A [" (( (a
an)ng (020--1))
+ P A p
A,
(21)
vp 10
Proof is Appendix C. Theorem D implies the evolution of risk are mostly determined by the evolution of AU. Note the lower bound Eq equation relies on & and e(t). To make the lower bound tighter, ideally should be close to while =(f) should close to O. will discuss ç and E(t) in details later.
3.1 EQUILIBRIUM STATE OF SMD
Though an analytical result is shown in Theorem T the global picture of the dynamics is still not clear, since the evolution of is associated with the evolution of AU Ar. Fortunately, it has been known that equilibrium of SMD must be reached, which A, 22 regardless the evolution of X, Hence, we can directly explore the evolution of r, in equilibrium, as the following corollary shows:
5

---

Under review as a conference paper at ICLR 2023
Corollary 1 (Equilibrium state dynamics). Assume Mo / /162-1002 Xn < 1, and Es > 0,
oim ett) S in Theorem / then 3C > 0, such that
2 + (1 fo re 2 SnSFtete
(22)
where
2
Up 41
v 22n + O(xn). की = 5(ap a1)
22 O(Ar),
(23)
vp- 10
vp 10
vp
lo
22 + O(vi) 8" Vp-lo
E( a1) 2 O(Ar).
(24)
Up 01
Proof is in Appendix D.I. Corollary shows shows that in equilibrium state of SMD, when < F", NRQ still converges linear rate regime, where the convergence rate is (roughly) proportional to the AU by equation 23 which is only determined by LR and WD factor X. However, the limiting risk is also (roughly) proportional to the AU by equation 24 Thus, a trade-off exists between the convergence rate and limiting risk when tuning AU: large AU make the loss decrease more quickly at the beginning. but will enlarge the limiting risk in the end, resulting a larger steady loss (see Figure 2 (a)- This explain why decreasing LR strategy always necessary to obtain the best performance when training models in practice.
Remark 4. Such trade-off between convergence rate and limiting risk also exists in the convergence behavior of NQM (Zhang et al., 2019). Zhang et al. (2019) claims the trade-off in NQM can be adjusted by tuning LR or gradient noise; while in NRQ. the trade-off can not only be adjusted by LR or gradient noise, but also WD factor. Besides, the association between AU and the convergence rate/limiting risk is consistent with the conjectures about the relation between AU and dynamics of normalized DNN in Wan et al. (202T), in which the authors suppose AU is correlated with the steady loss when training normalized DNN.
Stationary distribution of ft Corollary [ only presents a bound for the risk. With additional assumptions, the stationary distribution ft and limiting risk r, can be explicitly derived using Fokker Planck equation.
Theorem 2. Assume the spectrum of A takes only distinct real value ai = < an =0 ap. Denote the stationary distribution of ft by pf(.) Then
p.(f) x 2nd f à (1- fe [0,1]
(25)
where 6
VPT In addition, the limit of risk T4. exists and is given by 2
vp To
r- A lim rt = t
22 + ol 2An);
(26)
an
Moreover, there exists p. C' > O, such that for any & E (0, 1), the tail probability e(t) A P(f E) can be governed by
|e(() -0.0 Ce
(27)
in which e, is the stationary tail probability c. 56 Pf(f)
Proof is in Appendix D.2. Eq equation 26 supports our insights drawn from Theorem 1 that the limiting risk should be proportional to the theoretical value of AU; Besides, equation 27 implies that it is reasonable assume is close while the upper limit of e(t) is close to state in Theorem and Corollary
3.2 BEYOND EQUILIBRIUM OF SMD
We have presented a detailed analysis on the evolution of the NRQ in equilibrium of SMD, showing that the convergence rate and limiting risk mainly controlled by AU. Based the insights drawn from the equilibrium case, we can infer how evolution of AU dominates the evolution of NRQ beyond equilibrium.

---

Under review as a conference paper at ICLR 2023
(a) Risk
(b) Loss Lt
(c) AU A
(d) Weight norm
X
2
(c) Risk Ftt
(f) Loss L.
g AU A
(h) Weight norm
Xella
ii) Risk
ii) Loss Lt
(k) AU A. (l) Weight norm ||x
Figure 2: Experiments of NRQ. We exhibit the averaged results of 100 trials. (a)-(d): Evolution of NRQ equilibrium state; (e)-(h): Escape behavior of NRQ after LR decay. LR is divided by 10 when 10*, "Rescale" means X, is divided by (10) when learning shrunk; (i)-(1:: Evolution of NRQ with different and LR, WD factor and initialized standard deviation, denoted by (1,, a). Blue lines are masked by yellow lines in (i), (i), (k) since they have exactly same evolution.
"Escape" by the autonomous increase of AU The following corollary shows the evolution of A. can lead to an "escape" behavior of optimization trajectory, resulting in temporary "decreasing, then increasing" risk:
Corollary 2 (A sufficient condition of "escaping" behavior). Given 17, X, if the following conditions hold: 1) E > 0, VI > e1) < < 2) So = - :): Mo >
[-'=)(aa--1)) Then hE > 0,
(p-1)o"n
rT ro lim rr.
(28)
t-000
Proof is in Appendix E. equation 28 indicates a kind of "escape" behavior, because it implies that the trajectory of X, will first approach an optimal point X at the beginning, and then depart from X². Intuitively, this "escape" behavior is caused by increasing AU (which is also the main idea of the proof for Corollary [):: the initial weight norm /Mo is sufficiently large, so A, is relatively smaller than /2\r, at the beginning, which allows the risk re to reduce below the inferior of limiting risk given v 22n for while. But when equilibrium reached and AU increases v 2207, will force the risk to go back to its limit value. See the blue lines in Figure (c)-
Even though Corollary - only gives a sufficient condition for the "escape" behavior in NRQ, such "escape" phenomenon can be seen in real data experiments in practice. Wan et al. (2021) exhibits so-called "pseudo overfitting" phenomenon observed in CIFAR 10 experiments with commonly used settings. They speculate "pseudo overfitting" is caused by temporarily increasing AU after LR decay based on empirical observations. Corollary 2 offers strong theoretical evidence for their conjecture, showing increasing AU indeed can lead to "escape" behavior under specific conditions. We also apply "rescale" strategy proposed in Wan et al. (202T) on NRQ. in which when LR divided by k weight norm is divided by k1/1, The "rescale" strategy can indeed eliminate the "escape" behavior (see Figure 2 (c)-(h), red lines).
Initialized value of weight norm The evolution of AU can provide new interpretations on how initialization strategy affect the training of normalized model.
7

---

Under review as a conference paper at ICLR 2023
The weights of neural network are usually initialized as Gaussian ((00, (1), where o need to be carefully tuned. In mean field theory and NTK theory, standard derivation of Gaussian initializing strategy is crucial in obtaining good performance. Experiments real datasets seem support these theorems, where Gaussian initializing strategies with carefully tuned ā such as Kaiming He et al.] (2015) or Xavier Glorot & Bengio (2010) indeed outperform the naive Gaussian initializing strategy. However, when initializing normalized model, ā only influences the initialized value of
weight norm according the large number theorem: ||Xoll2 (x=( = po². The follow- ing corollary implies ē are not so crucial for normalized model:
+ Corollary 3. vk > 0, if Xo is mulliplied by k, enlarge 1, A b k2, E times respectively, T remains unchanged.
Proof is in Appendix E.2. Corollary 3 shows no matter how to set o, as long as LR 17 and WD factor are adjusted accordingly, the evolution dynamics of NRQ does not change at all. Because the adjustment in Corollary 3 can remain the evolution equation 10 by maintaining the evolution of A
(Comparing blue and yellow lines in Figure 2 (i)-()).
Furthermore, combining equation 17 and equation 11 we can interpret why initialization affect the evolution of NRQ: when X, 17 are given, the initialized value Mo can change the evolution of AU A, by changing equation 17 resulting in different training curve at beginning. But their limiting risk remains unchanged, since the theoretical value of AU does not change (Comparing blue and red lines in Figure 2 (i)- same phenomenon occurs when Mu, An are fixed, but X, 17 change.
Though in NRQ. the conclusion that "same limiting AU will lead to similar limiting risk" is true, same conclusion does not necessarily hold on real data experiments. The two local minima of Rayleigh Quotient have exactly same geometric characteristics, but in real data experiments, the loss landscape may have multiple local minima with different geometric characteristics. Even though the theoretical value of AU is fixed, different evolution of aus may make the optimization trajectory get close to different local minima, resulting in different final performance. This is the reason why in practice, with fixed LR, WD factor, and ă in Gaussian initialization still may influence the performance of neural network.
4 REAL DATA EXPERIMENTS
Aside from NRQ experiments, we also conduct experiments on CIFAR1O (Krizhessky et al. 2209), and ImageNet (Deng et al. 2009) respectively to verify the insights drawn from NRQ. We use pure SGD without momentum in all real data experiments to eliminate the possible effect of momentum, though Wan et al. (202)) claim SGD with momentum has similar SMD mechanism. In CIFARIO experiments, we adopt Resnet18 (He et al.) 2015) as the baseline model; total epochs is 200; LR is divided by at epoch 60, 120, 160; Batch size is 256. In ImageNet1000 (Deng et al. 2009) experiments, most settings follow Goyal al. (2017), except LR is initialized as 1, momentum is O. Smith et al. (2020) shows using pure SGD with larger LR can obtain similar performance as the standard SGDM setting.
Our experiments' results (Figure show the insights from NRQ also hold in real data experiments: in eifar experiments, when y fixed, AU in equilibrium of SMD remains unchanged, so do the steady loss and Accuracy (Figure 3 (d)- But different LR and WD factor can affect the evolution beginning. Wan et al. (2021) shows similar observations in ImageNet experiment: Figure exhibit the "pseudo overfitting" phenomenon shortly after the first LR decay (epoch 60). Rescaling strategy can avoid such "pseudo overfitting" by eliminating the increasing AU phenomenon after LR decay; From Figure (i) when initialized weight is enlarged by 10, the AU is smaller at the beginning, hence the training loss (test accuracy) decreases slower (increases quicker). When the LR and WD factor are adjusted according to corollary 3 the evolution of AU remains unchanged, and so do the evolution of training loss/test accuracy. ImageNet experiments have similar phenomenon (Figure ( (d)-
5 CONCLUSION
In this paper, we propose a simple yet representative normalized quadratic model, named as Noisy Rayleigh Quotient (NRQ), to study the effect of SMD on the evolution of SGD with WD. Our
8

---

Under review as a conference paper at ICLR 2023
Training loss
(b) Test Accuracy
(c) AU
(d) Weight Norm
Training loss
(f) Test Accuracy
g
Weight Norm
(i) Training Loss
(j) Test Accuracy
(k) AU
(I) Weight Norm
Figure 3: Resnet18 on CIFARIO, we exhibit the averaged results of 10 trials. (a)-(d): Training curves with different LR and WD factor, denoted (17, A); (e)-(h): pseudo overfitting in CIFAR10 experiments. LR is 0.5, WD factor is 10 "rescale" means all weights is divided by 5 at epoch 60; (i)-(I): Training with different LR, WD factors, and initialized standard deviation in convolution layer, denoted by (1), X, x, v3 denotes Kaiming Init (He et al. 2015).
Exart.f Exurt,5
sma $5.
(a) Training loss
(b) Test Accuracy
(c) AU
(d) Weight norm
Figure 4: Resnet50 on Imagenet Training with different LR, WD factors, and initialized standard
deviation convolution layer, denoted by (n, X, a). V2 denotes Kaiming Initialization (He al, 2015).
theoretical results show SMD influences the evolution SGD by controlling AU, and AU can dominate the convergence rate as well limiting risk of NRQ. Our real data experiments show the insights drawn from NRQ are consistent with empirical observations. We believe our theorems can deepen our understandings on the underlying mechanism of deep neural networks.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Sanjeev Arora, Zhiyuan Li, and Kaifeng Lyu. Theoretical analysis of auto rate-tuning by batch normalization. In International Conference on Learning Representations, 2018.
Nils Bjorck, Carla P Gomes, Bart Selman, and Kilian Q Weinberger. Understanding batch normal- ization. Advances in Neural Information Processing Systems, 31:7694-7705 2018.
Yongqiang Cai, Qianxiao Li, and Zuowei Shen. A quantitative analysis of the effect of batch nor- malization on gradient descent. In International Conference on Machine Learning, pp. 882-890.
PMLR. 2019.
Vitaliy Chiley, Ilya Sharapov, Atli Kosson, Urs Koster, Ryan Reece, Sofia Samaniego de la Fuente, Vishal Subbiah, and Michael James. Online normalization for training neural networks. Advances in Neural Information Processing Systems, 32:8433 8443, 2019.
Soham De and Sam Smith. Batch normalization biases residual blocks towards the identity function in deep networks. Advances in Neural Information Processing Systems, 33, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp.
leee, 2009.
Yonatan Dukler, Quanquan Gu, and Guido Montúfar. Optimization theory for relu neural networks trained with normalization layers. In International conference on machine learning, pp. 2751- 2760. PMLR, 2020.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference artificial intelligence and statistics, pp. 249-256. JMLR Workshop and Conference Proceedings, 2010.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint orXiv:/706.0220 7066026677 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Elad Hoffer, Ron Banner, Itay Golan, and Daniel Soudry. Norm matters: efficient and accurate normalization schemes in deep networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 2164 2174, 2018.
Roger A Horn and Charles R Johnson. Matrix analysis. Cambridge university press, 2012.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448 456. PMLR, 2015.
Jonas Kohler, Hadi Daneshmand, Aurelien Lucchi, Thomas Hofmann, Ming Zhou, and Klaus Neymeyr. Exponential convergence rates for batch normalization: The power of length- diiggtiiii decoupling in non-convex optimization. In The 22nd International Conference on Artificial Intel- ligence and Statistics, pp. 806-815. PMLR, 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Daniel Kunin, Javier Sagastuy-Brent Surya Ganguli, Daniel LK Yamins, and Hidenori Tanaka. Neural mechanics: Symmetry and broken conservation laws in deep learning dynamics. In Inter- national Conference on Learning Representations, 2021.
Qianxiao Li, Cheng Tai, and E Weinan. Stochastic modified equations and adaptive stochastic gradient algorithms. In International Conference on Machine Learning, pp. 2101-2110. PMLR, 2017.
10

---

Under review as a conference paper at ICLR 2023
Zhiyuan Li and Sanjeev Arora. An exponential learning rate schedule for deep learning. In Interna- tional Conference on Learning Representations, 2019.
Zhiyuan Li, Kaifeng Lyu, and Sanjeev Arora. Reconciling modern deep learning with traditional optimization analyses: The intrinsic learning rate. Advances in Neural Information Processing Systems, 33, 2020.
Zhiyuan Li. Sadhika Malladi, and Sanjeev Arora. On the validity of modeling SGD with stochas-
tic differential equations (SDEs). In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https: //openreview. net./forum d=gomddy_goooyoOOO
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Ekaterina Lobacheva, Maxim Kodryan, Nadezhda Chirkova, Andrey Malinin, and Dmitry Vetrov. On the periodic behavior of neural network training with batch normalization and weight decay. Advances in Neural Information Processing Systems, 34, 2021.
Tim Salimans and Durk P Kingma. Weight normalization: simple reparameterization accelerate training of deep neural networks. Advances in neural information processing systems, 29:901- 909, 2016.
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch nor- malization help optimization? In Proceedings of the 32nd international conference on neural information processing systems, pp. 2488-2498 2018.
Samuel Smith, Erich Elsen, and Soham De. On the generalization benefit of noise in stochastic gradient descent. In International Conference on Machine Learning, pp. 95589906.. PMLR, 2020.
Cecilia Summers and Michael Dinneen. Four things everyone should know to improve batch normalization. In International Conference on Learning Representations, 2019.
Twan Van Laarhoven. L2 regularization versus batch and weight normalization. arXiv preprint
13xiv:700.055335 2017.
Ruosi Wan, Zhanxing Zhu, Xiangyu Zhang, and Jian Sun. Spherical motion dynamies: Learning dynamics of normalized neural network using sgd and weight decay. Advances in Neural Infor- mation Processing Systems, 34, 2021.
Xiaoxia Wu, Edgar Dobriban, Tongzheng Ren, Shanshan Wu, Zhiyuan Li, Suriya Gunasekar, Rachel Ward, and Qiang Liu. Implicit regularization and convergence for weight normalization. Ad- vances in Neural Information Processing Systems, 33, 2020.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pp. 3-19, 2018.
Greg Yang. Jeffrey Pennington, Vinay Rao, Jascha Sohl-Dickstein, and Samuel S Schoenholz. A mean field theory of batch normalization. In International Conference on Learning Representa- tions, 2018.
Guodong Zhang, Lala Li, Zachary Nado, James Martens, Sushant Sachdeva, George Dahl, Chris Shallue, and Roger B Grosse. Which algorithmic choices matter which batch sizes? insights from a noisy quadratic model. Advances in neural information processing systems, 32:8196-8222 8207, 2019.
Zhanxing Zhu, Jingfeng Wu, Bing Yu, Lei Wu, and Jinwen Ma. The anisotropic noise in stochastic gradient descent: Its behavior of escaping from sharp minima and regularization effects. 2019.
A APPENDIX
You may include other additional sections here.


Reviews:
[{"review_id": "fCREGyV47BE", "reviewer": "Reviewer_ucmd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies diffusion model of Oja's algorithm for leading eigenvector problem. When the noise is Gaussian and isotropic, the matrix is diagonal, then author establish diffusion convergence to equilibrium state. They show that the algorithm exhibits a escaping property in they laboratory setting which is similar to SGD dynamics on neural networks with batch normalization. ", "summary_of_the_review": "I recommend authors present their theoretical results in context of stochastic power iteration after removing the restrictive assumptions. Going from power iteration with restrictive assumptions to the standard settings of training neural nets is not sound and convincing. ", "strength_and_weaknesses": "**Strength** \n\nStudying Oja's flow on leading eigenvalue problems is an important topic in stochastic approximation. Despite recent progress, there are many open-problems in stochastic power iteration methods. This paper studies a continuous-time model of Oja's flow that may enable simplifying the complexity of analyzing Oja's flow. \n\n**Weakness** \n- *Motivation.* In my opinion, there is a gap between optimization with batch normalization in neural nets and the eigenvalue problem. It is better to formulate the problem in a standard theoretical context, such as stochastic power iteration. This setting is standard and well-studied in the literature, which allows authors to compare. \n- *Restrictive assumptions.* Assumptions are rather strong and are not validated or discussed in the paper. \n  - Lemma 1 assumes that a random process is constant over time.  \n  - Assumption 1 enforces the noise of stochastic gradient is isotropic. \n  - Assumption 2 limits the analysis to find the leading eigenvector of a diagonal matrix. \n- *Generalization.* For a very limited toy example of an eigenvalue problem, the paper extrapolates their findings on *the escape phenomenon* to the setting of learning residual networks with 18 layers for image classification. I found this unnecessary. In my opinion, the settings of stochastic power methods are well-established.  \n- *Soundness.* I am not sure about some of the statements in the paper. \n  - Equation 11 is a random process (due $P_t$) while it is assumed to be deterministic in the integration of Eq. 12. \n  - Why does corollary 1. not need the same assumption as Lemma 1? \n- *Presentation.* \n  - Notation $\\epsilon(t)$ used in Thm. 1, Corollary 1 is not defined. \n  - Would be great to define escape phenomena mathematically and then characterize the required time for escape. I did not grasp the notion of escape. \n \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "d_JQk0FHvx", "reviewer": "Reviewer_UaUq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Deep neural networks typically contain normalization layers like layer norm or batch norm, which causes scale invariance in the network parametrization. This paper studies learning dynamics in this setting. Firstly, the authors introduce a simple optimization problem dubbed Noisy Rayleigh Quotient (NRQ) which exhibits this scale-invariant property. The authors then analyze the optimization behavior of this model under an SDE approximation for SGD. Under diagonal assumptions and constant noise, the authors give analytical expressions for the dynamics and limiting distribution of NRQ. At last, the authors verify that their theoretical results match experimental data.\n\n", "summary_of_the_review": "The authors introduce a natural model (NRQ) for studying the optimization of a normalized model and give theoretical results under strong but natural assumptions. The paper clearly shows how the theoretical results relate to empirical work, but the language and narrow topic make the paper somewhat inaccesible to a wider audience. \n\n", "strength_and_weaknesses": "Strengths:\n\n- Normalized models are an interesting and relevant topic for theoretical work.\n- The paper is well written.\n- The empirical verification is throughout with e.g. many samples used to obtain error bars.\n\nWeaknesses:\n- The assumptions of diagonal A and diagonal and constant noise are rather strong.\n- The paper is probably somewhat inaccessible to the wider ICLR audience. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "aeOedMDbUN", "reviewer": "Reviewer_mwkM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a theoretical analysis for the Stochastic Gradient Descent optimizer for normalized deep neural networks including weight decay. In particular, they study the behavior of the loss during training, taking into account that for such models the optimization converges to an equilibrium related to the weights norm. For the theoretical analysis a simple optimization problem is used. Empirical demonstrations verify the analysis.", "summary_of_the_review": "Questions:\n1. In my opinion some basic concepts should be explained more thoroughly. For example, you can use a 2-dim optimization problem (Eq. 1) together with an appropriate figure, in order to explain clearly what is the Spherical Motion Dynamics (SMD), what is the Angular Update (AU), what is the equilibrium state, etc. In general, I believe that the paper should be standalone.\n2. The scale-invariant model is the same as the normalized model? Can you give some examples for each of them.\n3. Let's consider a 2-dim optimization problem (Eq. 1). The optimization trajectory of a scale-invariant model (under SGD) behaves as a motion on the sphere. Following your example, let assume that the minimizer is the direction $e_1=\\pm [1,0]$. When the optimizer hits this valley of minima, then the sphere motion implies that the optimizer stays in this valey following an up-down trajectory?\n4. When we include the weight decay in the problem above, then the SMD applies. What is the implication? When the optimizer hits the valley of minima, then the trajectory moves closer to 0? And what the \"equilibrium\" state means? Does it mean that first the correct direction is found, then the norm of the weights is minimized (close to 0 for your problem), and then the optimizer due to the noise moves up-down?\n5. In Sec 2.1 a deterministic optimization problem is proposed together with a stochastic optimization procedure. I wonder what is the purpose of Remark 1? Does it imply that NRQ is harder than NQM? I think that NRQ without the unit norm constraint is not an equivalent problem.\n6. In Sec. 2.2 the paragraph before Eq. 10 is a bit confusing. Also, why Eq. 9 can be rewritten as Eq. 10 - 11 and get the associated solution?\n7. The sections 2 and 3 are quite confusing and unclear as well. In most cases the intuition is missing, there are a lot of technical derivations with no clear goals, and in general I think that the presentation is not coherent and accessible.\n\nIn general, the problem that the paper aims to study is very interesting, and perhaps the associated technical content. However, I think that the current version of the manuscript probably does not reflect its actual contribution and merits. In my opinion an improvement mainly on the clarity and the accessibility of the paper is necessary.\n", "strength_and_weaknesses": "I think that understanding the optimization dynamics for any model class is a very interesting problem. However, in my opinion the actual problem that the authors aim to analyze is not clearly defined and is not motivated well in the current version of the paper. As a non-expert in the field I believe that the related work is properly cited, but on the other hand I find the whole story of the paper a bit incoherent. The authors build on previous work that is assumed to be known, which I think makes the paper rather confusing and non accessible. The technical content seems to be correct, but I have not checked it in details. Also, a lot of simplifications and assumptions are used such that to develop the theoretical results, which might be a bit too optimistic/unrealistic compared to the actual scenarios. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_EB6ykWUeW", "reviewer": "Reviewer_y88R", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the dynamics of SGD with WD on a particular scale invariant loss function, the so-called Noisy Rayleigh Quotient (NRQ). The main contribution of the paper is the convergence analysis for NRQ under continuous approximation.\n", "summary_of_the_review": "Though the convergence analysis on NRQ model is a solid contribution, I think the current story this paper tries to deliver could be misleading to the community. Besides, the interpretations are more or less already known from previous works. Thus I think this paper does not meet the bar of ICLR.", "strength_and_weaknesses": "##Pros:\n\n1. Scale invariance in modern deep netowrks has been an hot and important topic in deep learning theory, due to the significance of various normalization layers in optimization. However, the community lack of good understanding of the training behavior of scale invariant models under SGD. Though there exists some theoretical analysis and hypothesis based experiments for general deep learning loss, it is not clear how the dynamics look like, e.g. the convergence rate of training loss, even for any simple model, because scale invariance also changes the loss landscape drastically, making it inherently non-convex.  This paper provides the first convergence analysis for concrete scale invariant loss functions, that is, on the NRQ model. \n\n2. Though anguler update (AU) is implicitly defined in the previous papers, this paper gives the first mathematical rigorous defition of AU and prove that it is equal to the $\\sqrt{2\\eta\\lambda}$.\n\n3. This paper also makes some analysis or intuitions from previous works more concrete, at least in the setting of NRQ. For example, as pointed out in [Li et al.,20], when decreasing learnign rate $\\eta$, the training loss can first decrease and then increase, due to the change of the parameter norm. \n\n\n##Cons:\n\n1. Though the theoretical convergence result is solid, but the interpretation and some phrasing are confusing. The paper is written in the way that AU is the cause of various unconventional optimization behavior of the scale invariant models. But in fact, AU in this paper, espceially under the assumption that trace of covariance is constant on unit norm ball, it merely a function of LR $\\eta$, WD $\\lambda$ and time $t$. In the end of page 6, the authors wrote that \"showing that the convergence rate and limiting risk are mainly controlled by AU\". I do not see any casual relationship between AU and convergence rate or limiting risk. They are all just consequences of SGD or SDE on a scale invariant loss. Thinking convergence rate or limiting risk as functions of AU doesn't seem to bring any conceptual convenience, nor convenience in the mathematical proof. For example in the proof Theorem C.0.3 in appendix, AU is not used at all. Even for the main theorem (Theorem 1), which has AU in its statement, it only follows from Theorem C.0.3 with some substitution, namely replacing the parameter norms by AU, which are equivalent up to simple transformation. Given that such equivlance relies on the assumption that trace of corvariance is constant exactly, it is not easy for me to see the benefit of explaining convergence in terms AU. \n\n2. Corolary 3 is already known. The SGD case is proved by [Lemma 2.4, Li et al.,19]  and the SDE case follows directly from Eq. (9) and (10) in [Li et al.,20]). I will not treat Corolary 3 as a contribution. Moreover, the derivation (even in this paper) does not use the concept of AU at all.\n\n3. The authors mentioned that in the abstract \"theoretically prove SMD can dominate the whole training process via controlling the evolution of angular update (AU), an essential feature of SMD\". It's not clear what \"SMD can dominate the whole process mean\". If it means SDE is a good approximation like we can ignore the impact of full-batch gradient on norm, then I cannot find the proof in the current draft. Also, AU is not a feature of SMD, as it can be defined for any dyanmics. The feature of SMD is that AU converges to a time-independent value under some assumptions.\n\n4. Missing related work: [Wang and wang, 2022] also analyzes convergence rate and gives a more fine-grained characterization for the distribution of the direction of the parameter under more general settings -- their analysis holds for all morse function, including NRQ. \n\n##Minor commens:**\n\n1. Figure 1. missing bracket, should be $(x^2+y^2)/(x^2+y^2) +(x^2+y^2)$.   \n2. In equation (5), no bracket is needed around $\\tilde \\Sigma$.\n3. On top of page 4, \"Eq equation 11\" appeared twice.\n4. In defition 1, it seems \"if\" is missing.\n5. In equation (20), $\\tilde G_\\tau$ is undefined. \n6. I found there is quite a gap between Theorem 1 and Theorem C.0.3. Might worth elaborate the proofs here.\n7. On page 7, \"temporarily increasing AU\", should be \"temporarily decreasing AU\"?\n\n##References:\n\n- Li, Zhiyuan, Kaifeng Lyu, and Sanjeev Arora. \"Reconciling modern deep learning with traditional optimization analyses: The intrinsic learning rate.\" Advances in Neural Information Processing Systems 33 (2020): 14544-14555.\n\n- Li, Zhiyuan, and Sanjeev Arora. \"An exponential learning rate schedule for deep learning.\" arXiv preprint arXiv:1910.07454 (2019).\n\n- Wang, Yi, and Zhiren Wang. \"Three-stage Evolution and Fast Equilibrium for SGD with Non-degerate Critical Points.\" In International Conference on Machine Learning, pp. 23092-23113. PMLR, 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject