Spherical Motion Dynamics: Learning Dynamics of Normalized Neural Network using SGD and Weight Decay
Authors: Ruosi Wan
Zhanxing Zhu
Xiangyu Zhang
Jian Sun
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: normalization
weight decay
stochastic gradient descent
momentum
neural network

Abstract: In this paper, we comprehensively reveal the learning dynamics of normalized neural network using Stochastic Gradient Descent (with momentum) and Weight Decay (WD), named as Spherical Motion Dynamics (SMD). Most related works focus on studying behavior of ``effective learning rate" in ``equilibrium" state, i.e. assuming weight norm remains unchanged. However, their discussion on why this equilibrium can be reached is either absent or less convincing. Our work directly explores the cause of equilibrium, as a special state of SMD. Specifically, 1) we introduce the assumptions that can lead to equilibrium state in SMD, and prove equilibrium can be reached in a linear rate regime under given assumptions; 2) we propose ``angular update" as a substitute for effective learning rate to depict the state of SMD, and derive the theoretical value of angular update in equilibrium state; 3) we verify our assumptions and theoretical results on various large-scale computer vision tasks including ImageNet and MSCOCO with standard settings. Experiment results show our theoretical findings agree well with empirical observations. We also show that the behavior of angular update in SMD can produce interesting effect to the optimization of neural network in practice.

Spherical Motion Dynamics: Learning Dynamics of Normalized Neural Network using SGD and Weight Decay
Ruosi Wan Megvii Technology wanruosi www.ooiiieegivi..ooom
Zhanxing Zhu University of Edinburgh zhanxing zwwded. ac uk zhanxing zhu@pku. edu. en
Xiangyu Zhang
Jian Sun Megvii Technology sunj aangeniiiiii....mmmm com
Megvii Technology zhangxianggyumegrri.ccm
Abstract
this paper, we comprehensively reveal the learning dynamics of normalized neural network using Stochastic Gradient Descent (with momentum) and Weight Decay (WD), named as Spherical Motion Dynamics (SMD). Most related works this topic focus on studying "effective learning rate" using "cquilibrium" sumption, i.c. assuming weight norm has converge to a fixed value. However, their be reached is either absent unjustified. To
discussion why equilibrium clarify the mechanism behind, our work directly explores the cause equilibrium, which should be regarded a special state of SMD. Specifically, 1) we introduce the assumptions that can lead to equilibrium state in SMD, and prove equilibrium substi-
can be reached in linear rate regime; 2) we propose "angular update" tute for effective learning rate to depict the state of SMD, and derive the theoretical assumptions and
value of angular update in equilibrium state; we verify theoretical results on various large-scale computer vision tasks including ImageNet and MSCOCO with standard settings. Experiment results show our thcoretical findings agree well with empirical observations. Furthermore, we provide intuitive interpretations, showing how the behavior of angular update in SMD affects the optimization of neural network, and yields unexpected phenomenon in practice. We believe our findings and theoretical results can deepen our understanding on current training techniques for deep neural network.
Introduction
Normalization techniques (e.g. Batch Normalization (loffe & Szegedy 2015) its variants) one of the most commonly adopted techniques for training deep neural networks (DNN). typical normalization can be formulated as following: consider single unit in neural network, the input is X, the weight of linear layer is w (bias is included in w), then its output is
y (X;w; w. 7: 8) = g( X ((x)
7 + +),
x
where g is nonlinear activation function like ReLU or sigmoid, μl, a are mean and standard deviation computed across specific dimension of X (like Batch Normalization (Ioffe & Szegedy 2015). Layer Normalization Ba et al. (2016), Group Normalization (Wu & He 2018) etc.). B, y are learnable
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

parameters to remedy for the limited range of normalized feature map. Aside from normalizing feature map, Salimans & Kingma (2016) normalizes weight by 12 norm instead:
y X wi; 7: g(
+);
1
where -1 denotes /2 norm of vector. Though formulated in different manners, all normalization techniques mentioned above share an interesting property: scale-invariance
Definition (Scale-invariance), Given loss function L(w), w is scale-invariane w.r.t. & if and only ifvk IR, we have L(w) = L(kw).
By definition of scale-invariance, can directly derive the following properties of scale-invariant weights Lemma]) [
Lemma 1. If w is scale-invariant with respect to L(w), then for all k 0, we have:
HL
(w0), Ow
0
(3)
w==
0C
1 ac
(4)
iw wk=kw:
k
dw w==w
Proof is in appendix. Lemma is also discussed in Hoffer et al. 2018): van Laarhoven (2017) Li &
Arora (2020); Li et al. (2020. it makes the learning dynamics of normalized neural network exhibit
an interesting phenomenon when using Stochastic Gradient Descent (SGD) with Weight Decay (WD):
a typical SGD update rule with WD
Oc
OC
W[+1 = Wi - ry( Ow
+ xZwx) = (1 - m/)2:
(5)
Du
where denotes learning rate, À denotes WD factor. Then dynamics of w; is like phys- ical process a satellite's 's motion around the earth (see illustration in Fig according to Eq. 3 -n0C//w w=w, (green line in Fig 1 is always perpendicular to w,; providing "centrifu- gal effeet" to make [w,+ i Ile larger than ||w; while -ny\w, (red line in Fig D is always in the opposite direction of wis providing "cen- tripetal effect" to make || ||2 smaller than |/wll2. Due to this "tug of war" between "cen- trifugal effect" and "centripetal effect", the norm of weight will change accordingly. Though the norm of scale-invariant weight doesn't influence the loss at all (by definition), but it will affect the optimizing trajectory by changing the scale of gradient (Eq 4 Therefore, to decouple the re- lation between weight norm and gradient norm, a common way (van Laarhoven 2017 Hoffer al. 2017; Chiley et al.) 2019) is to regard the learning dynamics of scale- invariant weight manifold learning process restricted on unit sphere, using "effective learning rate". defined as n/1/w,112 Akin to this reason, in this paper the learning dynamics of scale-invariant weight is viewed as motion a unit sphere, whose motion speed determined by weight norm (in original weight space), gradient norm, and hyper-parametent Therefore, we formally name it as Spherical Motion Dynamcis (SMD) and focus on discussing its unique properties.
W
0L
-We A
aw;
1
Weed
4
0
Figure 1: Illustration of optimization behavior with BN and WD. Angular update A, represents the an- gle between the updated weight w, and its former value W++1-
Concept of "Equilibrium" Since "fug-of-war" in SMD determines the relative sizes of || and || //ww question naturally arises: what will happen || ||2 = [/,]] van Laarhoven (2017) discuss this question first: Chiley et al. (2019) named this state where ww | in SMD as "equilibriam" and discuss its properties; Li Arora (2020) derives lemma about equilibrium in SGD with Momentum (SGDM); Li et al. (2020); Kunin et al. (2021 establish the

---

equilibrium in continuous approximation model. However, most early literature (van Laarhoven 2017; Chiley et al.] 2019 Li & Arora 2020 Kumin et al. 202T) do not discuss a fundamental question: "Does equilibrium really occur in practice?") van Laarhoven (017)) intuitively explains that equilibrium is caused by convergence of optimization. But there exists a contradiction between the interpretation of van Laarhoven (2017) and traditional view of optimization: if equilibrium (I [|w2el22 = ||w++1 l12) is caused by convergence of optimization, then gradient of loss /C/Ow)
should be 0, which makes the balance of "centrifugal effect" and "centripetal effect" impossible to reach (since centripetal effect comes from aC/0w/ u. )- Therefore, van Laarhoven
(017);
Chiley et al. (2019); ; Li & Arora (2000: Kunin et al. (2021) all essentially regard equilibrium as an assumption, and do not justify its existence in neither empirical nor theoretical aspects. "Equilibrium" was not even phenomenon observed in practice, but only a concept until recently.
Recent work (Li et al. 2020) successfully exhibits the existence of equilibrium by formulating SGD in 55 via Stochastic Differential Equation (SDE) in the continuous time limit. They theoretically prove equilibrium can be reached in SDE settings via convergence of ||/]|e. and the convergence time is ((1/(Ar)) (A, denote WD factor and learning rate respectively). However, due to the gap between discrete formulation of SGD and continuous formulation of SDE, theoretical results derived from SDE model can only provide intuitive understanding on empirical observations, some of which are even incorrect (will be discussed latter); Besides, SDE can hardly take SGD with momentum (Polyak 1964) into account, which has become default setting nearly all kinds of deep learning tasks. In summary, a thorough understanding on cause of "equilibrium" and its impact to leamnigg dynamics of normalized neural network is still needed.
In this paper, we comprchensively reveal Spherical Motion Dynamics (SMD). i.c. the learning dynamics of normalized neural network using SGD(M) and weight decay (WD). Our analysis on SMD is directly established on discrete settings. We interpret why equilibrium can be reached in SMD in both theoretical and empirical aspects, and show how SMD) affects the optimization trajectory of neural network. Specifically, our contributions are
We introduce the assumptions which can lead to equilibrium in SMD, and justify their rea- sonableness by sufficient experiments. We also prove under given assumptions, equilibrium can be reached as weight norm approach to its theoretical value in a linear rate regime. Our theorem show equilibrium is a dynamic state in SMD, convergence of weight norm is neither the precondition nor the consequence of equilibrium. This conclusion almost refutes the assumptions of all previous related work (van Laarhoven, 2017 Chiley al. 2019 Li & Arora 2020 Kunin et al. 202I), and the conclusion of Li et al. (2020).
We define novel index, angular update, measure the change of normalized neural network within single iteration. We also derive its theoretical value in equilibrium. Our results show that angular update is better than norm of weight indicate if equilibrium has been reached reached in SMD. Our empirical results further show angular update is an important index reflect the effect of SMD and equilibrium:
We verify our theorems on different computer vision tasks (including one of most challenging datasets ImageNet (Russakovsky et al.) 2015) and MSCOCO (Lin et al.] (014)) with various networks structures. Experiments show the theoretical value of angular update and weight norm agree well with empirical observation. We also show how SMD influence the optimization trajectory of normalized neural network by controlling angular update.
We believe SMD is one of the key reason why learning dynamics of normalized neural network is not consistent with traditional optimization theory (Li et al., (2020). We think is of great potential to take SMD and its equilibrium state into account while studying leaning dynamics of modern normalized neural network or designing novel efficient training strategy.
2 Theoretical results
In this section, we theoretically formulate Spherical Motion Dynamics (SMD) in discrete SGD/SGDM settings, and provide a precise description on "equilibrium" phenomenon; Then we prove equilibrium can be reached in SMD under specific assumptions; Finally new index, whose theoretical value in equilibrium can be derived, is proposed to indicate the state of SMD.
3

---

A new definition need to be given at first. Eq. implies though norm of scale- invee-invariant weights does not affect the output neural network, can influence norm of gradients. thus we define unit gradient in order to decouple weight norm and gradient norm.
Definition 2 (Unil Gradient). If w. + 0, vp = w///w//2, the unit gradient of 0L/Owl yomnn, is @C/Ow w=w..-
According to the definition of unit gradient, the unit gradient norm is independent of weight norm. Specifically, by setting k as 1/|| w//l22 in Eq. the relation among weight norm, gradient and unit gradient is
ac
1
0C
(6)
Dw w
Ow w=
Now, we can depict equilibrium of SGD and SGDM in theorem respectively.
Theorem 1. (Equilibrium in SGD) Assume the loss fanction is (Xx; w) with scale-invarian weight
w, denote gt Dw X.,... 9t gt ||we||2. Consider the update rule of SGD with weight decay,
0C
wi+1 we (gi + Xav))
(7)
where X, n E (0,1). If the following assumptions hold:
1) An < (o(Ar) can be omitted;;
2) Let L2 El wr). AV E RR+, AE MEEN N. Bi L,)p\wod] S V:
3) Vt E N Le satisfies L++11 1r < 4 4VV
4) 31 E R N 19 > 1,, 21 2Am 1 Lr.
Then 3B > 0, VI EN, wf 1/--1'// (2X), we have
Blllwd - (w))) s (1 (1-22m)) 2Xn) + 2V
(8)
Remark 1. The theoretical value of weight norm wi in Theorem is consistent with the magnitude
of weight norm (0( v /\) in equilibrium in van Luarhoven 2017), though van Laarhoven 2017))
assumes the equilibrium has been reached in advance, hence van Laarhoven (2017) cannol provide
the approuching rate and scale of bias/variance. The vanishing term ((1 2Xn) B) in Eq. g is
consistent with the mixing time ((1/(2r)) presented in Li et al. 2020)
The proof can be seen in appendix. Assumption 1 is consistent with commonly used settings in
practice (Goyal et al. 2017: He et al. 2017 Ma et al., 2018); Assumptions 2, 3, 4 all concern unit gradient: unit gradient norm should change smoothly (assumption 3) with bounded variance (assumption 2); besides, unit gradient norm should have a lower bound (assumption 4). We will see these assumptions can easily hold in practice in section 4.1.
Note in theorem 1 assume g 112 has uniform lowere bound I, while El[1g.11 [,)]] has a uniform upper bound V. But these are too strong assumptions and may not hold in practice. We only use them for ease of demonstration and proof. In fact the lower bound of llg and upper bound of E[(1gg 3 L2)² we] can be replaced with smoothly varying functions la and V respectively remove unnecessary assumptions on uniformness. The "trend" to reach equilibirum is actually a local property, which can be inferred from the proof.
Remark 2.
True meaning of "equilibrium": dynamic state of SMD Recall as we demonstrate in introduc- tion, the concept of cquilibrium is originally established on the assumption that weight norm is steady (1ww)]12 ||w++1 1 l12). But the assumption (l|w,|12 = ||wr+1ll22 is unrealistic due to the complex dynamics of training process and the variance of stochastic gradients. Now theorem I provides realistic meaning of equilibrium in SGD settings: equilibrium is just a dynamic state of SMD, meaning || oscillates around the theoretical value (w;) determined by hyperparameters and unit gradient norm. Its variance is bounded by 2V /l, which relatively small comparing with (wf) )d because
2V /1001
4V Xq O(An) < 1.
= -
99)
1
4

---

Aside form the stochastic behavior of | the "dynamic state" also reflects in the variation of the theoretical value (wz)². Because (w;)² is determined by L,, which is allowed to change smoothly across the whole training process in assumption 2 (See more discussion in appendix). In summary, the sign of equilibrium is neither the convergence of weight norm |wwlB (van Laarhoven 2017 Chiley et al. 2019) nor the convergence of |,wll2 in expectation (Li et al. 2020). The real sign of equilibrium is whether B/ is close to its theoretical value (w,)².
Theorem T also shows the dynamic equilibrium can be reached in a linear rate regime when vanishing term is larger than constant term in Eq. ( The approaching rate only determined by predefined parameters X, 77. Moreover, based on the proof of theorem [] the cause of equilibrium is independent of optimization process all. which implies the possibility that equilibrium can be reached long
before the convergence of loss function, which refutes the conjections that "equillbirum" equilibirum distribution of function space.
Now we extend theorem to momentum case. SGDM is more complex than SGD since momentum not always perpendicular to the weight, hence we need to modify assumptions.
Theorem 2. (Equilibrium in SGDM) Considering the update rule of SGDM (heavy ball method PPlyyk 1964)):
Vt or1 + Dt + Aw,
(10)
Wr+1 Wt 170z
(11)
where X, E (0,1), E (2. 1). If following assumptions hold:
5) An < L, An < (1 (1-va)²;
6) Define h = 1 + (1: h4 = he 1/ 11.. L [us). VV E ER, WENt
El( L,)3 [,)3(un) w.) VV
7) Vt E N L, satisfies L Lel < AvV(2)))
8) 31 E R.ENEN hy > 1>>>0 2[ (1+4)-8Xn(111)
GAn 12
then 3B,C > 0, C only depends a, w V/ V L1 (X(1 -2) An/(1+a)), we have
BlIlw,11 18 (w;) S 2\m
Vn²
1-a )B+ T
(12)
So far. no other work rigorously prove equilibrium can be reached in SGDM. The most relevant work (Li et al.) 2020)) only provides a conjecture on convergence rate of weight norm in SGDM. By regarding SGDM SGD) with larger learning rate, they guess that the mixing time to reach equilibrium SGDM case should be ((1/(2r)) same order as mixing time in SGD case. Their conjecture cannot provide further insight on difference between SGD and SGDM. While our results (vanishing terms in Eq.(8), [2] respectively) clearly reflect the difference: the approaching rate of SGDM should be 1/(1- a)) times larger than rate of SGD with same ni is usually set as 0.9 in practice, hence SGDM can reach equilibrium condition much faster than SGD.
Remark 3.
Proof can be seen in appendix. Like assumption 1, assumption 5 also holds for commonly used hyperparameter settings; Assumption 6, 7, 8 concerns not unit gradient norm |l 2 but an adjusted value h, which dominates the expectation and variance of ||we 11.. We empirically find the expectation of (v,-1,9.) is very close to 0, therefore the behavior of h, is similar to that of || 1| (see Figure 3(d)). justification of assumptions
We leave theoretical analysis on he future work. The experiments 6. 7, 8 can be seen in Figure 3 Comparing with Eq.(8) and Eq. 12) we can infer with same 7), X, SGDM can reach equilibrium state much faster than SGD, but it may have a larger variance, our experiments also verify our claim (see Figure 3(b) 3(e).
We have derived the theoretical value of weight norm in equilibrium, it allows us to check if equilibrium has been reached practice. But the theoretical value of weight norm still relies on the expectation of unit gradient norm, which is not easy to compute in practice. Besides, weight norm is of little value for studying normalized models since their weight is scale-invariant. Hence we introduce new and meaningful index, angular update, to reflect the effect of SMD and equilibrium.
5

---

Definition 3 (Angular Update). Let denote scale-invariant weight from neural network at iteration t, then angular update D is defined
(We, Wr+1)
A = L(wi, wi-1) = arccos
(13)
| ||
where 2...)) denotes the angle between two vectors,
(,,)) denotes the inner product.
Angular update has a concrete geometric meaning (see illustration in Figure D it is exactly the
geodesic distance between W and i 1 on S where wh = w 1l2, uit+1 = u /|| 12, angular update As better reflects the effective
Comparing with the Euclidean distance || update of the scale-invariant weight un on its intrinsic domain SP-1, Angular update is determined by the relative sizes of gradient norm and weight norm, while the relative sizes of gradient/weight norm are influenced by SMD, hence angular update is strongly affected by SMD. The following theorem exhibits the behavior of angular update when equilibrium is reached in SMD.
Theorem 3. (Theoretical value of Angular Update) In SGD(SGDM) case, if asssumptiinss in theorem 1(2) hold, v² < I, is sufficiently large so that vanishing terms in Eq. 8). (12 can be omitted, then
with probability at least
we have V Let
2 I+a
A
<O1V L,
(14)
In SGD case, a
Remark 4. Results of SGD and SGDM case are summarized in Eq. [4] in order to highlight the connection between SGD and SGDM. Theoretical value of angular update in Theorem 3 is partially consistent with previous works (Chiley et al. 2019 Li & Avora 2020 Li et al. 2020
Kumin et al.
2027), detailed discussion is buried in appendix. Note bias term in right side of Eq. (14) is of
O ((VV)LL)) which is too large comparing with its empirical value (see Figure 3(c) 3(f) we leave as a future work to improve the bound in Eq. 14).
Proof appendix. According to theorem 3 the theoretical value of angular update in equilibrium only depends on hyper-parantn paper-parameters: learning rate 17, WD factor X, and momentum factor Hence comparing with behavior of weight norm, angular update provides an easier way check whether equilibrium is reached. Since equilibrium can be reached in a linear rate regime as theorem D demonstrate, theorem 3 implies update efficiency of scale-invariant weights within single step eventually will be determined only by predefined hyperparameters, regardless other attributes of the weights (shape, size, position in network structure, or effects from other weights).
3
The Role of Angular Update in Sphereical Motion Dynamics
Far beyond just an index indicating whether equilibrium is reached, angular update is also a important way by which SMD can affect the optimization process of normalized neural net- work. In this section, we will provide an intu- itive explanation on the role angular update in folklore view of landscape exploration.
We
We+1
Wc+2
W2
Wy+1
The optimization process of gradient descent method is like an expedition to find the bottom of a basin in loss landscape (i.e. local mini- mum). It is well known that gradient descent with constant learning rate can always converge to the minimum in strong convex problem as the gradient norm will converge to zero (Boyd et al. 2004);; while SGD with constant learning rate can not converge to the minimum certainly, because noise of gradients make trajectory "escape from local minimum" Zhu et al. 2018
W542
Figure 2: Illustration of local minimum and angu- lar updates. Blue arrow indicates a single step with larger angular update, while green arrow indicates a single step with smaller angular update.

---

However, the behavior of SGD(M) in equilibrium is unique: due to the scale invariant property, the loss landscape is defined on unit sphere S so does the basin (local mimium). Then based on the geometrical meaning of angular update, angular update denotes "single step length" on the loss landscape. According to theorem (3] when equilibrium state has been reached in SMD, angular update is approximated equal to v /22m/(1+ ax. Hence with constant 7), A, and a, the optimization trajectory is "stuck" at the top of the basin, unable to reach the bottom (converge/reduce loss).juss like the illustration in Figure 2 The value of angular update in equilibrium is totally determined by hyper- pper-parameters, there is way the optimization trajectory cquilibrium due to overly large single step length. Based on the analysis above, we have following conjectures: if the trajectory stays in the same basin with different values of angular update in equilibirum, and momentum factor is fixed, then
move to the bottom of basin in
1. Larger angular update leads to higher training loss and lower test accuracy in equilibrium:
2. Even with different learning rate and weight decay, same angular update leads to similar training loss/test accuracy in equilibrium;
We will verify our conjectures in following empirical study. Note the influence of momentum is different from learning rate and weight decay: even momentum can decrease the value of angular update as theorem3 implies, it still can increase the oscillating range of the trajectory in the basin, resulting in larger training/testing loss.
Experiments
In this section, we verify our theorems and conjectures on SMD and equilibrium by empirical study. We show the equilibrium phenomenon described in our theorems really occurs in various computer vision tasks including ImageNet (Russakovsky et al.) 2015) and MSCOCO (Lin al. 20014). We also analyze an interesting phenomenon as an example to show how SMD can affect training process way different from traditional view on optimization of neural network.
4.1 Fixed learning rate
First verify our theorems on ImageNet (Russakovsky al. 2015) with fixed learning rate. We use Resnet50 (He et al. 2016 as the baseline model. In SGDM case, the momentum factor is 0.9. Form Figure 3(b) 3(e) 3(c) 3(f) we can empirical value of ||,,|| and A, differ from their theoretical value respectively at very beginning, because the initialized value of weight norm handcrafted, far away from the theoretical value in equilibrium. After several iterations, empirical values of weight norm and angular update agree with their theoretical values very well, which implies equilibrium has been reached. We also observe SGDM can achieve equilibrium much faster than SGD. According to Eq. 8 (2). the underlying reason might be with same learning rate and WD factor X, approaching rate of SGDM ATا is larger than approaching rate of SGD (Ar). Results in Figure 3 also prove our claim that equilibrium is a dynamical state: even equilibrium has been reached, [E| g 1/ (Eλ,) constantly increase, ||w,,11 increases accordingly. ||w, 2/ and A, always oscillate around their theoretical values respectively, showing equilibrium state maintains in SMD.
Then we train Resnet50 on Imagenet with different hyperparameters to verify our conejctures. The results are shown in Figure 4 We can see the experimental results strongly support our conjectures: in SGD case, whatever learning rate 77 and WD factor À is, they have similar angular update in equilibrium, well as similar training/testing curves; when angular update is larger, both training loss and test error are larger. These phenomena are even more obvious in SGDM case, when momentum factor is fixed. Besides, the role of momentum in SMD is exactly speculate: it can decrease angular update while increase training/testing loss.
4.2 Multi-stage learning rate
Now we study the behavior of angular update with SGDM and multi-stage learning rate schedule on Imagenet (Russakovsky et al. 2015) and MSCOCO (Lin et al.) 2014). In ImageNet classification task, we still use Resnet50 as baseline model. The training settings rigorously follow Goyal et al. (2017):
In MSCOCO experiment, conduct experiments on Mask-RCNI (He et al. (017)) benchmark
7

---

6.007
0.00
1.000
1.002
Tbach
(a) Unit gradient norm in SGD
(b) Weight norm in SGD
(c) Angular update in SGD
16
0.00
5.003
0.000
Trecoetzelaatiee
20
40
20
(d) Unit gradient norm square and in SGDM
(c) Weight norm in SGDM
Angular update in SGDM
Figure 3: Performance of 1oyer.2.0.comv. from Resnet50 in SGD and SGDM, respectively. In (a), (d), semitransparent line represents the raw value of @ ht while solid line represents the averaged value within consecutive 200 iterations estimate expectation of _. or Ji, conditioning on In (b), (e), blue solid lines represents the raw value of weight norm /w:122. while dashed line represent the theoretical value weight norm computed Theorem T
compute the theoretical value weight norm, we use the estimated IE] g 112 and Eh (solid lines) in (a) and (d) respectively; In (c), (f), red lines represent raw value of angular update during training, dashed lines represent the theoretical value of angular update computed by 22n and 22nn/(1 + a) respectively.
respectively.
(a) Training Loss (SGD)
(b) Top-1 accuracy (SGD)
(c) Angular update (SGD)
-000
(d) Training Loss (SGDM)
(c) Top-1 accuracy (SGDM)
(f) Angular update (SGDM)
Figure& 4: Resnet50 are trained on Imagenet with different hyper-parameters. The angular update computed by the weight in
of Resnet50. Dashed lines in (c), (f) represent the
theoretical value of angular update by
2Am

---

(a) Angular update in Imagenet
(b) Angular update in Imagenet
(c) Weight norm in Imagenet
(rescaled)
(d) Angular update in MSCOCO (c) Angular update MSCOCO
(f) Weight norm MSCOCO
(rescaled)
Figure 5: In (d)) solid lines with different colors represent raw value of angular update from all convolution layers; In (a), (d), training setting rigorously follows Goyal et al. (2017); He et al. (2019) respectively; In (b), (e), weight norm is divided by V10 long as learning rate is divided by 10; In (c), (f,, weight norm is computed on layer. I. 000.con22 in Resnet50 backbone. Blue line represent original settings, orange lines represent rescaled settings.
using a Feature Pyramid Network (FPN) (Lin et al. 2017), ResNet50 backbone and SyneBN (Peng et al. 2018) following the 4x setting in He et al. (2019).
There appears to be mismatch between theorems and empirical observations in Figure 5(a) 5(d) angular update A, in the last two learning rate stages is smaller than its theoretical value. This mismatch can be well interpreted by our theory: according to Theorem 1 when equilibrium state is reached, theoretical value of weight norm |/ulll22 satisfies ||w.||2 ε V However, when learning rate is divided by k. equilibrium state is broken, theoretical value of weight norm ||we|12 in the new equilibrium state is t/1/k times smaller. But equilibrium cannot be reached immediately (see Figure 5(c) 5()). following corollary gives the least number of iterations to reach new equilibrium. Corollary 3.1. In SGD case with learning rate T, WD factor X, if learning rate divided by iterations are required
and unit gradient norm remains unchanged, then at least [[og(k)/ to reach the new equilibrium state; In SGDM case with momentum coefficient a, then at least (0og(k)(1a))((M)))) (2)/ iterations are required to reach the new equilibrium state.
Corollary 3.1 implies SGD/SGDM with smaller learning rate requires more iterations to reach new equilibrium state. Hence, in second learning rate stage in Imagenet experiments, angular update A, can reach its new theoretical value within 15 epochs. But in last two learning rate stages of Imagenet/MSCOCO experiments, SGDM cannot completely reach new equilibrium by the end of training. As a result, we observe empirical value of A, is smaller than its theoretical value. Based on theorem, we can bridge the gap by skipping the intermediate process between old equilibrium and new one. Specifically, when learning rate is divided by h. norm of invariinv weight is also divided by vk, SGDM can reach new equilibrium immediately in new learning rate stage. (e)) in Figure show this simple strategy make angular update A, always close to its theoretical value across the whole training process though learning rate changes.
4.3 PPsduuoo overfitting" caused by learning rate decay
"Dropping test preformance(accuracyy
phenomenon is often explained as result of "overfitting" issue, which refers to the phenomenon where trained model can fit training data very well, but fails to fit additional data for validation prediction. To handle such issues, regularization methods

---

like early-stopping (Prechelt, 1998) and dropout (Srivastava et al., 2014) are proposed. But here we show in some cases "dropping test preformance" phenomena just "pseduo" overfitting, caused by learning rate decay.
Let's see the following experiment in which Resnet18 (He et al. 2016) is trained on CI- FARIO (Krizhevsky & Geoffrey) 2209) with standard settings (see experiment settings in appendix) Figure presents the training curves of the experiments. It can be seen from figure ((b) line inside red ellipse) that with standard implementation of SGDM, test accuracy severely drops during the 2nd learning rate stage. It seems a typical overfitting issue. However, two other phenomena cannot be well interpreted by overfitting: 1) test accuracy only severely decreases during 2nd learning rate stage, it performs normally in the 3rd, 4th learning rate stages (blue lines in Figure6(b)): 2) When other optimization method like Adam (Kingma & Bal (015) is used, test accuracy does not drop apparently in any learning rate stages (green line in Figure6(())
SUEMisknsdard
suomeolandars
Atpush tosune
The Accurnc
Epech
Rpoch
(a) Training loss
(b) Test Accuracy (Top-1)
(c) Angular update
Figure 6: Training curves of Resnetl 8 trained on CIFARIO (averaged across 5 seeds). Angular update is from layerI. layer1.0.conv/ 10.oovvl of Resnet18.
Now SMD can provide more reasonable in- terpretation: we have demonstrated the conjec- tures on the relationship between angular update and performance of model (Sec. 33. and the fact that learning rate decay can make angular update increase (Sec. [4.2) above, so it's rea- sonable to speculate that learing rate decay is the cause of the temporary "dropping test accu- racy" phenomenon. See illustration in Figure 7 after shrinking learning rate, smaller angular up- date allows the optimizaton trajectory to move towards the bottom of the basin, then angular update will increase, forcing trajectory to "escape" from the bottom, resulting in dorpping test performance. If this interpretation is ture, then rescaling strategy introduced in Sec. 4.2 should avoid "pseduo dropping test performance" by eliminating increasing angular update phenomenon. Further experiments support our speculation (Figure 6). They also explain why dropping test performance cannot be seen Adam: we track the angular update with Adam, and no increasing angular update phenomenon is observed. Please see more discussion in appendix.
We
4
We+1
Ae+1
We+4
Ac+3
Dct2
Wt+2
Wx+3
Figure 7: Increasing angular caused by learning rate decay.
5 Conclusion
In this paper, we comprehensively reveal the learning dynamics of normalized neural network with SGD/SGDM and weight decay (WD), named as Spherical Motion Dynamics (SMD). With mild assumptions, we strictly prove SMD will reach equilibrium state in a linear regime. We also propose novel index, angular update, to depict the state of SMD, and derive its theoretical property in equilibrium. Most importantly, we show our theorem is widely valid, they can be verified on challenging computer vision tasks, beyond synthetic datasets. Besides, dramatically effect the optimization of neural network by controlling angular update in practice. We believe our results on SMD make important step to understand the mechanism of deep neural networks, and can inspire new deep learning techniques.
show SMD can
10

---

Acknowledgments and Disclosure of Funding
This work is supported by The National Key Research and Development Program of China (No. 2017YFA0700800) and Beijing Academy of Artificial Intelligence (BAAI), We really appreciate the valuable suggestions on proof of the theorems from Dr. Donghao Wang and Dr. Siyi Yang.
References
Ba, J. L., Kiros, J. R., and Hinton, G. E. Layer normalization. arXiv preprint arXiv: arXiv:1607.04450 2016.
Boyd, S., Boyd, S. P., and Vandenberghe, L. Conver oplimization. Cambridge university press, 2004.
Chiley, V., Sharapov, I., Kosson, A., Koster, U., Reece, R., Samaniego de la Fuente, S., Subbiah, V., and James. M. Online normalization for training neural networks. In Advances in Neural Information Processing Systems 32. pp. 1433-8442 Curran Associates, Inc., 2019.
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L.. Kyrola, A., Tulloch, A., Jia, Y., and He, K. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint arXiv: 1706. 02677, 2017.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
He, K., Gkioxari, G., Dollár, P., and Girshick, R. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969 2017.
He, K., Girshick, R., and Dollar, P. Rethinking imagenet pre-training. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), October 2019.
Hoffer, E., Hubara, I., and Soudry, D. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. In Advances in Neural Information Processing Systems, pp. 1731-1741, 2017.
Hoffer, E., Banner, R., Golan, I., and Soudry, D. Norm matters: efficient and accurate normalization schemes in deep networks. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 31, pp. 2160-2170. Curran Associates, Inc., 2018.
loffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, pp. 448 456, 2015.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.
Krizhevsky, A. and Geoffrey, H. Learning multiple layers of features from tiny images. 2009.
Kunin, D., Sagastuy-Brena, J., Ganguli, S., Yamins, D. L., and Tanaka, H. Neural mechanics: Symmetry and broken conservation laws in deep learning dynamics. In International Conference on Learning Representations, 2021. URL htttp:://ppereeeeeeeeee
net/forummid beet/forumrtddgggg_...BBBB q8qLAbQBupm
Li, Z. and Arora, S. An exponential learning rate schedule for deep learning. In International Conference on Learning Representations, 2020. URL htttpps://pppeeeeeeeeeeeee///rre TJg8TeSFDH
Li, Z., Lyu, K., and Arora, Reconeiling modern deep learning with traditional optimization analyses: The intrinsic learning rate. Advances in Neural Information Processing Systems, 33, 2020.
Lin, T.-., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects context. In European conference on computer vision, pp- 740-755. Springer, 2014.

---

Lin, T.-.,, Dollár, P., Girshick, R., He, K., Hariharan, B., and Belongie, S. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2117-2125, 2017.
Ma, N., Zhang, X., Zheng, H.-T., and Sun, J. Shufflenet v2: Practical guidelines for efficient cnn architecture design. In Proceedings of the European Conference on Computer Vision (ECCV), pp- 116-131, 2018.
Peng, C., Xiao, T., Li, Z.,, Jiang, Y., Zhang, X.,, Jia, K., Yu, G., and Sun, J. Megdet: A large mini- batch object detector. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 6181-6189, 2018.
Polyak, B. T. Some methods of speeding up the convergence of iteration methods. USSR Computa- tional Mathematics and Mathematical Physics, 4(5):1-17, 1964.
Prechelt, L.. Early stopping-hut when? In Neural Networks: Tricks of the trade, pp. 55-69. Springer, 1998.
Russakovsky, O., Deng. J., Su, H.. Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla,
A., Bernstein, M. et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3)2211-22) 1-222,, 2015.
Salimans, T. and Kingma, D. P. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In Advances in Neural Information Processing Systems, pp- 901-909, 2016.
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1): 1929-1958, 2014.
van Laarhoven, T. L2 regularization versus batch and weight normalization. In Advances in Neural Information Processing Systems. 2017.
Wu, Y. and He, K. Group normalization. Proceedings of the European Conference on Computer Vision (ECCV). pp. 3-19, 2018.
Zhu, Z.,, Wu, J., Yu, B., Wu, I.., and Ma, J. The anisotropic noise in stochastic gradient descent: Its behavior of escaping from minima and regularization effects. arXiv preprint arXiv:1803.00199. 2018.
12


Reviews:
[{"review_id": "mHs2r49Zmq4", "reviewer": "Reviewer_uKSg", "summary": "This paper investigates the time-evolution of weights in normalized neural networks (e.g. one in which every layer has batch-norm applied). It proposes theorems regarding convergence to equilibrium, in which the norm of the weights approaches a fixed value. Compared to prior work, which established the fixed value, this work additionally gives the approach rate, and the scale of the variance due to stochastic gradients. The paper introduces the angular update parameter, and shows this approaches a fixed value. The paper emphasizes that equilibrium is not reached immediately, but is approached dynamically. It also asserts that equilibrium is not about convergence of stochastic oscillations to zero, but instead convergence to a fixed value, up to oscillations. The theoretical results are then investigated empirically, showing agreement with theoretical results for the weight norm and angular update, and the dynamical approach to equilibrium. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**\n\nAs stated in the paper, the fixed value of the weight norm was already found in van Laarhoven, so the extra piece this work adds here is the rate of approach and the scale of the stochastic variance. It also has a corresponding result for momentum. For the angular update, the equilibrium fixed value had already been found Chiley and other papers (see also Kunin). The added bit here seems to also be the momentum term. I am not aware of whether the other papers conducted a similar experimental verification of their results. It would be good to clarify if Kunin is to be treated as concurrent or as prior work. The related work section was vague on how this relates to recent work, lines 124-126.\n\n**Quality**\n\nI have not checked the proofs of the statements in the appendix, but the experiments seem to be of high quality overall. My main comment here is that fig 4b regarding the drop in test accuracy could be strengthened by running more seeds etc as this might just be a random variation. \n\n**Clarity**\n\nThe submission is clearly written and well organized. The weakest section is 4.2, where the motivation and explanation of the experiment is not that clear. Also, I’m not sure why a new term ‘spherical motion dynamics’ is necessary, and may also be misleading as the motion may sometimes not be restricted to a sphere (if I've understood correctly, when out of equilibrium, the motion is not restricted to the sphere). \n\n**Significance**\n\nIf the main original contributions are the rate of approach and scale of variance, why are these useful? This is not explained or motivated. Similarly for the extension of existing results to the case of momentum. It would be good to explain further why this is useful to a practitioner.\n\n\n-----\n\n**Edit**: \n\nI have updated my score from 6 to 7 based on the originality of this work being more clear to me, following author responses. I now see that the value to which the weight norm square tends to in equilibrium is not a constant value (as in prior/concurrent works), but is a non-constant value, due to the expected squared norm of the gradients being non-constant. This is also empirically demonstrated in Fig 2b,e. \n\nThe author reply also assured me how the clarity of the work will be improved in a possible camera-ready version.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper has no discussions of limitations (technical or otherwise), nor of societal impact. Since this is mostly theoretical, the societal impacts are not direct, so this seems ok. But I would like to see discussion of technical limitations - for example, what about networks in which not all weights are scale-invariant (just a subset), eg ones where only a few types of layer are normalzed. ", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "NzkxXisG_lS", "reviewer": "Reviewer_Zj3b", "summary": "This paper studies the dynamics of Spherical Motion Dynamics (Stochastic Gradient Descent with momentum and Weight Decay), more specifically, the behavior of “effective learning rate\" in the “equilibrium\" state. The paper (1) gives the assumptions for the equilibrium state and prove the equilibrium can be reached in linear time; (2) describes an “angular update” as a substitute for effective learning rate to depict the state of SMD; (3) verifies the assumptions and theories on ImageNet and MSCOCO.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n+ This paper gives solid and rigorous theoretical results on the dynamics of Spherical Motion Dynamics. The study is timely and important. The proofs are given in detail. The results are well motivated. It may shed light in understanding deep learning (while this part is not discussed sufficiently; see below).\n\n+ The theoretical results are well motivated. The authors give sufficient discussion to the idea of the theoretical study.\n\n+ The paper gives clear proof sketch. The proof idea is clearly presented. Readers can easily follow the idea without checking the whole proof details in the appendices.\n\n+ The paper gives comprehensive empirical results on very large-scale datasets, ImageNet and MSCOCO. The empirical results are in full agreement with the theory. This is much impressive for a theory paper. \n\nCons:\n\n- The paper does not give a sufficiently comprehensive literature review. The presented contributions are not well positioned in the current state of research. Readers may fill confusing for some of the results. The advantages and difference of this work from the existing results.\n\n- The paper does not give enough practical implications of the presented theory. The authors are encouraged to discussion how this work can contribute to future novel algorithm design.\n\n- The paper has many typos. It seems written in a rush. I recommend the authors carefully proofread the paper in the next version.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. The authors discuss the conflict between the theory and experiments in remark 3 though it is not so serious.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "one day"}, "extra_text": {}}, {"review_id": "15YEeVkp2sA", "reviewer": "Reviewer_VeBT", "summary": "In this paper, the dynamics of scale-invariant (SI) weights of normalized neural networks trained with SGD(M) + WD are studied. It is known that the scale-invariant parameters' intrinsic domain is a unit hypersphere, which the authors have effectively taken into account by introducing the \"angular update\" (AU) term, and the gradients w.r.t. SI parameters 1) scale inversely proportional to parameters norm and 2) are orthogonal to the radial direction in parameter space: the first property compels to introduce a unit gradient term and condition the dynamics of SI parameters on the intrinsic domain, the second one, together with weight decay, gives birth to the main concept of the paper &mdash; Spherical Motion Dynamics (SMD). Important attainment of this work is that, in contrast to the prior art, the notion of SMD equilibrium is not static (i.e., when $\\Vert w_{t} \\Vert_2 = \\Vert w_{t+1} \\Vert_2$) but represents a dynamic state when parameters norm oscillates around its theoretical value determined by hyperparameters and (generally varying) unit gradient norm. Interestingly, despite the dynamic nature of the equilibrium in terms of weight norm, the authors have shown that AU, in contrast, becomes approximately constant with its value only dependent on hyperparameters: learning rate, weight decay, and momentum factor. The authors corroborate their theoretical findings with experiments involving different computer vision tasks and networks architectures. They also provide an intriguing view on overfitting from the SMD perspective: when the learning rate drops, test error sometimes increases which could be interpreted as escaping from the local minimum due to an increase in the angular update.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**  \nDespite a large volume of the related literature (well cited by the authors), I find the contributions of this paper novel and original. I put a special emphasis on the notion of angular update, dynamic interpretation of the equilibrium, formulation of the main results in discrete setting (instead of continuous SDE approximation), and generalization to the SGD with momentum case.\n\n**Quality**  \nThe submission is overall technically sound, although there are some concerns about the reasonableness of assumptions, experimental setup, and clarity of proofs (see Weaknesses below). The main claims are supported both by theoretical analysis and experimental results. All in all, I consider this submission a complete piece of work.\n\n**Clarity**  \nErrata aside, this paper is clearly written and well organized. The authors provide enough details in the main text and appendix to reproduce the experiments and verify the claims.\n\n**Significance**  \nThe results of this work are important and provide valuable insight into the dynamics of learning of normalized neural networks significantly differing from previous works (see Originality). Notably, the authors state that the effective stepsize of SI weights will eventually be determined only by hyperparameters, regardless of any other attributes like shape, size, position, etc. While the results are indeed intriguing, their validity can be compromised by the limitations below.\n\n**Weaknesses**  \nBelow I provide a list of principal concerns about this submission. I would like the authors to address them primarily.\n1. Here I question the assumptions made in Theorem 1, though similar concerns apply to Theorem 2 as well.\n    1. Assumption 2 requires $V$ to be a *uniform* bound on the conditional squared unit gradient norm variance which can be a large value and lead to vacuous results (see below). Could this uniformness assumption on variance be weakened?\n    2. Even more important, in Assumption 4 you set a *uniform* lower bound $l$ on the square unit gradient norm that, in addition, must not differ too much from the conditional expected squared unit gradient norm $L_t$ for *all* iterations. The second part does not allow the derived results to hold when the network converges to a minimum and the unit gradient norm tends to zero, so the lower bound is not valid anymore. I believe that might be the reason why the plots in Figure 2 depict only the beginning of training when the gradient norm is still high. Again, could this assumption be further weakened?\n    3. Assumption 3 demands $L_t$ to change as $\\mathcal{O}\\left( (\\lambda \\eta)^{3/2} \\right)$, i.e., very slowly since in assumption 1 you state that $\\lambda \\eta \\ll 1$. I do not find this \"smoothness of mean squared gradient norm dynamics\" condition convincing enough, especially considering the possible heavy-tailness of the stochastic gradient noise yielding sudden jumps in gradient norm (see, e.g., https://arxiv.org/abs/1901.06053). Furthermore, a concurrent work (https://arxiv.org/abs/2106.15739) reveals that such jumps can in fact be regularly observed in normalized networks.\n2. Considering that $V$ can be larger than one (see Figure 2a) and $L_t l$ is always below one (as a product of mean and min values), the value $\\sqrt[3]{\\frac{V}{L_tl}}$ can easily exceed one, hence the statement in Theorem 3 becomes vacuous: something can happen with probability larger than a negative value. I appreciate that the authors note in Remark 3 that their theoretical bound is too large compared to the empirical evidence, however, this issue may be of secondary importance. I expect the authors to reconsider the statement in Theorem 3. Would a formulation in terms of the *expected* angular update fit better (especially in light of the empirically observed behavior)?\n3. In the process of reading, I encountered several questions about Figure 2, so some additional comments on it in the text would be appreciated. \n    1. Why does the gradient norm start from zero and only grow? If I understand correctly, it does so due to the vanishing gradient effect for the first layers at the beginning of training and a limited number of iterations. However, usually, the total gradient norm (i.e., w.r.t. all network SI parameters) behaves differently, so does the behavior hold for all layers and the whole network? A brief comment on this would help a reader. While the lines for different layers are presented in Figure 3, more examples in the setting with a fixed learning rate in the appendix would also be beneficial.\n    2. Does the behavior of angular updates remain stable after gradient norm saturation? Since the paper is about reaching a certain equilibrium state, adding at least one longer experiment would make the empirical results more convincing and even partially address the concerns about assumptions in Theorems 1 and 2.\n4. About the “Overfitting” part:\n    1. In line 331 of the main text, you write that \"Adam has no equilibrium state like SGDM\". Why so? Could you, please, comment more on this? A recent work https://arxiv.org/abs/2006.13382 could be relevant.\n    2. The explanation in lines 324-328 of the main text seems vague. Rewriting this part using the intuition from Appendix F would make the text clearer.\n    3. I suggest putting the bolded text from lines 493-494 of the appendix into the main text to avoid confusion that rescaling strategy helps generalization.\n\nMinor concerns:\n* When stating the main theorems, you use different iterations for defining certain important values in the main text and the appendix (compare, e.g., definitions of $w_t^*$ in line 148 of the main text and line 129 of the appendix). Also, heed the conditioning on $w_t$ when defining $L_t$ in lines 145 and 184 of the main text (compare, again, with lines 126 and 185 of the appendix, respectively). While I believe that the main claims are still valid, I would like the authors to fix these inconsistencies.\n* I admit that I did not thoroughly follow all the proofs, however, I managed to find several inaccuracies. For instance, please explain what is going on in Eq. (38): e.g., what is $L$? Also, I believe you missed $2$ in Eq. (58), however, I still do not completely understand how it follows from equations (56) and (57), namely, why did you put $(1 - 2 \\lambda \\eta)$ in the denominator? I also suppose that the inequality signs must be $>$ in the second line of Eq. (149). Considering these and other issues, I advise you to revise your proofs more thoroughly.\n* In Appendix E (and partially in the main text), you conjecture that larger angular updates lead to worse performance. However, I suppose that a more accurate formulation would be \"*too* large angular updates *may* lead to worse performance\" due to the following reasons: first, you consider only two AU values in Figure 3a-c in the appendix, which is not enough for making far-reaching conclusions; second, there exist plenty of works showing that larger learning rates can aid generalization by escaping from narrow minima (see, e.g., https://arxiv.org/abs/2003.03977 and references therein); third, the experiments with momentum, as noted in the text, contradict this conjecture (the explanation that momentum prevents optimization from reaching the bottom of the loss landscape is unconvincing since momentum is specifically tailored for improving convergence); finally, at the end of Appendix F you confirm that growth of AU is not always followed by test accuracy drop. I suggest relaxing the conjecture or provide more experiments supporting it.\n\n---\n\n**UPD**  \nAfter the discussion, I have found most of my concerns addressed and decided to increase my initial score. I also hope that the authors will improve their paper based on our discussion: formulate their theorems more accurately (e.g., modify the unnecessary uniformness of bounds), provide additional longer experiments with saturating gradients (or even periodic behavior), change the demonstration of some of their claims to evade misunderstanding (e.g., about \"Adam has no equilibrium state like SGDM\"), etc.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I appreciate that the authors have noted some limitations of their work and possible future research directions  (e.g., in Remark 3 and at the end of Section 4.2), however, I find several important flaws in this submission to be addressed (see Weaknesses above).", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "16-20"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a clear accept.  Aside from the high scores of all the reviewers, I myself have seen versions of this paper in the past on arxiv (it was inexplicably rejected at a previous submission, but is even improved here) and have really benefitted from this work's thinking.  I think the contribution of this work is clear and important.   I think we'd be foolish not to accept this work here this year.  
Decision: Accept (Spotlight)