Spherical Motion Dynamics: Learning Dynamics of Neural Network with Normalization, Weight Decay, and SGD
Authors: Ruosi Wan
Zhanxing Zhu
Xiangyu Zhang
Jian Sun
Venue:  (2021)
Keywords: Normalization
Weight decay
SGD
Momentum

Abstract: In this work, we comprehensively reveal the learning dynamics of neural network with normalization, weight decay (WD), and SGD (with momentum), named as Spherical Motion Dynamics (SMD). Most related works study SMD by focusing on "effective learning rate" in "equilibrium" condition, where weight norm remains unchanged. However, their discussions on why equilibrium condition can be reached in SMD is either absent or less convincing. Our work investigates SMD by directly exploring the cause of equilibrium condition. Specifically, 1) we introduce the assumptions that can lead to equilibrium condition in SMD, and prove that weight norm can converge at linear rate with given assumptions; 2) we propose "angular update" as a substitute for effective learning rate to measure the evolving of neural network in SMD, and prove angular update  can also converge to its theoretical value at linear rate; 3) we verify our assumptions and theoretical results on various computer vision tasks including ImageNet and MSCOCO with standard settings. Experiment results show our theoretical findings agree well with empirical observations.

Under review as a conference paper at ICLR 2021
SPHERICAL MOTION DYNAMICS: LEARNING DYNAM- ICS OF NEURAL NETWORK WITH NORMALIZATION, WEIGHT DECAY, AND SGD
Anonymous authors Paper under double-blind review
ABSTRACT
In this work, we comprehensively reveal the learning dynamics of neural network with normalization, weight decay (WD), and SGD (with momentum), named as Spherical Motion Dynamics (SMD). Most related works study SMD by focusing on "effective learning rate" in "cquilibrium" condition, i.c assuming the con- vergence of weight norm. However, their discussions condition can be reached in SMD either absent or less convincing. Our work in- vestigates SMD by directly exploring the cause of equilibrium condition. Specif- ically, 1) we introduce the assumptions that can lead to equilibrium condition SMD, and prove that weight norm can approach its theoretical value in linear rate regime with given assumptions; 2) we propose "angular update" as a substi- for effective learning rate to measure the evolving of neural network in SMD, and prove angular update can also approach to its theoretical value in a linear rate regime; 3) we verify our assumptions and theoretical results on various computer vision tasks including ImageNet and MSCOCO with standard settings. Experi- ment results show our theoretical findings agree well with empirical observations.
why this equilibrium
INTRODUCTION AND BACKGROUND
Normalization techniques (e.g. Batch Normalization (loffe & Szegedy 2015) its variants) are one of the most commonly adopted techniques for training deep neural networks (DNN). typical normalization can be formulated as following: consider a single unit in neural network, the input is X, the weight of linear layer w (bias is included in w), then its output
y X: wi 7: B) g( Xw v(xw) ( 7 7+ 3).
where g is a nonlinear activation function like ReLU sigmoid, μ4, mean and standard de- viation computed across specific dimension of X w (like Batch Normalization (Ioffe & Szegedy) 2015), Layer Normalization Ba et al. (2016). Group Normalization (Wu & Hel 2018). etc.). B, ? are learnable parameters to remedy for the limited range normalized feature map. Aside from normalizing feature map, Salimans & Kingma (2016) normalizes weight by la norm instead:
w
y ( w. 7) g(X X
7 + 3),
|w
2
where || 1/2 denotes l2 norm of a vector.
Characterizing evolving networks during training. Though formulated in different manners, all normalization techniques mentioned above share interesting property: the weight w affiliated with a normalized unit scale- intleeiiarian:: Ya IR+ (xx:W;; 3) y 7. Due to the scale-invariant property of weight, the Euclidean distance defined in weight space completely fails to measure the evolving of DNN during learning process. As a result, original definition of learning rate 17 cannot sufficiently represent the update efficiency of normalized DNN.
To deal with such issue, van Laarhoven (2017); Hoffer et al, (2018): Zhang et al. (2019) propose "ef- fective learning rate" substitute for learning rate measure the update efficiency of normalized

---

Under review as a conference paper at ICLR 2021
neural network with stochastic gradient descent (SGD), defined
77
Teff |/
Joint effects of normalization and weight decay. van Laarhoven 2017) explores the joint effect of normalization and weight decay (WD), and obtains the magniudes of weight by assuming the convergence of weight, i.e. if w; = wr+1, the weight norm can be approximated as |/wel12 (()n))), where À is WD coefficient. Combining with Eq.3), we have Reff VTyX. A more intuitive demonstration about relationship between normalization and weight decay presented in Chiley et (2019) (see Figure I): due to the fact that the gradient of scale invariant weight aC/Ow (L is the loss function of normalized network without WD part) is always perpendicular to weight w, one can infer that gradient component aL/0w always tends to increase the weight norm, while the gradient component provided by WD always tends to reduce weight norm. Thus if weight norm remains unchanged, or "equilibrium has been reached" one can obtain
Wi W++1 v 2n Blja//0wl12
ac/Ow
Wr
44. implies the magnitude of update scale-invariant of gradients, and effective learning rate should be V2nAA. Li & Arora (2020) manages to estimate the magnitude of update in SGDM, their
result is presented in limit and accumulation manner: if both limy ooc Rr = 1/T 1/ ===
||,,| and
limy- + Dr = 1/T E
||w, wit1ll exist, then we have
lim DT
2nd
(5)
200 RT 1+a
Though not rigorously, one can easily speculate from (q. the magnitude of update in SGDM cases should be 22n2/(1 + a.) in equilibrium condition. But proof of 5q. requires more strong assumptions: not only convergence of weight norm, but also convergence of update norm || wall2 (both in accumulation manner).
As discussed above, all previous qualitative results about "effective learning rate" (van Laarhoven 2017] Chiley et al., 2019
Arora (220)) highly rely equilibrium condi- tion, but none them explores why such equi- librium condition
0L aw; 77
-W,- n
be achieved. Only yan
Laarhoven (017)) simply interprets the occur- rence of equilibrium as natural result of con- vergence of optimization, i.e. when optimiza- tion is close to be finished, w, = W:+1, result- ing in equilibrium condition. However, this in- terpretation has an apparent contradiction: ac- cording to Eq.44) and (5). when equilibrium condition is reached, the magnitude of update is constant, only determined by syper-parameters, which means optimization process has not con- verged yet. Li & Arora (2020) also notices the non-convergence of SGD with BN and WD, so they do not discuss reasonableness of assump- tions adopted by (5. In word, previous re- sults about "effecttive learning rate" in equilib- rium condition can only provide vague insights, they are difficult to be connected with empirical observation.
W
4
Figure 1: Illustration of optimization behavior with BN and WD. Angular update A, represents the angle between the updated weight W, and its former value W++1-
In this work, we comprehensively reveal the learning dynamics of normalized neural network using stochastic gradient descent without/with momentum (SGD/SGDM) and weight decay, named as Spherical Motion Dynamics (SMD). Our investigation aims answer the following question:
'"Weight norm remains unchanged" means [w |2 ||ws+illa Chiley et al. 2019 calls this condition as "equilibrium" which will also be used in the following context of this paper. Note equilibrium condition is not mathematically rigorous, we only use for intuitive analysis.
2

---

Under review as a conference paper at ICLR 2021
Why and how equilibrium condition can be reached in Spherical Motion Dynamics?
Specifically, our contributions are
We introduce the assumptions that can lead to equilibrium condition in SMD, and justify their reasonableness by sufficient experiments. We also prove under given assumptions, equilibrium condition can be reached as weight norm approach to its theoretical value in a linear rate regime in SMD. Our assumptions show the equilibrium condition can occur long before the finish of the whole optimization:
We define a novel index, angular update, to measure the change of normalized neural network in single iteration, and derive its theoretical value under equilibrium condition in SMD. We also prove angular update can approach to its theoretical value in a linear regime along with behavior of weight norm. Our results imply that the update efficiency of SGD/SGDM on normalized neural network only depends on pre-defined syyper-parametes in both SGD and SGDM case;
We verify our theorems on different computer vision tasks (including one of most challeng- ing datasets ImageNet (Russakovsky et al., 2015) and MSCOCO (Lin et al.) 2014)) with various networks structures and normalization techniques. Experiments show the theoreti- cal value of angular update and weight norm agree well with empirical observation.
Recently, parallel work (Li et al., 2020b) is published to analyze the equilibrium condition of normalized neural network with weight decay in SGD case. They model the learning dynamic of normalized network as SDE, and propose the concept "intrinsic learning rate" (Ae An) to measure the converge rate of weight norm and effective learning rate. Their main result, in which converging time to equilibrium in SGD case is ((1/(An)), is consistent with our theory. The major difference between Li et al.] (2020b) and our work is that all results from Li et al. (2020b) are work derives the quantitative
qualitatively results conjectures limited on SGD case. While results on both SGD/SGDM cases, our analysis allows us to precisely predict the value of weight norm/angular update in large scale data experiments, and clarify the difference of approaching rate between SGD and SGDM case. Besides, [i et al. (2020b) conducts empirical study to discuss the connection between the equilibrium state and accuracy of trained model. Our work mainly focuses on the equilibrium condition itself, connection between equilibrium condition and performance of trained model is beyond our discussion in this paper. The experiments we present in main text are only for verification of our theory.
Our theory on equilibrium condition in Spherical Motion Dynamics implies equilibrium condition mostly relies on the update rules of SGD/SGDM with WD, and invlriinvariant property. The cause of equilibrium condition is independent of decrease of loss or trajectory of optimization, but equi- librium condition turns out to significantly affect update efficiency of normalized network by con- trolling the relative update (4). >. We believe equilibrium condition is one of the key reason why learning dynamic of normalized neural network is not consistent with traditional optimization the- ory (Li et al. 2020b). We think it has great potential to study the leaning dynamic of normalized network or develop novel efficient learning strategy under the view of Spherical Motion Dynamics.
2 RELATED WORK
Normalization techniques Batch normalization(BN (Toffe & Szegedy 2015) is proposed to deal with gradient vanishing/explosion. and accelerate the training of DNN. Rapidly, BN has been widely used in almost all kinds of deep learning tasks. Aside from BN, more types of normalization tech-
niques have been proposed to remedy the defects of BN (loffe 2017 Wu He 2018 Chiley al.] | et al. (020) or to achieve better performance (Ba et al. Ulyanov et al. [016]] Sal- imans Kingmal| 2016)) [Shao al. 2019 Singh Krishnan 2020). Though extremely effective, the mechanism of BN still remains as mystery. Existing works attempt to analyze the function of BN: Loffe & Szegedy (2015) claims BN can reduce the Internal Covariance Shift (ICS) of DNN;
Santurkar et al. (2018) argue that the effectiveness of BN is not related to ICS, but the smoothness of normalized network: Luo et al. (2019) shows BN can be viewed as an implicit regularization
technique; Cai et al. (2019) proves that with BN orthogonal least square problem can converge at linear rate; Dukler et al. (2020) proves weight normalization can speed up training in a two-layer ReLU network.
3

---

Under review as a conference paper at ICLR 2021
Weight decay Weight decay (WD) is well-known as l2 regularization, or ridge regression, in statis- tics. WD is also found to be extreme effective when applied in deep learning tasks. Krizhevsky & Geoffrey 2009) shows WD sometimes
even improve training accuracy not just generalization performance: Zhang et al. (2019) show WD can regularize the input-output Jacobian norm and re- duce the effective damping coefficient; Li al.] (2020a) discusses the disharmony between WD and weight normalization. A more recent work Lewkowycz & Gur-Ari (2020) empirically finds the number of SGD steps T until model achieves maximum performance satisfies T x Xrj where X, A. are weight decay factor and learning rate respectively, they interpret this phenomenon under the view of Neural Tangent Kernel (Jacot et al., 2018). showing that weight decay can accelerate the training process. Notice their result has no connection with equilibrium condition discussed in this work. Our results shows the cause of equilibrium condition can be reached long before neural network can get its highest performance.
Effective learning rate Due to the scale invariant property caused by normalization, researchers start to study the behavior of effective learning rate. |van Laarhoven (2017); Chiley et al. (2019) estimate the magnitude of effective learning rate under equilibrium assumptions in SGD case: Hoffer jet al.] (2018) quantify effective learning rate without equilibrium assumptions, so their results are much weaker; Arooa al. (2019) proves that without WD, normalized DNN can still converge with fixed/decaying learning rate in GD/SGD cases respectively; ZZhag et al. (2019) shows WD can increase effective learning rate; Li & Arora 2020) proves standard multi-stage learning rate schedule with BN and WD is equivalent to an exponential increasing learning rate schedule without WD. As proposition, Li Arora (2020) quantifies the magnitude of effective learning rate in SGDM case. But none of them have ever discussed why equilibrium condition can be reached. A recent work [i et al. (2020b) studies the convergence of effective learning rate by SDE, proving that the convergence time of ([L/(An)), where A, 17 are weight decay factor and learning rate respectively. Their result can only provide intuitive understanding, and is limited on SGD case.
PRELIMINARY ON SPHERICAL MOTION DYNAMICS
First of all, we review the property of scale invariant weight, and depict Spherical Motion Dynamics (SMD) in SGD case. Notice except definitions, all intuitive statements or derivations in this section mostly comes from previous literature, they are not mathematically rigorous. We summarize them to provide background of our topic and preliminary knowledge for readers.
Lemma 1. If ur is scale-imariant with respect to L(ur), then for all k: > O, we have:
7L
(wi,
(6)
Ow
w=
OC
OL k Ow w==w,
1
Ow w=kws
Proof can be seen in Appendix B.1 Lemma Tis also discussed in Hofter et al. (2018) van Laarhoven (2017); Li & Arora (2020). 77 implies gradient norm is influenced by weight norm, but weight norm does not affect the output of DNN, thus define unit gradient to eliminate the effect of weight norm.
Definition 1 (Unit Gradient). If w. 0. i w/|[w|1a the unit gradient of @C/Ow/ w-ur. aC/Ou w==i..
Setting 1: as 1/1
[wlll 1/|/
inEq.77) have
HL
1
0C
(8)
dw n mmmer
||w
Dw u u.ma..
A typical SGD update rule without WD is
wi+1 wi Dw w=
(9)
[|, 12 = |/ dividing both side of Eq. (9) by ||w/|12. 112. we have
ac = wi Neff
dC Fw w=w,
0 = aū
(10)
[w OeD w=
4

---

Under review as a conference paper at ICLR 2021
(q.(10) shows effective learning rate be viewed as learning rate of SGD unit sphere S (Hoffer et al., 2018). But effective learning rate still cannot properly represent the magni- tude of update, since unit gradient norm is unknown. Therefore we propose the angular update defined below.
Definition 2 (Angular Update), Assuming wi is scale-invariant weight from a neural network at iteration t, then the angular update A is defined as
(we, Wc+1)
A = <(wr, wt-1) = arccos
(11)
||wzll [|we+1ll
where ((..)) denotes the angle between two vectors,
( denotes the inner product.
According to Eq. /C/0w is perpendicular to weight w. Therefore, if angular update A, is small its
enough, can be approximated by first order Taylor series expansion of Lan Ar, then connection between effective learning rate and unit gradient norm
A tan (A.)
77
ac
1l2
Teff
Oc
(12)
w
dw w=
dw w=w
Another deduction from Eq. 6 is that weight norm always increases because
W 1w++11l 11 = ||w,112 12 + (nlll Oc
112) [well
(13)
dw
From Eq. we can infer that increasing weight norm can lead to smaller gradient norm if unit gradient norm unchanged. Zhang al. (2019) states the potential risk that GD/SGD without WD but BN will converge to a stationary point not by reducing loss but by reducing effective learning rate due to increasing weight norm. |rroa et al. (2019) proves that full gradient descent can avoid the risk and converge a stationary point defined in S but their results still require sophisticated learning rate decay schedule SGD case. Besides, practical implementation suggests training DNN without WD always suffers from poor generalization (Zhang al. 2019
Bengio & LeCun 2007
Lewkowyez & Gur-Ari 2020
Now considering the update rule of SGD with WD:
W1+1 = wt 77 Dw Awe)).
(14)
We can approximate the update of weight norm by
2C
| l2 = ||well22 - Any well
112
(15)
2| ww
18
Ow w=
Eq 15) implies WD can provide di- rection to reduce weight norm, hence Chiley et al. (2019) Zhang al. (2019) points out the possibility that weight norm can be steady, but do not explain this clearly. Here we demon- strate the mechanism deeper (see Figure I]: if unit gradient
The derivation of Eq. 15) is presented Appendix A.
remains unchanged, note "cen- tripetal force" (-Anl] is proportional to weight norm, while "centrifugal force" (2)
(2) is inversely proportional to cubic weight norm. As a result, the dynamics weight
norm is like spherical motion in physics: overly large weight norm makes centripetal force larger than centrifugal force, leading to decreasing weight norm; while too small weight norm makes cen- tripetal force smaller than centrifugal force, resulting in the increase of weight norm. Intuitively, equilibrium condition tends to be reached if the number of iterations is sufficiently large.
Notice that the core assumption mentioned above is "unit gradient norm is unchanged". In fact this assumption can solve the contradiction we present in Section convergence of w, leads to nonzero [w w,ll2/ | w.ll2. Our analysis implies the convergence of weight norm is not equivalent to convergence weight, steady unit gradient norm can also make weight norm converge, steady unit gradient norm does not rely on the fact that optimization has reached an optimum solution. But a problem about unit gradient assumption rises: unit gradient norm cannot remain unchanged during training in practice, it is not reasonable assumption. In next section, we formulate this assumption in a reasonable manner, and rigorously prove the existence of equilibrium condition in SGD case. Discussion about SGD cannot be casily extended SGDM case, because momentum is not always perpendicular to weight as unit gradient. But we can still prove the existence of equilibrium condition in SGDM case under modified assumptions.
5

---

Under review as a conference paper at ICLR 2021
4 MAIN RESULTS
First of all, we prove the existence of equilibrium condition in SGD case and provide the approaching rate of weight norm.
Theorem 1. (Equilibrium condition in SGD) Assume the loss function is ((X;w) with scale-
invariant weight w, denole 9t Dw w. 9t 9: -1wwlllz- Considering the update rule of SGD with weight decay,
UC
W++1 wi - .7- 7-(g. + Xwws)
(16)
where X. E (0, 1). If the following assumptions hold: 1) An) << 1; 2) 31, V,E Rt,
B |3 [wr]] = I. E[ - 1 [uo.] S 3) 31 E R+, |l9e| 2 > 1, > 2[ 1-1 2L. Then
BB > O. w²= = LLmm(2A), we have
E - S (1- (1m) B Vn²
(17)
Remark 1.
The theoretical value of weight norm w" in Theorem 7 iss consistent with the derivation
of weight norm in equilibrium in van Laarhoven (2017). though van Laarhoven (2077) assumes the
equilibrium condition has been reached in advance, hence van Laarhoven (2017) cannot provide
the upproaching rate and scale of bias/variance. The vanishing lerm consistent with the convergence time ((1/(2r)) presented in et al. 2020b
4xn)) B) in Eg. 17) is
Proof of Theorem can be seen in Appendix B.2 It shows the square of weight norm can approach to its theoretical value in linear rate regime (when vanishing term is larger than noisy term in Eq.917) and its variance is bounded by Vn² which empirically small. Now we discuss the reasonableness of assumptions in Theorem settings; assumptions and 3 imply the | remains unchanged within several iterations, and its lower bound cannot be far from its expectation.
Assumption I is consistent with commonly used
We need to clarify assumption 2 further: we do not require the Ell 112 must be constant across the whole training process, but only remains unchanged locally. On one hand, small learning rate(n) can guarantee B|[ga [/ changes slowly; on the other hand, when IE| [ 1l changes, means that square of weight norm will approach to its new theoretical value as Theorem descrrbes. Experiment result in Figure 2 strongly justifies our analysis. We also conduct extensive experiments to verify our claim further, please refer to Appendix C.T
Now we extend Theorem to SGDM case. SGDM is more complex than SGD since momentum is not always perpendicular to the weight, therefore we need to modify assumptions.
Theorem 2. (Equilibrium condition in SGDM) Considering the update rule of SGDM (heavy ball method Polyyk 1964)):
V; Qv; + 9: + Aww;
(18)
Wi+1 w.- 170,
(19)
where ,, (0, 1). If following assumptions hold: 4) An << 1, xn < (1 val2:
5) Define he |1g.113 + (v. hn by || 1
3I, V,E E R-. E E/h,| [x,] I.
El( - [)p/wr]] S V: 6331 E RRt, h4 > 1, > [[ (1)) 8An(
6λT
1 1-x)
then 3B > 0,
"= V 1/Ln((D() (2 An/(1 + aa)),, we have
4A7 4
yr+ 3V + 4a² + at) a(1
E|ll wrl 11 (w=))22 < 3B (1
1--1--0 1-a
(20)
So far, no other work rigorously prove equilibrium condition can be reached in SGDM cases. Even the most relevant work (Li et al. 2020b) only provides their conjecture on approaching rate of weight norm in SGDM, they speculate that the time of approaching to equilibrium should be ((1(()n)), same order as approaching time in SGD case. their conjecture cannot provide further insight. While our results (vanishing terms in Eq. 17 (0 respectively) can clearly reflect the difference: the approaching rate of SGDM should be 1/ a times faster than rate of SGD with same n) is usually set as 0.9, SGDM can reach equilibrium condition much faster than SGD.
Remark 2.
6

---

Under review as a conference paper at ICLR 2021
Proof can be seen in Appendix [3. used
Like assumption 1, assumption 4 is also satisfied for commonly
settings. Besides, An < (1 - -va² is also mentioned in Li & Arora (2020) for other purpose; Assumption shows not unit gradient gradient norm ll 12 but adjusted value hü dominates the expectation and variance of the weight norm square. We empirically find the expectation of (w/_1,9)) is very close to therefore the behavior of h, is similar to that of ll 2 (see (d) in Figure 2) making square of weight norm approaching its theoretical value in SGDM case. We leave theoretical analysis on he as future work. As for assumption 6, commonly used as an very small lower bound for I/L. settings (nd << 1) can make 2[ -Bx
6Am
The experiments on justification of assumptions 4,5,6 can be seen in Figure 2 and appendix C.1 Comparing with Eq.(17) and Eq. (20), it implies that with same 7,, A, SGDM can reach equilibrium condition much faster than SGD, but may have larger variance, our experiments also verify that(see (b), (e) in 2i
Since have proven weight norm will approach to its theoretical value in SMD in linear rate regime, we can derive the theoretical value of angular update A and its variance.
Theorem 3. (Theoretical value of Angular Update) In SGD case, if assumptions in theorem I holds, then 3C > 0, we have
V E(Ar - 22 <1 - 44 LI
(21)
In SGDM case, if "aassumptionss theorem holds. 3C > we have
4nd 2 1 I-a 1-a yTC (1 (1- a2) 4a² + atVV
E(A: V 1+aa
<
(22)
4Llaa
Remark 3. Theoretical value of angular update in Theorem 3 is consistent with Eq. 4
5 from
Chiley et al. (2019); Li & Arora (2020) respectively. Notice the variance
in 2q. 22 is of
Ll), it is too large comparing with its empirical value, we leave it as future work to improve the bound of variance term. Though connection between performance of neural network and equi- librium condition is beyond the main discussion of this paper, the findings of a parallel work (Li et al. y can inspire us the possible advantage of momentum: Li et al. (020b) interprets that smaller Xn can get higher performance when training DNN, but the approaching rate (O()r)) is slow; larger Xn has faster approaching rate but it can only get bad performance. Inspired by et al.) 22020b), we suspect that Dc x /xq is highly correlated to the performance of DNN: smaller angu- lar update in equilibrium condition can lead to better performance of DNN. According to theorem 3 with same Xn, momentum method has faster approaching rate(Xn) (1 a)) and smaller angular update ( (222n/(1 + a)) than pure SGD, This means momentun method can simultaneously accel- erate training process and improve the final performance, comparing pure SGD method. We leave this conjecture as a future work.
According to theorem 3 the theoretical value of angular update and its approaching rate almost only depends syper-parameters: weight decay factor X, learning rate 77, (and momentum factor a). It implies update efficiency of scale-invariam weights within a single step is totally controlled by predefined syper-parameters in equilibrium condition, regardless other attributes of the weights (shape, size, position in network structure, or effects from other weights). That's the key reasons why we propose Angular Update (Bq. (Fq to replace effective learning rate (Eq.3)): effective learning rate can only reflect the influence of weight norm, cannot reveal how unit gradient norm affect the relative update (see Eq. IOD. while angular update in equilibrium condition implies that both weight norm and unit gradient norm do not affect the scale of update within a single iteration, only paper-parameters (learning rate, weight decay factor, momentum factor) matter. Our experiment results strongly prove our claim (see Figure
Proof of theorem 3 is shown in Appendix B.4
3(a) 3(b) 3() 3
EXPERIMENTS
In this section, we show that our theorems can agree well with empirical observations Ima- geNet (Russakovsky et al., 2013) and MSCOCO [Ln et al. 2014). We conduct experiments two cases. In the first case we train neural network with fixed learning rate to verify our theorems in SGD and SGDM, respectively; in the second case we investigate SMD with more commonly used settings, multi-stage learning rate schedule.
7

---

Under review as a conference paper at ICLR 2021
5.1 FIXED LEARNING RATE
With fixed learning rate, we train Resnet50 (He et al. 2016) with SGD/SGDM on ImageNet. Learn- ing rate is fixed as 7/ = 0.2; WD factor is 10 with SGDM, the momentum factor a 0.9. Figure 2 presents the unit gradient norm square, weight norm, and angular update of the weights Resnet50 in SGD and SGDM cases, respectively. can inferred from Figure 2 the behavior of 12. he and hyper-parameter settings satisfy our assumptions in Theorem T and 2 therefore theoretical value of weight norm and angular update agree with empir- ical value very well. We also can observe SGDM can achieve equilibrium more quickly than SGD. According to Eq. ( the underlying reason might be with same learning rate r) and WD factor X, approaching rate of SGDM (1
from LAYER. W.V.R.2.0..OV..
Ån
is smaller than that of SGD Xn).
T-a
[il thepretze velue
34
0.00
10
0.000
0.00
0.000
Tracorientantee
20
00
20
-0
80
20
80
(a) Unit gradient norm in SGD
(b) Weight norm in SGD
(c) Angular update in SGD
16
34
12
0.00
10
0.001
Chan wulas
Esech
Zoect
Eoach
(d) Unit gradient norm square and in SGDM
(e) Weight norm in SGDM
(f) Angular update in SGDM
Figure 2: Performance of layer.2.0.conv2 from Resnet50 in SGD and SGDM, respectively. In (a), (d), semitransparent line represents the raw value of 1lg:11 or h. while solid line represents the averaged value within consecutive 200 iterations to estimate the expectation of g or h.s In (b), (e), blue solid lines shows the raw value of weight norm | while dashed line represent the theoretical value of weight norm computed in Theorem 1 respectively. The expectations E|lgl? and Eh are estimated as solid lines (a) and (d) respectively; In (c), (f), red lines represent raw value of angular update during training, dashed lines represent the theoretical value of angular update computed by /2 and 22m/(1 respectively.
5.2 MULTI-STAGE LEARNING RATE SCHEDULE
Now we turn to study the behavior of angular update with SGDM and multi-stage leanring rate schedule Imagenet (Russakovsky et al. 2015) and MSCOCO (Lin al. 2014). In ImageNet classification task, we still adopt Resnet50 as baseline for is a widely recognized network struc- ture.The training settings rigorously follow Goyal et al. (2017): learning rate is initialized as 0.1, and divided by 10 at 30, 60, 80-th epoch; the WD factor is 10 4, the momentum factor is 0.9. In MSCOCO experiment,
Mask RCNN (He al. 2017) benchmark using a Feature Pyramid Network (FPN) (Lin et al.) 2017). ResNet50 backbone and SyncBN (Peng et al. 2018) following the 4x setting in He al. (2019): total number of iteration is 360, 000, learning rate is initialized as 0.02, and divided by 10 at iteration 300, 000, 340, 000; WD coefficient is 10
conduct experiments
There appears be some mismatch between theorems and empirical observations in (a), (b) of Figure 3 angular update in the last two stages is smaller than its theoretical value. This phenomenon can be well interpreted by our theory: according to Theorem I 2 when equilibrium condition is reached, theoretical value of weight norm satisfies |we/|22
therefore when learning rate is
8

---

Under review as a conference paper at ICLR 2021
(a) Angular update in Imagenet
(b) Angular update in Imagenet (rescaled)
(c) Weight norm in Imagenet
33
(d) Angular update MSCOCO (e) Angular update in MSCOCO (f) Weight norm MSCOCO (rescaled)
Figure 3: In (d))()))d))(()), solid lines with different colors represent raw value of angular update of weights from all convolutional layer in the model; In (a), (d), training setting rigorously follows Goyal et al. (0117); 7); He et al. (2019) respectively; In (b), (c), weight norm is dividied by +10 as long as learning rate is divided by 10; In (c), (O, weight norm is computed on layer. 1.0.conv2 in Resnet50 backbone. Blue line represent original settings, orange line represent rescaled settings.
divided by k, equilibrium condition is broken, theoretical value of weight norm in new equilibrium condition should get //1/k smaller. But new equilibrium condition cannot be reached immediately (see (c), (f) in Figure 3) following corollary gives the least number of iterations to reach the new equilibrium condition.
Corollary 3.1. In SGD case with learning rate n, WD coefficient X, if learning rate is divided by k and unit gradient norm remains unchanged, then at least [[)og()]/(4>m)) iterations are required reach the new equilibrium condition; In SGDM case with momentum coefficient a, then at least [[lo(()) a4)]/44))]] iterations are required to reach the new equilibrium condition.
Corollary 3.1 also implies SGD/SGDM with smaller learning rate requires more iterations to reach new equilibrium condition. Hence, in second learning rate stage in Imagenet experiments, angular update can reach its new theoretical value within 15 epochs, but during last two learning rate stages of Imagenet/MSCOCO experiments, SGDM cannot completely reach new equilibrium by the end of training procedure. As result, we observe empirical value of angular update seems smaller than its theoretical value. Based on our theorem, we can bridge the gap by skipping the intermediate process from old equilibrium to new one. Specifically, when learning rate is divided by k. norm of scale-invariant weight is also divided by VE, SGDM can reach new equilibrium immediately. (xperiments((b).e) in Figure 3) show this simple strategy can make angular update always approx- imately equal to its theoretical value across the whole training process though learning rate changes.
CONCLUSION
In this paper, we comprehensively reveal the learning dynamics of DNN with normalization, WD, and SGD/SGDM, named as Spherical Motion Dynamics (SMD). Different from most related works (yan Laarhoven 2017; Hoffer et al.) 2018,Chiley et al. 2019; Li & Arora 2020p, we directly explore the cause of equilibrium. Specifically, we introduce the assumptions that can lead to equi- librium condition, and show these assumptions are easily satisfied by practical implementation of DNN; Under given assumptions, we prove equilibrium condition can be reached at linear rate, far before optimization has converged. Most importantly, we show our theorem is widely valid, they can be verified believe our theorems on SMD can bridge the gap between current theoretical progress and practical usage deep learning techniques.
of most challenging computer vision tasks, beyond synthetic datasets. We

---

Under review as a conference paper at ICLR 2021
REFERENCES
Sanjeev Arora, Zhiyuan Li, and Kaifeng Lyu. Theoretical analysis of auto rate-tuning by batch normalization, In International Conference on Learning Representations, 2019. URL https:
//openzeview.co net htt////ooummndd/dddd rkx nA9FX
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.064450, 2016.
Yoshua Bengio and Yann LeCun. Scaling learning algorithms towards AI. In Large Scale Kernel Machines. MIT Press, 2007.
Yongqiang Cai, Qianxiao Li, and Zuowei Shen. quantitative analysis of the effect of batch nor- malization gradient descent. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Pro- ceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, Long Beach, California, USA, 09-15 Jun 2019. PMLR. URL m
http::/proceed:: pte.sssssssssssse/seeeeep.e.sses (ptpes//////ca11..hhr
Vitaliy Chiley, Ily Sharapov, Atli Kosson, Urs Koster, Ryan Reece, Sofia Samaniego de la Fuente, Vishal Subbiah, and Michael James. Online normalization for training neural networks. In Ad- vances in Neural Information Processing Systems 32, pp. 8433- 8433-8443. Curran Associates, Inc., 2019.
Yonatan Dukler, Quanquan Gu, and Guido Montúfar. Optimization theory for relu neural networks trained with normalization layers. In International conference on machine learning, 2020.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint Sxiv:1706.026777 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference computer vision, pp. 2961-2969, 2017.
Kaiming He, Ross Girshick, and Piotr Dollar. Rethinking imagenet pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
Elad Hoffer, Ron Banner, Itay Golan, and Daniel Soudry. Norm matters: efficient and accurate normalization schemes in deep networks. S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 2160 21700 Curran Associates, Inc., 2018
Sergey loffe. Batch renormalization: Towards reducing minibatch dependence in aatch-normalized models. In Advances in neural information processing systems, pp. 1945-1953. 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In JCML, pp. 448 456, 2015.
Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kemnl: Convergence and generalization neural networks. In S. Bengio, H. Wallach, H. Larochelle, Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 8571-8580. Curran Associates, Inc., 2018. URL http: /ttp:/t::/ppaee http:/pperr...ee nips cttp.ccccpppppee wwtp:as..pppccccppppppppppaepp and www.geeeeeaaaieee
pdf
Alex Krizhevsky and Hinton Geoffrey. Learning multiple layers of features from tiny images. 2009.
Aitor Lewkowycz and Guy Gur-Ari. On the training dynamics of deep networks with / regular- ization. Advances in Neural Information Processing Systems, 33, 2020.
10

---

Under review as a conference paper at ICLR 2021
Xiang Li, Chen Shuo, and Yang Jian. Understanding the disharmony between weight normalization family and weight decay. In AAAI, 2020a.
Zhiyuan Li and Sanjeev Arora. An exponential learning rate schedule for deep learning. In Interna- tional Conference on Learning Representations, 2020. URL htttps://oppeeeeiiiiiii net/
Eorum?id=rJ@@Ten
Zhiyuan Li, Kaifeng Lyu, and Sanjeev Arora. Reconciling modern deep learning with traditional optimization analyses: The intrinsic learning rate. Advances in Neural Information Processing Systems, 33, 2020b.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Tsung-Yi Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference com- puter vision and pattern recognition, pp. 2117-2125, 2017.
Ping Luo. Xinjiang Wang, Wenqi Shao, and Zhanglin Peng. Towards understanding regularization in batch normalization. In International Conference on Learning Representations, 2019. URL
https: http:///pppeevviieee net forummid
Ningning Ma, Xiangyu Zhang, Hai-Ta Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 116-131, 2018.
Chao Peng, Tete Xiao, Zeming Li, Yuning Jiang, Xiangyu Zhang, Kai Jia, Gang Yu, and Jian Sun. Megdet: A large mini-batch object detector. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6181-6189, 2018.
Boris T Polyak. Some methods of speeding up the convergence of iteration methods. USSR Com- putational Mathematics and Mathematical Physics, 4(5):1-17, 1964.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 11(3))211-252, 2015.
Tim Salimans and Durk Kingma. Weight normalization: simple reparameterization to accelerate training of deep neural networks. In Advances in Neural Information Processing Systems, pp. 901-909, 2016.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018.
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How Does Batch Nor- malization Help Optimization? arXiv e-prints, art. arXiv:1805.10 1604, May 2018.
Wenqi Shao, Tianjian Meng, Jingyu Li, Ruimao Zhang, Yudian Li, Xiaogang Wang, and Ping Luo. Ssn: Learning sparse switchable normalization via sparsestmax. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 143-451, 2019.
Saurabh Singh and Shankar Krishnan. Filter response normalization layer: Eliminating batch de- pendence in the training of deep neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in- gredient for fast stylization. arXiv preprint arXiv: 1607.00000. 2016.
Twan van Laarhoven. L2 regularization versus batch and weight normalization. In Advances in Neural Information Processing Systems. 2017.
11

---

Under review as a conference paper at ICLR 2021
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European Conference Computer Vision (ECCV), pp. 3-19, 2018.
Junjie Yan, Ruosi Wan, Xiangyu Zhang, Wei Zhang, Yichen Wei, and Jian Sun. Towards stabi- lizing batch statistics in backward propagation of batch normalization. In International Confer- ence on Learning Represenlations, 2020. URL httph:t/////cc fttt:s://openrevie..eese net/ fortmt/foum??r ?id= SkgGjRVKDS
Guodong Zhang, Chaoqi Wang, Bowen Xu, and Roger Grosse. Three mechanisms of weight decay regularization. In International Conference on Learning Representations, 2019, URL https:
//openreview. net/forum?id-BlL.. 3Rct?
12

---

Under review as a conference paper at ICLR 2021
A
MISSING DERIVATIONS
A.1 APPROXIMATION OF WEIGIIT NORM UPDATE
Now considering the update rule of SGD with WD:
0C
W1++1 17
+ Xana).
(23)
Dw w=
Since 0c
is perpendicular to we, have
w==w
OC
/ l|e0 n
+ xxwx))12
(24)
Er w=
OC Ow w=
= (1 1/0r111 + 1|
113
(25)
Therefore
0C
| - 1/w/|22
V (1 \n) 11 112 + +
113
||
w
(26)
Ow
(1 (
/w 113
1
ec De
||m,
12
w m
(27)
(1 2n) 1
1
0c
Well2
Now assume 1), À is extremely small, so that
ac
Oc
(1 1- w 11 + 1/2 (1 (1- ( | + n/2
11 12011 RC 2 + 1
| (28)
Ow
Ow utc
OC Ow w=
11 +
R 2| w/
(29)
Therefore, we have
11
ac ow
17
8C
||e+1 10004111211ww// x -2n||w;/1a+
112
-m/l/
12
21 w
w==
2| w. 2
Ow w==
(30)
where 0C twe 1==
is the unit gradient defined in Definition
B PROOF OF THBOREMS
Remark 4. In the following context, we will use the following conclusion multiple times: Vo, E IR,
f8 << 1,1=| < 1, then we have:
1 R 1 5,
(1 + (1+))² 1
VI+001+2
(31)
2' 1+8
B. PROOF OF LEMMA
Proof. Given wo E RP ((), since Wk > 0, C(wo) = L(kwo)) then we have
@((w)
aCkww
aL(w)
(32)
Ow
Ew
W==Moo
Ew
w==kwp
aL(kw)
0L(w)
1
aL(w)
w00
wo)
=
(33)
Ok
Or
dw 40=
13

---

Under review as a conference paper at ICLR 2021
B.2 PROOF OF THEOREM 1
Lemma 2. If the sequence { satisfie
L
2t 2 at:-1 +
(34)
I1-1
where 21 > 0, L > 0
Then, we have
L 2
L
It 2 V a
xl
(35)
1-a
Proof. If x 2 V I-a since v LL/(1-0) (1-a) 2 2v aL then we have
4 > ay 1-a VL/(1 - - a
L レ+
L
T++1 2 03t +
(36)
Ft
which means Wk > xR 2 v L/(1 a). If LL(11 a). then have
Vi-a ViLa- z+t1 S (a VL(1-a) a)
I
L
(( 1-a xt) a(1 1 xt).
(37)
24
Therefore, by deduction method we have if oT
< TT/(1 (x), then 14 E [1, T1, 1]. we have
< 84 < T++1
< Xx <
(38)
1
L
xz)
< a'11()
x1).
(39)
1-a
1-a
In summary, we have
L at1
L
Ft 1 1-a
|V
1-a
x1
(40)
Proof of Theorem 1. since (w..g:) gi) = 0. then have:
|6x 2022"
| 11 (1 m/a 11
(41)
Denote It as |w,, Le as
|g:l 12 and omit ((())² part. V. Eq. 41b is equivalent to
Then Le 1. E[L[
L.
Var(L,lay = E[L 12
Len²
Je+1 (1 2\n) +
(42)
21
According to Lemma
we have
Inj
bn
34
(1 - 22x))- -
(43)
2X
2X
log((v? (on((v2-1),, 1)
Log( (1011//47/44) (4//(4)))
[m/((4A)
43)
implies when t > 1 +
log(1 22n))
we have rt >
43
Now, denote xt as the
La 2X
then we have
Lnd
L, L 2
E[(x++1 -xa 2+) = E[((1 - 2\n
(x, x²) +
(44)
Lnd
E[(, 2 1e)"
(1 (1-2X1
)2 (2 x²)² 2 +
F,x²
x7
14

---

Under review as a conference paper at ICLR 2021
Ift> 1,, =1 (on(l22-Dy///((A))ooog (bn/((11) 43. implies 1og(1-2An)
43)
2L -V 2\m < 1 1221
Laj²
1 - 2
(45)
2t
Combining with assumption in Theorem 1, we have 1 - -2nm / 17 2An 0, which means
L
0 < 1 2
(46)
x: Ite
Combining with Eq.
(4).
(43
46),
if l > T(a, X, xm), we have
- (x0) (1 (xte 4V
E
-
(47)
Considering the expectation with respect to the distribution of 24, we have
E[x++1 - x²)² 2Xn) x²)² 4V X
(48)
Approximate (1 (1-20) 2222))² = 1 - 4Am, and iterate Eq. 48)
E8) fort - T(a, x, 1,,, times, we have
- x²)² + Vn²
Ex - x²)² < (1
(49)
T(a, A, 1,, is finite, and only depends on at X, 13, 1,,1,a0 xo. Hence we can set maz((1
4An) 'E([I, )²)² 0, T(a, X, 17, xo)). note B is finite by
L (1 (1=42)) +2(1 2+ min(2. bn/ (2X))
V+
Exi+1 E[(1-2Xn) 2 +2 2 L2
xt
(50)
therefore, Vt > 0, we have
B(x,-=")' x²)² (1 427) Vi²²
(51)
B.3 PROOF OF THEOREM 2
Lemma 3. Assume a, 8,8 (0,1), where << 1. Denote diag(1 1-a- 0, a²
23
1-a B A,
2a
hs (1-a)) -(-a)² (1-a)2 )T. 1)T Ife 1/1
2a a²
1
1-al then Vd E IR, have
43
II(A - =8 )d < (1-
||d|
(52)
1
Proof. Omit O(3²) part, we have
43
43
II(A (1--)3310022k T Jdl = (1 (1-1-1 + 22 +o (1+
-202
1-
(53)
(di + de + dd) - -2addz + ada
[d + (1 - 22² )d2?
[d] + (1 + a dd]]
1 + 8
+
- -11/2+ 2a
+ (a - -2a22222200 2d2d
2
2
2
a
a
a
5
2 ( + 20 +
-2a²
+
2
2
2
2 - 3d dg
2
(54)
15

---

Under review as a conference paper at ICLR 2021
Then have
4,3
+333)d + (a + 4,
5,6
[I(A - EB(1 - kk) " (1
+ +
1-a +
)dz (55)
1
2
Since E < 31
8
we have
8
1 1-a
43
α
+ 33E
<
(56)
1 1
4,a 4
5ßE
43
a² +
1
(57)
1-a a
2
1-a
Hence, we have
43
II(A - 1.A--8(11- a)² "aee"ddl
) < (1
||d|
(58)
a
Proof of Theorem 2. The update rule is
Wi+1 = we - 1/2r
9t
W; 77(00-111
Xwe)
|20,1
(59)
n1a Wr-1
Wt
+
9:
+ Xww)
1
||we||
= (1 - 11 + a)w,
1:10-11 9:1]-
Then we have
W+++1 112 = (1 - 1IÀ + w - 2a(1 + a- m),) W; + w12
+ +1g.:1127² + 2((r0))
(1 1/1 + w 2x(1 + a m))) w;; w) [
+11g:1127 2(a(w: + -)1111),
(60)
(1 - +
2x(1 (A, a² 1
hon²
|w
(Wi+1, w))) (1 a An)] awwe, wi-1)
(61)
Let X,, A, e denote:
/w:11 113
at
X, =
bt
(w,, WT
(62)
Ct
//200111
(1 +
Xn)
-2a + Xn)
a²
A =
1 + 0 Ar)
(63)
1
0
1
0 )
(64)
0
(65)
respectively. The Eq. 61 is formulated as a iterative map:
har?
X AX, +
(66)
exx
16

---

Under review as a conference paper at ICLR 2021
When An < (1 (a), the eigen value of A are all real number:
d (1+a-An)² + a1 An) 4a
2
+ 0(2 (67)
1-a
22 a,
(68)
- An) 4a
2a
A
a² +
Xyy
0(2
and they satisfy
0< < A2 =a x1 <1
(70)
Therefore, A can be formulated as
S S A,
(71)
where A is diagonal matrix whose diagonal elements are the eigen value of A: the column vector of S is the eigen vectors of A, note the fromulation of S, A are not unique. Specifically, we set A, S as
A1
0
0
=
A2
(72)
0
0
A
S
1+a-Ar An
1+a-Ar An
1ta-An
(73)
a+
at
a+
Moreover, the inverse of S exists, and can be explicitly expressed as
(a+Ai)A (2- -aa)
((a
\1
())-A))(A (1+0--
(A1 -
S-l
2a
20(0+ (Aa+))
20dj1g
(74)
(A1 (A--a)aa-a Ag
(Ar aaaalla- As)l (1a
A aa-alla-l Ag]
2 Aglia+
A
aa- (o-Ax)(A1->>) -Aa)
(A la (110--
As
let Y, S S- combining with Eq.66 66 we have
hRn
Y AY, +
S-le.
(75)
(5'e)
Combining with Eq. 73) and Eq.
74 and set Y. (ār, 2T)_ we rewrite Eq. 75
hanp
(a A1JA1
d++1
Xrā +
(76)
αg + b 1 de A1-a)(Ap As)
h1p1
20
b = ab,
(77)
ãi b +
(\1 a)) - 2xa)
hnn²
(a + >a)Ass
ē1+1 dzā= +
(78)
à; + by + E.
(a- >a) - >a)
Note ||w;|
ā be + ot Now we prove the following inequations by mathematical induction
b 0,
(79)
E7 > 0.
(80)
(a \1 )be
> (A1
(81)
ā₂ - b, + E7 > 0,
(82)
han ā + b + C5
ăet + be41 + E2t1, >
A1 + b + (x) +
(83)
17

---

Under review as a conference paper at ICLR 2021
Since the start point X₁ (1,)0)))1) (01 0), the start point Y =S-1 X1. Combining with Eq. (4). we have
2a²An An
b1 a) A3) 01
(84)
+ + a1 - a An)
- a- An
â = (As a) A1) t xn)
11)01.
(85)
1 - + Xn
by which we have (a- }1 > (Λ1 - As) Besides ã +b + = =e SY = = 01 >
0.
Suppose for T. Eq. (79).
hold, combining with Eq.77).
78
we can derive
br+1= ar-1 0, so Eq. 79). 80 hold for T + 1; Besides, we have
hEn?
2a²
(a - (a-A1)bb: A1)bb+ a(a \1)
är bt + ör
(a- A3)
(a + >3)d3
(86)
> Aa(Ai - λa)ōT +
(ax - Aa) Ør + by + öT
= (A1 - A3)ōr41,
thus Eq. 81 holds for t = T = 1. Sum Eq. 76. Eq.
Eq. 78
due to Eq. 81
we have
hn
Trt + br+1 + ort1 = A1r + abr + dzēr + äT + by + èT
(87)
hen²
A1 by + δT)
ār + by + pr
Eq. holds for T + 1, combining with the fact that är br + êr > O, we have är+1 + brt1 + 8rt1 > 0.
According to Lemma we can estimate the lower bound of â + be ā; when 14
log((V2-1)//h/(an) log (o][]l /bn/(4Ai))
log(1- (Am)
in
In(1 a)
āe + by z
(88)
2(1 A1)
41
Now we can analyze the expectation of distance(12 norm) between Y, = (ā,, GT) and the fixed
point Y (ā b0, )T) which satisfies
Lap
Y AY + (STe) S-Le.
(89)
Assume 2, = āz + b. + ot x- = 8" + b- + Ex > 0, then we have:
Laj²
ke (Y; Y*) (he L)n?
Y441 Y (A
k
(90)
24
where (ki, k2 ka) S-le. In the following context, we will omit the ( part since An & 1. k1, k2, ky can be approximated as
h 1
a +0(3),
(91)
20
l
(1 a)2 0(8),
(92)
a²
kg
(1-a)² + 0(3).
(93)
Then we have
ke (Y, (Y; x + E( - 2)2
E Y Y
Yi = [I(A
|
(94)
x7
18

---

Under review as a conference paper at ICLR 2021
The fixed point of Eq. (89) is computed as x² ā" + ~-
Lg
and have
V (2
An
1
known that if log((v2 1) (n/(())/4)))))) log(l] mo (1n/(42]) have
log(1 22m)
st >
(n(1-o)
41
therefore we
Lnp
2L 2A7
(95)
4V A
(96)
z
1(1 a)
According to assumption 3, we can prove
2L
(1-o)
1 0²
8
(97)
3
3
1 a
combining with Lemma
3 we have
4hg
4V 11
B[IlY++1 Y
1 <
(1
Y| -
(98)
1 a
1(1 a)
which implies
4xn
Vij + 4a + + 1(1-a)
B| 1 < (1
- Yr Y 1
(99)
1
where T [1 log((v2-1) /t/ (d^)) log(lllop
Therefore, similar to the proof of
(-Arn
theorem 1, 3B 0,
4xm
Vi + 4a + ta)) 1(1 x1)+
E Y41 1 (1
'B+
(100)
Recall ||w, 12 = e Yr, therefore
4An
7 441++22-000 + a
[[ 1/ - (10 < 3E - <
'B +
(101)
1(1-a)4
Remark 5. By Eq. 101 the variance of 1| is bounded by 3V 3V ( which is not small 1(1-a}
enough. But somehow reduce it: according Eq. 16)) if x; is close to its theoretical value, then (h, 1 1/2 (1--X hence variance of // can be bounded by 3V (144014400 4a*)
B.4 PROOF OF THEOREM 3
Proof. In SGD case, we have
(Wi+1, wi) -11 >xn) >x)]|w,,1l.
(102)
then we have
cos2 A. (W 1, w.)2
(1 2An)
(103)
|wrll3 ||0 112
[w
According to the definition of A. A, 2 0. and A, very close 0, hence we have
A sin A.
cos? A
(104)
102
(1 An)
(105)
12e+++1 1
24 Fe--1
(1 An)2
(106)
19

---

Under review as a conference paper at ICLR 2021
where ze: #++1 denotes ||2/|122 ||t+1ll2 respectively as in Eq. (42 Assume is sufficiently large
so that Ie; 21+1 are close to V 2X
Ln
the first order of Taylor series expansion of Eq. 106 at
It I++1 = x=
A 22 (1-Xn)² 1
Xn)
1[1++11-000 x²) (2ee x²]].
(107)
2/2Xn) x²
Reorganizing Eq. 107p. and applying Cauchy Inequality, we have
(1- (1-An)
A. v /2n
(x+)² [[1241 -x²) (xt x² )}²
(108)
8An
21
<
2[( x²)²² + (x++1 - x²)²]
(109)
8An
Ln
(Iz x²)² + (I141 x²)²
(110)
2
Combining with (5. and Eq. 1080, we have
E(x, - x²1² + B(x++1 - x11) s 4 .C V
E A - V 2Am 2
(111)
2Ln
L1
where C 2L, B, B defined in Eq.51 51p.
In SGDM case, the angular update can be computed by
A = sin A
(112)
cos2 A.
(113)
(we, w
(114)
|wel
11
|0,
bi
(115)
Ct
where (at,by be,ce) = (1|wg] 12, (w. W > W 12). According to the proof of theorem 2 when t
is sufficiently large, (ag, bi, ce) will be close to the fixed point of Eq. 14.666) (a" c²), where a*
c² = =(1 111 (1-a)(2- (2-a))2-12)) 1 )' 6*
L+a-Aa" a* Then the first order Taylor series expansion of Eq. 115 1ta
at (ar = a", a.b = b², ct- =c²) is
b2 Ct
A
(116)
2\m
V1+ a
An
)2 a"
2(b -b*)
c
(1
(117)
llaa
2x2xn
1
b*
Now substituting (at, be, (t) with (āe, by. (t) defined in Eq.75
75
76 77
78). Eq.
107
is rewritten
2 Vl+a
An 22 a
a)²
(1
(ā -)) +0)m0(m))))
(118)
l+a
2/2Xn
1+
a²
where (ā", 80, 2*) is the fixed point of E9.
Omit O(Ar), we have
2 1
1+a (1-
An
X(1 a)
1tal (1 (1-a) aa)
An
A
a (c- (-
1 1+a
81y
1+a a
Lm
(1 (1- a= a)*
Y
(120)
4L
20

---

Under review as a conference paper at ICLR 2021
where Y,, Y is defined in Eq. 75 89
respectively. According to Eq. 100). the square emror
A
2Am can be bounded by
2
(1 (1- a (1 a)4
E A
1
B Y - Y
(121)
1+a
aL
2 (1 1 )CC + (1- a1-a²)(1 a) Vn² 4a2 a*)
4Am
(122)
4.
11-aa)
4xm
V(1 - a²) + 4o. + +
(1 1-a yC+
(123)
4Llat
(1-a")(1-0) a
where
B, B is defined Eq.
B.5
PROOF OF COROLLARY 3.1
Proof of Corollary 3.1. In SGD case, and Eq. 42 we have
|w 12 > (1 2An) w
(124)
which means
|w > (1 - 2\n) (\ 11
(125)
On the other hand, know that when is divided by ki ww should be divided by vk reach the new equilibrium condition, therefore we have
|
1
Vk > (1- -2
(126)
||w,|1
Since An < 1, log(1 2An) = 22 thus
T log(k)
(127)
4An
In SGDM case, by Eq. (87) have
2Xn
W 112 > (1
)1,,,113
(128)
Similar to SGD case, have
log(k)(1 a))
T>
(129)
4hn
21

---

Under review as a conference paper at ICLR 2021
C EXPERIMENTS
C.I EXPERIMENTS ON SYTHETIC DATA
In this section we apply experiments on sythetic data to prove our claim section 4: theorem only requires expected square norm of unit gradient is locally steady, the expectations do not need to remain unchanged across the whole training process. The proof theorem implies square norm of weight is determined by the following iterative map:
Len
J++1 (1 2An) +
(130)
24
(0, 1), L, denotes the square of unit gradient norm. Hence we simulate I, with different type of h Results in Figure shows as long as the local variance of square norm of unit gradient is too much, and expectation of L, changes smoothly, weight norm can quickly converge to its theoretical value base on expectation of square norm of unit gradient.
where An
We also simulate SGDM case by following iteration map
Lan2
X = AX, +
e,
(131)
[0]
where A, Xx, is defined as Eq. 62
63
Simulation results shown Figure
C.2 COMPLEMENTARY IN MULTI- STAGE LEARNING RATE SCHEDULE
In this section we present complementary results in Multi-Stage Learning Rate Schedule experiment.
The plots of weight norm (empirical and predicted values) in multi-learning rate stage is shown in Figure 6 We also present the test performance of resent50/maskrenn on Imagenet/MSCOCO with multi-stage learning rate schedule mentioned in Section 5.2. We only provide complementary results for reference only. We do not intend to prove the advantages or disadvantages of rescaling strategy here, it is beyond the discussion of this paper.
Top Accuracy($5)
Standard Rescale
76.25
76.27
Table I: Performance of Resnet50 on Imagenet with multi-stage learning rate scheduler.
Abbbo
A/abooz
APbboz
Apmask
A AP50 Pmmakk
A APpaas Pmnask
Standard
39.25
58.88
42.95
35.16 34.49
56.09
37.48
Rescale
38.38
57.98
42.31
55.27
36.71
Table 2: Performance of Resnet50 on Imagenet with multi-stage learning rate scheduler in Section.
C.3 RETHINKING LINEAR SCALING PRINCIPLE IN SPIERICALL MOTION DYNAMICS
In this section, we will discuss effect of Linear Scaling Principle (LSP) under view SMD. Linear Scaling Principle is proposed by Goyal et al. (2017) to tune the learning rate y with batch size B by 7 x B. The intuition of LSP is if weights do not change too much within iterations, then k iterations of SGD with learning rate 7) and minibatch size B (Eq. (32)) can be approximated
by a single iteration of SGD with learning rate kn and minibatch size k: (3q.133
133
0L
Wr+k
Wt - -1 Σ
+ >wW++));
(132)
B
Om
JKk
REB
aL
W1+1
wi
kan
+ Aww))
(133)
KB
Fuo 1er.
jck
22

---

Under review as a conference paper at ICLR 2021
Goyal al. (2017) shows that combining with gradual warmup, LSP can enlarge the batch size up to (292 x 32) without severe degradation on ImageNet experiments.
LSP has been proven extremely effective in wide range of applications. However, from the perspec- tive SMD, the angular update mostly relics on the pre-defined hyper-parametens and is hardly affected by batch size. To clarify the connection between LSP and SMD, explore the learning dynamics of DNN with different batch size by conducting extensive experiments with ResNet50 on ImageNet, the training settings rigorously follow [Goyal et al. (2017): momentum coefficient is = 10-4; Batch size is denoted by B; learning rate is initialized as 256 0.1; Total training epoch is 90 epoch, and learning rate is divided by at 30, 60, 80 epoch respectively.
10 10-4, WD coefficient
The results of experiments(Figure) 8) suggests that the assumption of LSP does not always hold in practice because of three reasons: first, the approximate equivalence between a single iteration large batch setting, and multiple iterations in small batch setting can only hold pure SGD formulation, but momentum method is far more commonly used; Second, according Theorem 2 the enlargement ratio angular update is only determined by increase factor of learning rate. Figure7shows in practice, the accumulated angular update <(w,, W(++) in small batch batch setting much larger than angular update <(w, wx+1) a single iteration in larger batch setting when using Linear Scaling Principle; Third, even in pure SGD cases, the enlargement of angular update still relies on the increase of learning rate, and has no obvious connection to the enlargement of gradient's norm when equilibrium condition is reached (see Figure
In conclusion, though LSP usually works well in practical applications, SMD suggests we can find more sophisticated and reasonable schemes tune the learning rate when batch size increases.
C.4 SPHERICAL MOTION DYNAMICS
DIFFERENT NETWORK STRUCTURES
We also verify our theory on other commonly used network structures (MobileNet-V2 (Sandler jtt al., 2018), ShuffleNet-V2 V2+ (Ma et al. (0188) with standard training settings. The results is shown in Figure
23

---

Under review as a conference paper at ICLR 2021
20
18
20.0
0.2
12
5.-
arorical
10
50000 15000 120000 250300 175000
120000 339000 200000
(a) L 10
13
22
12-
20
16
14
ergincal wake
shaaret ca valas
25300 scrco reeeo t00000 120000 190300 E7oino z000on
25260 s0000 meeeo mnnoo 125000 199390 175000 m0000
(b)
10
11.5
100
(c) T. 10 sin( (
10
(d) J., 10 sin( (6=)
Figure 4: Simulation of SGD (Eq. 130 ),
0.1, x 0.001. zo = 10, ~ U(-3,3) Orange
lines represent the square of unit gradient norm; blue solid lines represent simulated value of weight norm square; black dashed lines represent theoretical value of weight norm square
24

---

Under review as a conference paper at ICLR 2021
20.0
0.2
30
00-
5.-
arorical
10
50000 15000 120000 250380 175000
120000 339000 200000
(a) L 10
13
12-
40
omgincal wake
shaaret ca valas
25300 scrco reeeo t00000 120000 190300 Eroino X000on
25200 s0000 meeeo mn00o 125000 199300 1P5000 200000
(b) L.
10
11.5
60
100
30
(c) T. 10
sin(
(
70
60
10
(d) L., 10 sin( (6
Figure 5: Simulation of SGDM (Eq. 131 7 x= 0.001, vo 10, ~ U(-3, 3), a = 0.9.
Orange lines represent the square of unit gradient norm; blue solid lines represent simulated value of weight norm square; black dashed lines represent theoretical value of weight norm square
25

---

Under review as a conference paper at ICLR 2021
20
1613
23
168
208ch
gpcch
(a) 110112 and h in standard settings
(b) 11@l12 and h in rescaled settings
Standard
Theoretical value(Standard)
Rescale
10
Theoretical value(Rescale)
8
Epoch
(c) Weight norm in Imagenet with theoretical value
Figure 6: 1|0.112 hy and norm of weight from layer: 1.0.com22 in Resnet50 backbone. (a). present 11 h in multi-state learning rate schedule experiments discussed Section 5.2. semitranspar- ent line represents the raw value of |@:11 and hT, solid line represents the averaged value within consecutive 200 iterations to estimate the expectations E l0:1 and Eh, (c) presents the empirical value of weight norm and its theoretical value in standard and rescaled cases. The theoretical value computed by estimated expectations Eh, in (a). (b) respectively.
26

---

Under review as a conference paper at ICLR 2021
2-256
s--0 Alm my.
1.2
1.0
Koqpler Soda
(a) B 256 versus 1024
(b) B 256 versus 4096
Figure 7: Angular update of weights from layerI.0.com/ in ResNe150. The blue lines represent the angular update of weights within a single iteration in when batch settings is B 1024(4096); The red lines represent the accumulated angular update within 4(16) iterations in smaller batch setting(B = 256).
1.95
3.00
bnargiret betto Creaiin
2.75
2.20
3.75
40
Goocr
Rooch
(a) lgg (1024)
[gg (4098)
(b)
Tg (ot
(200)
Figure 8: Enlargement ratio of gradients" norm of weights from layerl 00.oon22 when batch size increases.. (k) represents the gradient's norm computed using A samples(not average).
27

---

Under review as a conference paper at ICLR 2021
26-3
16-22
1.4
1.0
Anguar Urdkne
(a) Angle Update of MobileNet-V2
(b) Angular Update of ShufleNea-V24
Figure 9: The angular update A, of MobileNet-V2 (Sandler et al. 2018) and ShuffleNet-V2+ (Ma et al., 2018). The solid lines with different colors represent all scale-invariant weights from the model; The dash black line represents the theoretical value of angular update, which is computed by
2An V 1+a Learning rate 7 is initialized as 0.5, and divided by 10 at epoch 30, 60, 80 respectively; WD
coefficient x 10 10-5; Momentum parameter a is set as 0.9.
28


Reviews:
[{"review_id": "r-vhaiM03zD", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The main goal of the paper is to establish theoretically some previous known results that for scale invariant networks the weight norm has a fixed point with ||w||^4=eta/lambda ||\\tilde{g}|| . They also discuss the angular update, which because of scale invariance is basically equivalent to arccos (1-eta lambda) |w_t|^2/|w_t+1|^2 and it thus comes mainly from the gradient. They have some experiments which they compare with the predicted equilibrium values for the angular update/ weight norm. \n\nThe theorems are the main results and I don't think they are very powerful given what is known in the literature (the van Laarhoven paper but also the more recent Li-Arora and Lewkowycz-Gurari where the relevant eta lambda time scale has been shown to be characterize the convergence rate). Also theorem 3 assumes that the normalized gradient norm is constant through training, but it actually changes a lot. \n\nThe experiments do not scale invariant networks (even if they have batch-norm, the last layer for example is clearly not scale invariant) so it is not that clear why the theory applies to that case, and the authors should probably discuss this. \nIn the learning rate decay setup, the experiments don't include the equilibrium value for the weights and it would be good to include them, and it does not seem like the equilibrium analysis is valid there as the authors discuss. The authors propose to rescale the weights together with decaying the learning rate, it would be good if they could mention how this changes the model validation accuracy. \n\nI find it really intriguing that in figures 2ef , the weight norm tracks so well the equilibrium one (which by definition has \\Delta |w|=0) but at the same time it, the weight norm changes considerably? Could the authors comment on how this is possible? I suspect that this is because the predicted one is actually really noisy due to batch noise (as shown in figures 2ad) and thus the system is never at equilibrium. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "e6RW7wkfGVs", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tried to analyze gradient descent with weight decay and momentum for scale-invariant networks. Convergence rate regarding the norm of the iterate is provided.\n\nI think this paper has several weaknesses and hence do not recommend accepting the paper at the current point.\n\n1. The paper is poorly written. The analysis and arguments are very poorly explained.\n\nPage 2:\n\n\"If $w_t \\approxeq w_{t+1}$, then the weight norm can be approximated as $\\|w_t \\|_2 \\approxeq $...\"\n--- Why is that?\n\n\"while the gradient component provided by WD always tends to reduce weight norm.\" \n--- Why? Is there any proof to support this argument?\n\nThe authors say \"one can obtain\" equality (4). But why (4) holds? it is very unclear and more explanation is necessary\n\nThe authors say \"one can easily speculate from eq (5) the magnitude of update in SGDM cases should be ...\" but the reasoning is unclear.\n\n\nPage 3:\n\"Eq.(7) implies gradient norm is influenced by weight norm, but weight norm does not affect the output of DNN\" \n--- Why?\n\nHow to derive (8)?\n\n\"If \\| w_t \\| \\approxeq \\| w_{t+1} \\|_2, one can obtain (10)\" \n--- What do you exactly mean when you write \"\\| w_t \\| \\approxeq \\| w_{t+1} \\|_2\"? and why the condition leads to (10)?\n\nThe authors say \"one can derive\" (12). But again this is unclear.\n\n\"From Eq.(7) we can infer that increasing weight norm can lead to smaller gradient norm if unit gradient norm is unchanged.\" \n--- This needs more explanation.\n\n\"Arora et al. (2019) proves that full gradient descent can avoid such problem and converge to a stationary point...\" \n--- To avoid which problem?\n\n\"Besides, practical implementation suggests training DNN without WD always suffers from poor generalization\" \n--- This needs references. \n\nPage 4:\n\n\"We can easily derive (15)\" How? Also, what does $\\approxeq$ on (15) mean? Can you give a precise statement?\n\n\"Notice that the core assumption mentioned above is unit gradient norm is unchanged. In fact this assumption can solve the contradiction we present in Section 1: the convergence of weight norm is not equivalent to convergence of weight, steady unit gradient norm can also make weight norm converge\" \n--- I found the paragraph hard to parse. What do you mean by \"unit gradient norm is unchanged\"? Of course, convergence of weight norm is not equivalent to convergence of weight. So why is it a \"contradiction\"?\n\n\n2. The proof is not rigorous and the writing of the proof is very sloppy.\n\nI believe the proof in the appendix is not rigorous and I feel unease. The authors use the following phrases (not an exhaustive list) to skip a lot of detailed proof, which does not meet the standard of mathematical writing.\n\n\"we can easily prove\" (page 11)\n\n\"we can prove\" (page 13)\n\n\"can be formulated as\" (page 14)\n\n\"it's easy to prove\" (page 14)\n\n\"can be explicitly expressed\" (page 14)\n\n\"which it's easy to prove\"  (page 15)\n\n\"it's easy to obtain\" (page 16)\n\n\"therefore it is easy to derive\" (page 17) \n\n\n3. The theorems are not very insightful.\n\nWhat insight does the convergence of weight norm provide? for example, it doesn't rule out the case of converging to a bad local min.\n\nWhat is the meaning of the value $w^* = ( L \\eta / 2 \\lambda)^{1/4}$? Can you explain why w^* should have the dependency on each parameter?\n\nDoes the theorem say that it will converge to the value $w^*$ regardless of any initialized point?\n\nWhat does the condition $\\lambda \\eta << 1$ exactly mean?\n\nWhat does the condition $\\| \\tilde{g}_t \\|^2 > l$ mean? Why does it hold?\nWhen the iterate reaches at a stationary point, isn't it zero?\n\n\n4. Comparison to the related works are not clear.\n\nRemark 1 and 2 state the results of this paper are \"consistent\" with some related works but did not provide any elaboration. What results exactly are being compared with? If the results are consistent, then what is new here?\nI hope the authors can explain more.\n\n5. Experiment does not really support the theorems.\n\nIt seems to me that the authors didn't really show that the weight norm converges to $w^*$ of Theorem 1 or 2.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "7vkzTDeC4R4", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Unfortunately, I am not qualified enough in the literature of learning dynamics of the neural network to comment on the novelty and contribution of this paper. Therefore I may take the author’s word and would like to rely on the comment on other reviewers.\n\nIn general, the motivation of this paper is clear, since the theoretical result  in literature mainly relies on the assumption of equilibrium condition but does not discuss why this equilibrium can be reached. \n\nThe main contribution comes from  Theorem 1 and Theorem 2.  This paper proves that weight norm can converge at the linear rate under mild assumption . It also defines another index called angular update to measure the change of normalized neural network in a single iteration and prove its convergence with linear rate. Overall the theoretical result looks novel and the experimental result matches the theory.Hence, I lean toward accepting.\n\nMy only concern is whether this theory can help us to train the deep neural network or give us some insights to design certain layers of the neural network. But it maybe that the overall novelty/contribution outweights this concerns. Can the author further explain that?\n \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "5DzSQqsv4UG", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The equilibrium condition is important during the theoretical analysis of spherical motion dynamics. \nThis paper focuses on the reasonableness of the equilibrium condition. \nConcretely, the authors first show that the weight norm can converge to its theoretical result under some assumptions (with SGD, SGDM settings).\nFinally, the experiments verify their conclusions.\n\n### CONTRIBUTIONS\n\na) The analysis of the equilibrium condition is important. The authors analyzed the equilibrium conditions under different settings, including SGD, SGDM, angular. \n\nb) The problem is motivated well. The authors gave a detailed introduction to the question of why we need to analyze the equilibrium conditions.\n\nc) The authors conducted several experiments to show the reasonableness of the theorems/assumptions. And the theoretical findings agree well with empirical observations.\n\n### MAJOR CONCERNS\n\nI am still concerned with the explanations in Eqn.17. (Of course, the same concern in Eqn.20.)\nIn Eqn.17, the formula (left) is the MSE form, which is the combination of the bias and the variance. \nThe authors explained it like “the square of weight norm can linearly converge to its theoretical value in equilibrium, and its variance is bounded by the variance of the square of unit gradient norm multiplying the square of learning rate”. \nCould the authors tell us why the first term (vanish as t \\to \\infty) on the right hand is actually the bias? \nIn my view, one cannot say the estimator *converge* to its theoretical value if the right-hand does not converge to zero.\n\nI think the most important point in the paper is the convergence of the estimator to the theoretical value, which is also the thing the authors tried to verify in Figure2 (b, e). \nTo reach the conclusion, one needs to at least claim “V\\eta^2/l” is small. As shown in Figure2 (a), I do not think V is such small. \nAlso, the claims in the main text that “the learning rate \\eta is small” is not satisfying. I am not sure if I have skipped some important claims in the main text. \nI would like to raise my score if the authors can explain it better.\n\n### OTHER CONCERNS\n\na) Eqn.12 (as well as its statements) is not clear. It is still unclear to me why one needs to use an angular update. \n\nb) There are several typos (and also something like log->\\log, min->\\min) in the appendix. I believe the conclusions are mostly correct (although I did not check the details), but I still spend a hard time reading the appendix.\n\nc) In Theorem3, the assumption “when V=0” is a little bit strong and unrealistic (shown in Figure2(a)). The importance of the theorem will be larger if the authors could use a mild assumption.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies gradient descent with weight decay and momentum for scale-invariant networks. While this is an interesting research direction, the clarity of the paper suffers from poor writing. In its current form, I doubt this paper would have a large impact in the community. In addition, some reviewers pointed out that the analysis is not rigorous (some steps are missing, the authors re-use some unjustified steps from previous papers, ...). The authors claim these steps are obvious, I certainly don't think that's the case and at the very least, one would expect that detailed and rigorous derivations in the appendix if the lack of space is the real issue in the main paper.

I think the feedback provided by the reviewers is valuable and I strongly encourage the authors to take advantage of this feedback to improve their work and submit to another venue.


Decision: Reject