SPHERICAL MOTION DYNAMICS: LEARNING DYNAM-
Year: 2020

Abstract:
In this work, we comprehensively reveal the learning dynamics of neural network with normalization, weight decay (WD), and SGD (with momentum), named as Spherical Motion Dynamics (SMD). Most related works study SMD by focusing on “effective learning rate” in “equilibrium” condition, i.e. assuming the convergence of weight norm. However, their discussions on why this equilibrium condition can be reached in SMD is either absent or less convincing. Our work investigates SMD by directly exploring the cause of equilibrium condition. Specifically, 1) we introduce the assumptions that can lead to equilibrium condition in SMD, and prove that weight norm can approach its theoretical value in a linear rate regime with given assumptions; 2) we propose “angular update” as a substitute for effective learning rate to measure the evolving of neural network in SMD, and prove angular update can also approach to its theoretical value in a linear rate regime; 3) we verify our assumptions and theoretical results on various computer vision tasks including ImageNet and MSCOCO with standard settings. Experiment results show our theoretical findings agree well with empirical observations.

heading: 1 INTRODUCTION AND BACKGROUND
text: Normalization techniques (e.g. Batch Normalization (Ioffe & Szegedy, 2015) or its variants) are one of the most commonly adopted techniques for training deep neural networks (DNN). A typical normalization can be formulated as following: consider a single unit in a neural network, the input is X , the weight of linear layer is w (bias is included in w), then its output is
y(X;w; γ;β) = g( Xw − µ(Xw)
σ(wX) γ + β), (1)
where g is a nonlinear activation function like ReLU or sigmoid, µ, σ are mean and standard deviation computed across specific dimension of Xw (like Batch Normalization (Ioffe & Szegedy, 2015), Layer Normalization Ba et al. (2016), Group Normalization (Wu & He, 2018), etc.). β, γ are learnable parameters to remedy for the limited range of normalized feature map. Aside from normalizing feature map, Salimans & Kingma (2016) normalizes weight by l2 norm instead:
y(X;w; γ;β) = g(X w
||w||2 γ + β), (2)
where || · ||2 denotes l2 norm of a vector.
Characterizing evolving of networks during training. Though formulated in different manners, all normalization techniques mentioned above share an interesting property: the weight w affiliated with a normalized unit is scale-invariant: ∀α ∈ R+, y(X;αW ; γ, β) = y(X;w; γ, β). Due to the scale-invariant property of weight, the Euclidean distance defined in weight space completely fails to measure the evolving of DNN during learning process. As a result, original definition of learning rate η cannot sufficiently represent the update efficiency of normalized DNN.
To deal with such issue, van Laarhoven (2017); Hoffer et al. (2018); Zhang et al. (2019) propose “effective learning rate” as a substitute for learning rate to measure the update efficiency of normalized
neural network with stochastic gradient descent (SGD), defined as
ηeff = η
||w||22 . (3)
Joint effects of normalization and weight decay. van Laarhoven (2017) explores the joint effect of normalization and weight decay (WD), and obtains the magnitudes of weight by assuming the convergence of weight, i.e. if wt = wt+1, the weight norm can be approximated as ||wt||2 = O( 4 √ η/λ), where λ is WD coefficient. Combining with Eq.(3), we have ηeff = √ ηλ. A more intuitive demonstration about relationship between normalization and weight decay is presented in Chiley et al. (2019) (see Figure 1): due to the fact that the gradient of scale invariant weight ∂L/∂w (L is the loss function of normalized network without WD part) is always perpendicular to weight w, one can infer that gradient component ∂L/∂w always tends to increase the weight norm, while the gradient component provided by WD always tends to reduce weight norm. Thus if weight norm remains unchanged, or “equilibrium has been reached”1, one can obtain
wt −wt+1 ||wt||2
= √ 2ηλ ∂L/∂w
E||∂L/∂w||2 . (4)
Eq.(4) implies the magnitude of update is scale-invariant of gradients, and effective learning rate should be √ 2ηλ. Li & Arora (2020) manages to estimate the magnitude of update in SGDM, their
result is presented in limit and accumulation manner: if both limT−→∞RT = 1/T ∑T t=0 ||wt|| and
limT−→∞DT = 1/T ∑T t=0 ||wt −wt+1|| exist, then we have
lim T−→∞ DT RT = 2ηλ 1 + α . (5)
Though not rigorously, one can easily speculate from Eq.(5) the magnitude of update in SGDM cases should be √ 2ηλ/(1 + α) in equilibrium condition. But proof of Eq.(5) requires more strong assumptions: not only convergence of weight norm, but also convergence of update norm ‖wt+1 − wt‖2 (both in accumulation manner).
As discussed above, all previous qualitative results about “effective learning rate” (van Laarhoven, 2017; Chiley et al., 2019; Li & Arora, 2020) highly rely on equilibrium condition, but none of them explores why such equilibrium condition can be achieved. Only van Laarhoven (2017) simply interprets the occurrence of equilibrium as a natural result of convergence of optimization, i.e. when optimization is close to be finished, wt = wt+1, resulting in equilibrium condition. However, this interpretation has an apparent contradiction: according to Eq.(4) and (5), when equilibrium condition is reached, the magnitude of update is constant, only determined by hyper-parameters, which means optimization process has not converged yet. Li & Arora (2020) also notices the non-convergence of SGD with BN and WD, so they do not discuss reasonableness of assumptions adopted by Eq.(5). In a word, previous results about “effective learning rate” in equilibrium condition can only provide vague insights,
they are difficult to be connected with empirical observation.
In this work, we comprehensively reveal the learning dynamics of normalized neural network using stochastic gradient descent without/with momentum (SGD/SGDM) and weight decay, named as Spherical Motion Dynamics (SMD). Our investigation aims to answer the following question:
1”Weight norm remains unchanged” means ||wt||2 ≈ ||wt+1||2, Chiley et al. (2019) calls this condition as “equilibrium”, which will also be used in the following context of this paper. Note equilibrium condition is not mathematically rigorous, we only use it for intuitive analysis.
Why and how equilibrium condition can be reached in Spherical Motion Dynamics?
Specifically, our contributions are
• We introduce the assumptions that can lead to equilibrium condition in SMD, and justify their reasonableness by sufficient experiments. We also prove under given assumptions, equilibrium condition can be reached as weight norm approach to its theoretical value in a linear rate regime in SMD. Our assumptions show the equilibrium condition can occur long before the finish of the whole optimization;
• We define a novel index, angular update, to measure the change of normalized neural network in a single iteration, and derive its theoretical value under equilibrium condition in SMD. We also prove angular update can approach to its theoretical value in a linear regime along with behavior of weight norm. Our results imply that the update efficiency of SGD/SGDM on normalized neural network only depends on pre-defined hyper-parameters in both SGD and SGDM case;
• We verify our theorems on different computer vision tasks (including one of most challenging datasets ImageNet (Russakovsky et al., 2015) and MSCOCO (Lin et al., 2014)) with various networks structures and normalization techniques. Experiments show the theoretical value of angular update and weight norm agree well with empirical observation.
Recently, a parallel work (Li et al., 2020b) is published to analyze the equilibrium condition of normalized neural network with weight decay in SGD case. They model the learning dynamic of normalized network as a SDE, and propose the concept “intrinsic learning rate” (λe = λη) to measure the converge rate of weight norm and effective learning rate. Their main result, in which converging time to equilibrium in SGD case is O(1/(λη)), is consistent with our theory. The major difference between Li et al. (2020b) and our work is that all results from Li et al. (2020b) are qualitatively results or conjectures limited on SGD case. While our work derives the quantitative results on both SGD/SGDM cases, our analysis allows us to precisely predict the value of weight norm/angular update in large scale data experiments, and clarify the difference of approaching rate between SGD and SGDM case. Besides, Li et al. (2020b) conducts empirical study to discuss the connection between the equilibrium state and accuracy of trained model. Our work mainly focuses on the equilibrium condition itself, connection between equilibrium condition and performance of trained model is beyond our discussion in this paper. The experiments we present in main text are only for verification of our theory.
Our theory on equilibrium condition in Spherical Motion Dynamics implies equilibrium condition mostly relies on the update rules of SGD/SGDM with WD, and scale-invariant property. The cause of equilibrium condition is independent of decrease of loss or trajectory of optimization, but equilibrium condition turns out to significantly affect update efficiency of normalized network by controlling the relative update (Eq.(4)). We believe equilibrium condition is one of the key reason why learning dynamic of normalized neural network is not consistent with traditional optimization theory (Li et al., 2020b). We think it has great potential to study the leaning dynamic of normalized network or develop novel efficient learning strategy under the view of Spherical Motion Dynamics.

heading: 2 RELATED WORK
text: Normalization techniques Batch normalization(BN (Ioffe & Szegedy, 2015)) is proposed to deal with gradient vanishing/explosion, and accelerate the training of DNN. Rapidly, BN has been widely used in almost all kinds of deep learning tasks. Aside from BN, more types of normalization techniques have been proposed to remedy the defects of BN (Ioffe, 2017; Wu & He, 2018; Chiley et al., 2019; Yan et al., 2020) or to achieve better performance (Ba et al., 2016; Ulyanov et al., 2016; Salimans & Kingma, 2016; Shao et al., 2019; Singh & Krishnan, 2020). Though extremely effective, the mechanism of BN still remains as a mystery. Existing works attempt to analyze the function of BN: Ioffe & Szegedy (2015) claims BN can reduce the Internal Covariance Shift (ICS) of DNN; Santurkar et al. (2018) argue that the effectiveness of BN is not related to ICS, but the smoothness of normalized network; Luo et al. (2019) shows BN can be viewed as an implicit regularization technique; Cai et al. (2019) proves that with BN orthogonal least square problem can converge at linear rate; Dukler et al. (2020) proves weight normalization can speed up training in a two-layer ReLU network.
Weight decay Weight decay (WD) is well-known as l2 regularization, or ridge regression, in statistics. WD is also found to be extreme effective when applied in deep learning tasks. Krizhevsky & Geoffrey (2009) shows WD sometimes can even improve training accuracy not just generalization performance; Zhang et al. (2019) show WD can regularize the input-output Jacobian norm and reduce the effective damping coefficient; Li et al. (2020a) discusses the disharmony between WD and weight normalization. A more recent work Lewkowycz & Gur-Ari (2020) empirically finds the number of SGD steps T until a model achieves maximum performance satisfies T ∝ 1λη , where λ, η are weight decay factor and learning rate respectively, they interpret this phenomenon under the view of Neural Tangent Kernel (Jacot et al., 2018), showing that weight decay can accelerate the training process. Notice their result has no connection with equilibrium condition discussed in this work. Our results shows the cause of equilibrium condition can be reached long before neural network can get its highest performance.
Effective learning rate Due to the scale invariant property caused by normalization, researchers start to study the behavior of effective learning rate. van Laarhoven (2017); Chiley et al. (2019) estimate the magnitude of effective learning rate under equilibrium assumptions in SGD case; Hoffer et al. (2018) quantify effective learning rate without equilibrium assumptions, so their results are much weaker; Arora et al. (2019) proves that without WD, normalized DNN can still converge with fixed/decaying learning rate in GD/SGD cases respectively; Zhang et al. (2019) shows WD can increase effective learning rate; Li & Arora (2020) proves standard multi-stage learning rate schedule with BN and WD is equivalent to an exponential increasing learning rate schedule without WD. As a proposition, Li & Arora (2020) quantifies the magnitude of effective learning rate in SGDM case. But none of them have ever discussed why equilibrium condition can be reached. A recent work Li et al. (2020b) studies the convergence of effective learning rate by SDE, proving that the convergence time is of O(1/(λη)), where λ, η are weight decay factor and learning rate respectively. Their result can only provide intuitive understanding, and is limited on SGD case.

heading: 3 PRELIMINARY ON SPHERICAL MOTION DYNAMICS
text: First of all, we review the property of scale invariant weight, and depict Spherical Motion Dynamics (SMD) in SGD case. Notice except definitions, all intuitive statements or derivations in this section mostly comes from previous literature, they are not mathematically rigorous. We summarize them to provide background of our topic and preliminary knowledge for readers. Lemma 1. If w is scale-invariant with respect to L(w) , then for all k > 0, we have:
〈wt, ∂L ∂w ∣∣∣ w=wt 〉 = 0 (6)
∂L ∂w ∣∣∣ w=kwt = 1 k · ∂L ∂w ∣∣∣ w=wt . (7)
Proof can be seen in Appendix B.1. Lemma 1 is also discussed in Hoffer et al. (2018); van Laarhoven (2017); Li & Arora (2020). Eq.(7) implies gradient norm is influenced by weight norm, but weight norm does not affect the output of DNN, thus we define unit gradient to eliminate the effect of weight norm. Definition 1 (Unit Gradient). If wt 6= 0, w̃ = w/||w||2, the unit gradient of ∂L/∂w|w=wt is ∂L/∂w|w=w̃t .
Setting k as 1/||wt||2 inEq.(7), we have ∂L ∂w ∣∣∣ w=wt = 1 ||wt|| · ∂L ∂w ∣∣∣ w=w̃t . (8)
A typical SGD update rule without WD is
wt+1 = wt − η ∂L ∂w ∣∣∣ w=wt , (9)
if ||wt||2 = ||wt+1||2, dividing both side of Eq.(9) by ||wt||2, we have
w̃t+1 = w̃t − η ||wt||22 ∂L ∂w ∣∣∣ w=w̃t = w̃t − ηeff · ∂L ∂w ∣∣∣ w=w̃t . (10)
Eq.(10) shows effective learning rate can be viewed as learning rate of SGD on unit sphere Sp−1 (Hoffer et al., 2018). But effective learning rate still cannot properly represent the magnitude of update, since unit gradient norm is unknown. Therefore we propose the angular update defined below. Definition 2 (Angular Update). Assuming wt is a scale-invariant weight from a neural network at iteration t, then the angular update ∆t is defined as
∆t = ](wt,wt+1) = arccos ( 〈wt,wt+1〉 ||wt|| · ||wt+1|| ) , (11)
where ](·, ·) denotes the angle between two vectors, 〈·, ·〉 denotes the inner product.
According to Eq.(6), ∂L/∂w is perpendicular to weight w. Therefore, if angular update ∆t is small enough, it can be approximated by first order Taylor series expansion of tan ∆t, then we can see its connection between effective learning rate and unit gradient norm
∆t = tan(∆t) = η ||wt|| · || ∂L ∂w ∣∣∣ w=wt ||2 = ηeff · ∂L ∂w ∣∣∣ w=w̃t . (12)
Another deduction from Eq.(6) is that weight norm always increases because
||wt+1||22 = ||wt||22 + (η|| ∂L ∂w ∣∣∣ w=wt ||2)2 > ||wt||22. (13)
From Eq.(7) we can infer that increasing weight norm can lead to smaller gradient norm if unit gradient norm is unchanged. Zhang et al. (2019) states the potential risk that GD/SGD without WD but BN will converge to a stationary point not by reducing loss but by reducing effective learning rate due to increasing weight norm. Arora et al. (2019) proves that full gradient descent can avoid the risk and converge to a stationary point defined in Sp−1, but their results still require sophisticated learning rate decay schedule in SGD case. Besides, practical implementation suggests training DNN without WD always suffers from poor generalization (Zhang et al., 2019; Bengio & LeCun, 2007; Lewkowycz & Gur-Ari, 2020).
Now considering the update rule of SGD with WD:
wt+1 = wt − η( ∂L ∂w ∣∣∣ w=wt + λwt). (14)
We can approximate the update of weight norm by
||wt+1||2 ≈ ||wt||2 − λη||wt||2 + η2 2||wt||32 · || ∂L ∂w ∣∣∣ w=w̃t ||22. (15)
The derivation of Eq.(15) is presented in Appendix A.1. Eq.(15) implies WD can provide direction to reduce weight norm, hence Chiley et al. (2019); Zhang et al. (2019) points out the possibility that weight norm can be steady, but do not explain this clearly. Here we demonstrate the mechanism deeper (see Figure 1): if unit gradient norm remains unchanged, note “centripetal force” (−λη||wt||2) is proportional to weight norm, while “centrifugal force” ( η 2
2||wt||32 · || ∂L∂w ∣∣∣ w=w̃t
||22) is inversely proportional to cubic weight norm. As a result, the dynamics of weight norm is like a spherical motion in physics: overly large weight norm makes centripetal force larger than centrifugal force, leading to decreasing weight norm; while too small weight norm makes centripetal force smaller than centrifugal force, resulting in the increase of weight norm. Intuitively, equilibrium condition tends to be reached if the number of iterations is sufficiently large.
Notice that the core assumption mentioned above is “unit gradient norm is unchanged”. In fact this assumption can solve the contradiction we present in Section 1: convergence of wt leads to nonzero ||wt+1 − wt||2/||wt||2. Our analysis implies the convergence of weight norm is not equivalent to convergence of weight, steady unit gradient norm can also make weight norm converge, steady unit gradient norm does not rely on the fact that optimization has reached an optimum solution. But a problem about unit gradient assumption rises: unit gradient norm cannot remain unchanged during training in practice, it is not a reasonable assumption. In next section, we formulate this assumption in a reasonable manner, and rigorously prove the existence of equilibrium condition in SGD case. Discussion about SGD cannot be easily extended to SGDM case, because momentum is not always perpendicular to weight as unit gradient. But we can still prove the existence of equilibrium condition in SGDM case under modified assumptions.

heading: 4 MAIN RESULTS
text: First of all, we prove the existence of equilibrium condition in SGD case and provide the approaching rate of weight norm.
Theorem 1. (Equilibrium condition in SGD) Assume the loss function is L(X;w) with scaleinvariant weight w, denote gt = ∂L∂w ∣∣ Xt,wt
, g̃t = gt · ||wt||2. Considering the update rule of SGD with weight decay, wt+1 = wt − η · (gt + λwt) (16) where λ, η ∈ (0, 1). If the following assumptions hold: 1) λη << 1; 2) ∃L, V,∈ R+, E[||g̃t||22|wt] = L, E[(||g̃t||22 − L)2|wt] ≤ V ; 3) ∃l ∈ R+, ||g̃t||22 > l, l > 2[ 2λη 1−2λη ] 2L. Then
∃B > 0, w∗ = 4 √ Lη/(2λ), we have
E[||wT ||22 − (w∗)2]2 ≤ (1− 4λη)TB + V η2
l . (17)
Remark 1. The theoretical value of weight norm w∗ in Theorem 1 is consistent with the derivation of weight norm in equilibrium in van Laarhoven (2017), though van Laarhoven (2017) assumes the equilibrium condition has been reached in advance, hence van Laarhoven (2017) cannot provide the approaching rate and scale of bias/variance. The vanishing term ((1 − 4λη)TB) in Eq.(17) is consistent with the convergence time O(1/(λη)) presented in Li et al. (2020b).
Proof of Theorem 1 can be seen in Appendix B.2. It shows the square of weight norm can approach to its theoretical value in a linear rate regime (when vanishing term is larger than noisy term in Eq.(17)), and its variance is bounded by V η 2
l , which is empirically small. Now we discuss the reasonableness of assumptions in Theorem 1. Assumption 1 is consistent with commonly used settings; assumptions 2 and 3 imply the E||g̃t||22 remains unchanged within several iterations, and its lower bound cannot be far from its expectation.
We need to clarify assumption 2 further: we do not require the E||g̃t||22 must be constant across the whole training process, but only remains unchanged locally. On one hand, small learning rate(η) can guarantee E||g̃t||22 changes slowly; on the other hand, when E||g̃t||22 changes, it means that square of weight norm will approach to its new theoretical value as Theorem 1 describes. Experiment result in Figure 2 strongly justifies our analysis. We also conduct extensive experiments to verify our claim further, please refer to Appendix C.1.
Now we extend Theorem 1 to SGDM case. SGDM is more complex than SGD since momentum is not always perpendicular to the weight, therefore we need to modify assumptions.
Theorem 2. (Equilibrium condition in SGDM) Considering the update rule of SGDM (heavy ball method (Polyak, 1964)):
vt = αvt−1 + gt + λwt (18) wt+1 = wt − ηvt (19)
where λ, η, α ∈ (0, 1). If following assumptions hold: 4) λη << 1, λη < (1 − √ α)2;
5) Define ht = ||gt||22 + 2α〈vt−1, gt〉, h̃t = ht · ||wt||22. ∃L, V,∈ R+, E[h̃t|wt] = L, E[(h̃t − L)2|wt] ≤ V ; 6)∃l ∈ R+, ht > l, l > 2[ 6λη(1−α)3(1+α)−8λη(1−α) ]
2L. then ∃B > 0, w∗ = 4 √ Lη/(λ(1− α)(2− λη/(1 + α))), we have
E[||wT ||22 − (w∗)2]2 ≤ 3B · (1− 4λη
1− α )T +
3V η2(1 + 4α2 + α4)
l(1− α)4 , (20)
Remark 2. So far, no other work rigorously prove equilibrium condition can be reached in SGDM cases. Even the most relevant work (Li et al., 2020b) only provides their conjecture on approaching rate of weight norm in SGDM, they speculate that the time of approaching to equilibrium should be O(1/(λη)), same order as approaching time in SGD case, their conjecture cannot provide further insight. While our results (vanishing terms in Eq.(17), (20) respectively) can clearly reflect the difference: the approaching rate of SGDM should be 1/(1 − α) times faster than rate of SGD with same ηλ. α is usually set as 0.9, so SGDM can reach equilibrium condition much faster than SGD.
Proof can be seen in Appendix B.3. Like assumption 1, assumption 4 is also satisfied for commonly used hyper-parameter settings. Besides, λη < (1 − √ α)2 is also mentioned in Li & Arora (2020) for other purpose; Assumption 5 shows not unit gradient gradient norm ||g̃t||22 but an adjusted value h̃t dominates the expectation and variance of the weight norm square. We empirically find the expectation of 〈vt−1, gt〉 is very close to 0, therefore the behavior of h̃t is similar to that of ||g̃t||22 (see (d) in Figure 2), making square of weight norm approaching its theoretical value in SGDM case. We leave theoretical analysis on h̃t as future work. As for assumption 6, commonly used settings (ηλ << 1) can make 2[ 6λη(1−α)3(1+α)−8λη(1−α) ]
2 as an very small lower bound for l/L. The experiments on justification of assumptions 4,5,6 can be seen in Figure 2 and appendix C.1. Comparing with Eq.(17) and Eq.(20), it implies that with same η, λ, SGDM can reach equilibrium condition much faster than SGD, but may have a larger variance, our experiments also verify that(see (b), (e) in Figure 2).
Since we have proven weight norm will approach to its theoretical value in SMD in a linear rate regime, we can derive the theoretical value of angular update ∆t and its variance. Theorem 3. (Theoretical value of Angular Update) In SGD case, if assumptions in theorem 1 holds, then ∃C > 0, we have
E(∆T − √
2λη)2 < (1− 4ηλ)TC + V Ll
(21)
In SGDM case, if assumptions in theorem 2 holds, ∃C > 0, we have E(∆t − √ 2λη
1 + α )2 < (1− 4ηλ 1− α )TC + (1− α2)(1 + 4α2 + α4)V 4Llα4
(22)
Remark 3. Theoretical value of angular update in Theorem 3 is consistent with Eq.(4, 5) from Chiley et al. (2019); Li & Arora (2020) respectively. Notice the variance term in Eq.(21,22) is of O(V/Ll), it is too large comparing with its empirical value, we leave it as a future work to improve the bound of variance term. Though connection between performance of neural network and equilibrium condition is beyond the main discussion of this paper, the findings of a parallel work (Li et al., 2020b) can inspire us the possible advantage of momentum: Li et al. (2020b) interprets that smaller λη can get higher performance when training DNN, but the approaching rate (O(λη)) is slow; larger λη has faster approaching rate but it can only get bad performance. Inspired by Li et al. (2020b), we suspect that ∆t ∝ √ λη is highly correlated to the performance of DNN: smaller angular update in equilibrium condition can lead to better performance of DNN. According to theorem 3, with same λη, momentum method has faster approaching rate(λη/(1− α)) and smaller angular update ( √ 2λη/(1 + α)) than pure SGD. This means momentum method can simultaneously accelerate training process and improve the final performance, comparing pure SGD method. We leave this conjecture as a future work.
Proof of theorem 3 is shown in Appendix B.4. According to theorem 3, the theoretical value of angular update and its approaching rate almost only depends on hyper-parameters: weight decay factor λ, learning rate η, (and momentum factor α). It implies update efficiency of scale-invariant weights within a single step is totally controlled by predefined hyper-parameters in equilibrium condition, regardless other attributes of the weights (shape, size, position in network structure, or effects from other weights). That’s the key reasons why we propose Angular Update (Eq.(11)) to replace effective learning rate (Eq.(3)): effective learning rate can only reflect the influence of weight norm, it cannot reveal how unit gradient norm affect the relative update (see Eq.(10)), while angular update in equilibrium condition implies that both weight norm and unit gradient norm do not affect the scale of update within a single iteration, only hyper-parameters (learning rate, weight decay factor, momentum factor) matter. Our experiment results strongly prove our claim (see Figure 3(a), 3(b), 3(d), 3(e)).

heading: 5 EXPERIMENTS
text: In this section, we show that our theorems can agree well with empirical observations on ImageNet (Russakovsky et al., 2015) and MSCOCO (Lin et al., 2014). We conduct experiments in two cases. In the first case we train neural network with fixed learning rate to verify our theorems in SGD and SGDM, respectively; in the second case we investigate SMD with more commonly used settings, multi-stage learning rate schedule.

heading: 5.1 FIXED LEARNING RATE
text: With fixed learning rate, we train Resnet50 (He et al., 2016) with SGD/SGDM on ImageNet. Learning rate is fixed as η = 0.2; WD factor is λ = 10−4; with SGDM, the momentum factor is α = 0.9. Figure 2 presents the unit gradient norm square, weight norm, and angular update of the weights from LAYER.2.0.CONV2 of Resnet50 in SGD and SGDM cases, respectively. It can be inferred from Figure 2 the behavior of ||g̃t||22, h̃t and hyper-parameter settings satisfy our assumptions in Theorem 1 and 2, therefore theoretical value of weight norm and angular update agree with empirical value very well. We also can observe SGDM can achieve equilibrium more quickly than SGD. According to Eq.(17),(20), the underlying reason might be with same learning rate η and WD factor λ, approaching rate of SGDM (1− λη1−α ) is smaller than that of SGD (1− λη).

heading: 5.2 MULTI-STAGE LEARNING RATE SCHEDULE
text: Now we turn to study the behavior of angular update with SGDM and multi-stage leanring rate schedule on Imagenet (Russakovsky et al., 2015) and MSCOCO (Lin et al., 2014). In ImageNet classification task, we still adopt Resnet50 as baseline for it is a widely recognized network structure.The training settings rigorously follow Goyal et al. (2017): learning rate is initialized as 0.1, and divided by 10 at 30, 60, 80-th epoch; the WD factor is 10−4; the momentum factor is 0.9. In MSCOCO experiment, we conduct experiments on Mask-RCNN (He et al., 2017) benchmark using a Feature Pyramid Network (FPN) (Lin et al., 2017), ResNet50 backbone and SyncBN (Peng et al., 2018) following the 4x setting in He et al. (2019): total number of iteration is 360, 000, learning rate is initialized as 0.02, and divided by 10 at iteration 300, 000, 340, 000; WD coefficient is 10−4.
There appears to be some mismatch between theorems and empirical observations in (a), (b) of Figure 3: angular update in the last two stages is smaller than its theoretical value. This phenomenon can be well interpreted by our theory: according to Theorem 1, 2, when equilibrium condition is reached, theoretical value of weight norm satisfies ||wt||2 ∝ 4 √ η λ , therefore when learning rate is
divided by k, equilibrium condition is broken, theoretical value of weight norm in new equilibrium condition should get 4 √ 1/k smaller. But new equilibrium condition cannot be reached immediately (see (c), (f) in Figure 3), following corollary gives the least number of iterations to reach the new equilibrium condition. Corollary 3.1. In SGD case with learning rate η, WD coefficient λ, if learning rate is divided by k, and unit gradient norm remains unchanged, then at least d[log(k)]/(4λη)e iterations are required to reach the new equilibrium condition; In SGDM case with momentum coefficient α, then at least d[log(k)(1− α)]/(4λη)e iterations are required to reach the new equilibrium condition.
Corollary 3.1 also implies SGD/SGDM with smaller learning rate requires more iterations to reach new equilibrium condition. Hence, in second learning rate stage in Imagenet experiments, angular update can reach its new theoretical value within 15 epochs, but during last two learning rate stages of Imagenet/MSCOCO experiments, SGDM cannot completely reach new equilibrium by the end of training procedure. As a result, we observe empirical value of angular update seems smaller than its theoretical value. Based on our theorem, we can bridge the gap by skipping the intermediate process from old equilibrium to new one. Specifically, when learning rate is divided by k, norm of scale-invariant weight is also divided by 4 √ k, SGDM can reach in new equilibrium immediately. Experiments((b),(e) in Figure 3) show this simple strategy can make angular update always approximately equal to its theoretical value across the whole training process though learning rate changes.

heading: 6 CONCLUSION
text: In this paper, we comprehensively reveal the learning dynamics of DNN with normalization, WD, and SGD/SGDM, named as Spherical Motion Dynamics (SMD). Different from most related works (van Laarhoven, 2017; Hoffer et al., 2018; Chiley et al., 2019; Li & Arora, 2020), we directly explore the cause of equilibrium. Specifically, we introduce the assumptions that can lead to equilibrium condition, and show these assumptions are easily satisfied by practical implementation of DNN; Under given assumptions, we prove equilibrium condition can be reached at linear rate, far before optimization has converged. Most importantly, we show our theorem is widely valid, they can be verified on one of most challenging computer vision tasks, beyond synthetic datasets. We believe our theorems on SMD can bridge the gap between current theoretical progress and practical usage on deep learning techniques.

heading: A MISSING DERIVATIONS

heading: A.1 APPROXIMATION OF WEIGHT NORM UPDATE
text: Now considering the update rule of SGD with WD:
wt+1 = wt − η( ∂L ∂w ∣∣∣ w=wt + λwt). (23)
Since ∂L∂w ∣∣∣ w=wt is perpendicular to wt, we have
||wt+1||22 = ||wt − η( ∂L ∂w ∣∣∣ w=wt + λwt)||22 (24)
= (1− λη)2||wt||22 + η2|| ∂L ∂w ∣∣∣ w=wt ||22. (25)
Therefore ||wt+1||2 − ||wt||2 = √
(1− λη)2||wt||22 + η2|| ∂L ∂w ∣∣∣ w=wt ||22 − ||wt||2 (26)
= (1− λη)2||wt||22 + η2|| ∂L∂w ∣∣∣ w=wt
||22 − ||wt||22√ (1− λη)2||wt||22 + η2|| ∂L∂w ∣∣∣ w=wt ||22 + ||wt||2 . (27)
Now assume η, λ is extremely small, so that
(1− λη)2||wt||22 + η2|| ∂L ∂w ∣∣∣ w=wt ||22 − ||wt||22 ≈ −2λη||wt||22 + η2|| ∂L ∂w ∣∣∣ w=wt
||22 (28)√ (1− λη)2||wt||22 + η2||
∂L ∂w ∣∣∣ w=wt ||22 + ||wt||2 ≈ 2||wt||2. (29)
Therefore, we have
||wt+1||2−||wt||2 ≈ −λη||wt||2+ η2 2||wt||2 || ∂L ∂w ∣∣∣ w=wt ||22 = −λη||wt||2+ η2 2||wt||32 || ∂L ∂w ∣∣∣ w=w̃t
||22 (30)
where ∂L∂w ∣∣∣ w=w̃t is the unit gradient defined in Definition 1.

heading: B PROOF OF THEOREMS
text: Remark 4. In the following context, we will use the following conclusion multiple times: ∀δ, ε ∈ R, if |δ| 1, |ε| 1, then we have:
(1 + δ)2 ≈ 1 + 2δ, √
1 + δ ≈ 1 + δ 2 ,
1
1 + δ ≈ 1− δ, (1 + δ)(1 + ε) ≈ 1 + δ + ε. (31)

heading: B.1 PROOF OF LEMMA 1
text: Proof. Given w0 ∈ Rp\{0}, since ∀k > 0,L(w0) = L(kw0), then we have
∂L(w) ∂w ∣∣∣ w=w0 = ∂L(kw) ∂w ∣∣∣ w=w0 = ∂L(w) ∂w ∣∣∣ w=kw0 · k (32)
∂L(kw) ∂k ∣∣∣ w=w0 = 〈∂L(w) ∂w ∣∣∣ w=kw0 ,w0〉 = 1 k · 〈∂L(w) ∂w ∣∣∣ w=w0 ,w0〉 = 0 (33)

heading: B.2 PROOF OF THEOREM 1
text: Lemma 2. If the sequence {xt}∞t=1 satisfies
xt ≥ αxt−1 + L
xt−1 (34)
. where x1 > 0, L > 0
Then, we have
xt ≥ √ L
1− α − αt−1|
√ L
1− α − x1| (35)
Proof. If xt ≥ √ L 1−α , since √ L/(1− α) ≥ 2 √ αL then we have
xt+1 ≥ αxt + L
xt ≥ α
√ L
1− α + L√ L/(1− α) =
√ L
1− α , (36)
which means ∀k > t, xk ≥ √ L/(1− α). If xt < √ L/(1− α), then we have√
L
1− α − xt+1 ≤ (α− √ L(1− α) xt )( √ L 1− α − xt) < α( √ L 1− α − xt). (37)
Therefore, by deduction method we have if xT < √ L/(1− α), then ∀t ∈ [1, T − 1], we have
0 < xt < xt+1 < XT <
√ L
1− α , (38)
(
√ L
1− α − xt) < αt−1(
√ L
1− α − x1). (39)
In summary, we have
xt ≥ √ L
1− α − αt−1|
√ L
1− α − x1| (40)
Proof of Theorem 1. since 〈wt, gt〉 = 0, then we have:
||wt+1||22 = (1− ηλ)2||wt||22 + ||g̃t||22η2
||wt||22 (41)
Denote xt as ||wt||22, Lt as ||g̃t||22 and omit O((ηλ)2) part. Then Lt > l, E[Lt|xt] = L, V ar(Lt|xt) = E[(Lt − L)2|xt] < V . Eq.(41) is equivalent to
xt+1 = (1− 2λη)xt + Ltη
2
xt . (42)
According to Lemma 2, we have
xt >
√ lη
2λ − (1− 2λη)t−1|x0 −
√ lη
2λ |. (43)
Eq.(43) implies when t > 1 + log(( √ 2−1) √ lη/(4λ))−log(|x0− √ lη/(4λ)|) log(1−2λη) , we have xt > √ lη 4λ .
Now, denote x∗ as the √
Lη 2λ , then we have
E[(xt+1 − x∗)2|xt] = E[((1− 2λη − Lη2
xtx∗ )(xt − x∗) + Lt − L xt )2|xt]
= (1− 2λη − Lη 2
xtx∗ )2(xt − x∗)2 +
E[(Lt − L)2|xt]η4
x2t
(44)
If t > T (α, λ, η, l, x0) = 1 + log((
√ 2−1) √ lη/(4λ))−log(|x0− √ lη/(4λ)|)
log(1−2λη) , Eq.(43) implies 1− 2λη − √ 2L
l · 2λη < 1− 2λη − Lη
2
x∗xt . (45)
Combining with assumption 3 in Theorem 1, we have 1− 2λη − √
2L l · 2λη > 0, which means
0 < 1− 2λη − L x∗xt . (46)
Combining with Eq.(44), (43), (46), if t > T (α, λ, η, l, x0), we have
E[(xt+1 − x∗)2|xt] < (1− 2λη)2(xt − x∗)2 + 4V η3λ
l . (47)
Considering the expectation with respect to the distribution of xt, we have
E(xt+1 − x∗)2 < (1− 2λη)2E(xt − x∗)2 + 4V η3λ
l . (48)
Approximate (1− 2λη)2 = 1− 4λη, and iterate Eq.(48) for t− T (α, λ, η, l, x0) times, we have
E(xt − x∗)2 < (1− 4λη)t−T (α,λ,η,l,x0)E(xT (α,λ,η,l,x0) − x ∗)2 +
V η2
l . (49)
T (α, λ, η, l, x0) is finite, and only depends on α, λ, η, l, x0. Hence we can set B = max{(1 − 4λη)−tE(xt − x∗)2|t = 0, 1, 2, ..., T (α, λ, η, l, x0)}, note B is finite by
Ex2t+1 = E[(1−2λη)2x2t+2(1−2λη)Lt+ L2t x2t ] < (1−4λη)Ex2t+2(1−2λη)L+ V + L2 min(x20, lη/(2λ)) . (50) therefore, ∀t > 0, we have
E(xt − x∗)2 < (1− 4λη)tB + V η2
l . (51)

heading: B.3 PROOF OF THEOREM 2
text: Lemma 3. Assume α, β, ε ∈ (0, 1), where β << 1. Denote diag(1 − 2β1−α , α, α 2 + 2α
2
1−αβ as Λ,
k = ( 1(1−α)2 ,− 2α (1−α)2 , α2 (1−α)2 ) T , e = (1, 1, 1)T . If ε < 13 [ 1−α2 β − 8 1−α ], then ∀d ∈ R p, we have
||(Λ− εβ(1− α)2keT )d||22 < (1− 4β
1− α )||d||22 (52)
Proof. Omit O(β2) part, we have
||(Λ−εβ(1−α)2keT )d||22 = (1− 4β
1− α )d21+α 2d22+α 4(1+
4β
1− α )d23−2εβ(d1+d2+d3)(d1−2α2d2+α4d3)
(53) (d1 + d2 + d3)(d1 − 2α2d2 + α4d3)
= [d1 + (1− 2α2)d2]2
2 +
[d1 + (1 + α 4)d3] 2
2 − (1/2 + 2α4)d22 −
1 + α8
2 + (α4 − 2α2)d2d3
≥− (1 2 + 2α4 + α4 2 )d22 − ( α8 2 + α4 2 − 2α2 + 5 2 )d23 ≥− 3d22 − 5
2 d23.
(54)
Then we have
||(Λ− εβ(1− α)2keT )d||22 ≤ (1− 4β
1− α )d21 + (α 2 + 3βε)d22 + (α 4 +
4βα4 1− α + 5βε 2 )d23. (55)
Since ε < 13 [ 1−α2 β − 8 1−α ], we have
α2 + 3βε < 1− 4β 1− α , (56)
α4 + 4βα4
1− α +
5βε 2 < 1− 4β 1− α . (57)
Hence, we have
||(Λ− εβ(1− α)2keT )d||22 < (1− 4β
1− α )||d||22 (58)
Proof of Theorem 2. The update rule is
wt+1 = wt − ηvt
= wt − η(αvt−1 + g̃t ||wt|| + λwt) = wt − η(α wt−1 −wt
η + g̃t ||wt|| + λwt)
= (1− ηλ+ α)wt − αwt−1 − gtη.
(59)
Then we have
||wt+1||22 = (1− ηλ+ α)2||wt||22 − 2α(1 + α− ηλ)〈wt,wt−1〉+ α2||wt−1||22 + ||gt||22η2 + 2〈αwt−1, gtη〉 = (1− ηλ+ α)2||wt||22 − 2α(1 + α− ηλ)〈wt,wt−1〉+ α2||wt−1||22 + ||gt||22η2 + 2〈α(wt + ηvt−1), gtη〉 = (1− ηλ+ α)2||wt||22 − 2α(1 + α− ηλ)〈wt,wt−1〉+ α2||wt−1||22
+ h̃tη
2
||wt||22 .
(60)
〈wt+1,wt〉 = (1 + α− λη)||wt||2 − α〈wt,wt−1〉, (61) Let Xt,A, e denote:
Xt = ( at bt ct ) =  ||wt||22〈wt,wt−1〉 ||wt−1||22  , (62) A =
 (1 + α− λη)2 −2α(1 + α− λη) α21 + α− λη −α 0 1 0 0  , (63) e =
( 1 0 0 ) , (64)
(65)
respectively. The Eq.(60), (61) is formulated as a iterative map:
Xt+1 = AXt + h̃tη
2
eTXt e. (66)
When λη < (1− √ α)2, the eigen value of A are all real number:
λ1 = (1 + α− λη)2 + (1 + α− λη)
√ (1 + α− λη)2 − 4α
2 − α = 1− 2λη 1− α +O(λ2η2), (67)
λ2 = α, (68)
λ3 = (1 + α− λη)2 − (1 + α− λη)
√ (1 + α− λη)2 − 4α
2 − α = α2 + 2α
2
(1− α) λη +O(λ2η2),(69)
and they satisfy 0 < λ3 < λ2 = α < λ1 < 1. (70)
Therefore, A can be formulated as S−1AS = Λ, (71)
where Λ is a diagonal matrix whose diagonal elements are the eigen value of A; the column vector of S is the eigen vectors of A, note the fromulation of S,Λ are not unique. Specifically, we set Λ,S as
Λ = ( λ1 0 0 0 λ2 0 0 0 λ3 ) , (72)
S =  1 1 11+α−λη α+λ1 1+α−λη α+λ2 1+α−λη α+λ3
1 λ1
1 λ2
1 λ3  . (73) Moreover, the inverse of S exists, and can be explicitly expressed as
S−1 =  (α+λ1)λ1 (λ1−α)(λ1−λ3) − 2λ1(α+λ1)(α+λ3) (λ1−λ3)(λ1−α)(1+α−β) λ1λ3(α+λ1) (λ1−α)(λ1−λ3) − 2α 2
(λ1−α)(α−λ3) 2α(α+λ1)(α+λ3) (λ1−α)(α−λ3)(1+α−β) − 2αλ1λ3
(λ1−α)(α−λ3) (α+λ3)λ3
(α−λ3)(λ1−λ3) − 2λ3(α+λ3)(α+λ1) (λ1−λ3)(α−λ3)(1+α−β) (α+λ3)λ1λ3 (α−λ3)(λ1−λ3)  . (74) let Yt = S−1Xt, combining with Eq.(66), we have
Yt+1 = ΛYt + h̃tη
2
(STe)TYt S−1e. (75)
Combining with Eq.(73) and Eq.(74), and set Yt = (ãt, b̃t, c̃t)T , we rewrite Eq.(75) as
ãt+1 = λ1ãt + h̃tη
2
ãt + b̃t + c̃t · (α+ λ1)λ1 (λ1 − α)(λ1 − λ3) , (76)
b̃t+1 = αb̃t − h̃tη
2
ãt + b̃t + c̃t · 2α
2
(λ1 − α)(α− λ3) , (77)
c̃t+1 = λ3c̃t + h̃tη
2
ãt + b̃t + c̃t · (α+ λ3)λ3 (α− λ3)(λ1 − λ3) . (78)
Note ||wt||22 = ãt + b̃t + c̃t. Now we prove the following inequations by mathematical induction
b̃t < 0, (79) c̃t > 0, (80)
(α− λ1)b̃t > (λ1 − λ3)c̃t, (81) ãt + b̃t + c̃t > 0, (82)
ãt+1 + b̃t+1 + c̃t+1, > λ1(ãt + b̃t + c̃t) + h̃tη
2
ãt + b̃t + c̃t , (83)
Since the start point X1 = (a1, a1, a1)T (a1 > 0), the start point Y1 = S−1X1. Combining with Eq.(74), we have
b̃1 = − 2α2λη
(λ1 − α)(α− λ3) a1, (84)
c̃1 = λ3(λ3 + α)(1− α+ λη) (λ3 − α)(λ1 − λ3)(1 + α− λη) ( 1− α− λη 1− α+ λη − λ1)a1, (85)
by which we have (α− λ1)b̃1 > (λ1 − λ3)c̃1. Besides ã1 + b̃1 + c̃1 = eTSY1 = eTX1 = a1 > 0. Suppose for t = T , Eq. (79), (80), (81), (82) hold, combining with Eq.(77), (78), we can derive b̃T+1 < 0, ãT+1 > 0, so Eq.(79), (80) hold for t = T + 1; Besides, we have
(α− λ1)b̃T+1 = α(α− λ1)b̃T + h̃tη
2
ãT + b̃T + c̃T · 2α
2
(α− λ3)
> λ3(λ1 − λ3)c̃T + h̃tη
2
ãT + b̃T + c̃T · (α+ λ3)λ3
(α− λ3) = (λ1 − λ3)c̃T+1,
(86)
thus Eq.(81) holds for t = T + 1. Sum Eq.(76), Eq.(77), Eq.(78), due to Eq.(81) we have
ãT+1 + b̃T+1 + c̃T+1 = λ1ãT + αb̃T + λ3c̃T + h̃tη
2
ãT + b̃T + c̃T
> λ1(ãT + b̃T + c̃T ) + h̃tη
2
ãT + b̃T + c̃T ,
(87)
Eq.(83) holds for t = T + 1, combining with the fact that ãT + b̃T + c̃T > 0, we have ãT+1 + b̃T+1 + c̃T+1 > 0.
According to Lemma 2, we can estimate the lower bound of ãt + b̃t + c̃t: when t > 1 + log(( √ 2−1) √ lη/(4λ))−log(|||w0||22− √ lη/(4λ)|)
log(1−2λη) ,
ãt + b̃t + c̃t ≥
√ lη 2(1− λ1) ≈ √ lη(1− α) 4λ (88)
Now we can analyze the expectation of distance(l2 norm) between Yt = (ãt, b̃t, c̃t)T and the fixed point Y ∗ = (ã∗, b̃∗, c̃∗)T which satisfies
Y ∗ = ΛY ∗ + Lη2
(STe)TY ∗ S−1e. (89)
Assume xt = ãt + b̃t + c̃t, x∗ = ã∗ + b̃∗ + c̃∗ > 0, then we have:
Yt+1 − Y ∗ = (Λ− Lη2
xtx∗ keT )(Yt − Y ∗) +
(h̃t − L)η2
xt k (90)
where k = (k1, k2, k3)T = S−1e. In the following context, we will omit the O(λ2η2) part since λη 1. k1, k2, k3 can be approximated as
k1 = 1
(1− α)2 +O(β), (91)
k2 = − 2α
(1− α)2 +O(β), (92)
k3 = α2
(1− α)2 +O(β). (93)
Then we have
E[||Yt+1 − Y ∗||22|Yt] = ||(Λ− L
xtx∗ keT )(Yt − Y ∗)||22 +
E[(h̃t − L)2|Yt]η4
x2t ||k||22 (94)
The fixed point of Eq.(89) is computed as x∗ = ã∗ + b̃∗ + c̃∗ = √
Lη
λ(1−α)(2− λη1+α ) , and we have
known that if t > 1 + log(( √ 2−1) √ lη/(4λ))−log(|||w0||22− √ lη/(4λ)|) log(1−2λη) , xt > √ lη(1−α) 4λ ,, therefore we have
Lη2 xtx∗ <
√ 2L
l · 2λη, (95)
E[(h̃t − L)2|Yt] x2t η4 < 4V η3λ l(1− α) . (96)
According to assumption 3, we can prove√ 2L
l <
(1− α)2 3 [ 1− α2 β − 8 1− α ], (97)
combining with Lemma 3, we have
E[||Yt+1 − Y ∗||22|Yt] < (1− 4λη
1− α )||Yt − Y ∗||22 + 4V η3λ||k||22 l(1− α) , (98)
which implies
E||Yt+1 − Y ∗||22 < (1− 4λη
1− α )t−TE||YT − Y ∗||22 +
V η2(1 + 4α2 + α4)
l(1− α)4 , (99)
where T = [1 + log(( √ 2−1) √ lη/(4λ))−log(|||w0||22− √ lη/(4λ)|)
log(1−2λη) ]. Therefore, similar to the proof of theorem 1, ∃B > 0,
E||Yt+1 − Y ∗||22 < (1− 4λη
1− α )tB +
V η2(1 + 4α2 + α4)
l(1− α)4 . (100)
Recall ||wt||22 = eTYt, therefore
E[||wt||22 − (w∗)2]2 ≤ 3E||Yt+1 − Y ∗||22 < 3(1− 4λη
1− α )tB +
3V η2(1 + 4α2 + α4)
l(1− α)4 . (101)
Remark 5. By Eq.(101), the variance of ||wt||22 is bounded by 3V η2(1+4α2+α4)
l(1−α)4 , which is not small enough. But we somehow can reduce it: according to Eq.(96), if xt is close to its theoretical value, then E[(h̃t−L)
2|Yt] x2t η4 < 4V η 3λ(1−α) l , hence variance of ||wt|| 2 2 can be bounded by 3V η2(1+4α2+α4) 2L(1−α)2 .

heading: B.4 PROOF OF THEOREM 3
text: Proof. In SGD case, we have
〈wt+1,wt〉 = (1− λη)||wt||22, (102)
then we have
cos2 ∆t = 〈wt+1,wt〉2 ||wt||22 · ||wt+1||22 = (1− 2λη) ||wt|| 2 2 ||wt+1||22 . (103)
According to the definition of ∆t, ∆t ≥ 0, and ∆t is very close to 0, hence we have ∆t = sin ∆t = √ 1− cos2 ∆t (104)
= √ 1− (1− λη)2 ||wt|| 2 2
||wt+1||22 (105)
= √ 1− (1− λη)2 xt
xt+1 (106)
where xt, xt+1 denotes ||wt||22, ||wt+1||22 respectively as in Eq. (42). Assume t is sufficiently large so that xt, xt+1 are close to x∗ = √ Lη 2λ , the first order of Taylor series expansion of Eq.(106) at xt = xt+1 = x ∗ is
∆t = √ 2λη + (1− λη)2
2 √ 2λη · 1 x∗ · [(xt+1 − x∗)− (xt − x∗)]. (107)
Reorganizing Eq.(107), and applying Cauchy Inequality, we have
|∆t − √ 2λη|2 = (1− λη) 4
8λη · 1 (x∗)2 · [(xt+1 − x∗)− (xt − x∗)]2 (108)
≤ 1 8λη · 2λ Lη · 2[(xt − x∗)2 + (xt+1 − x∗)2] (109)
= 1
2Lη2 [(xt − x∗)2 + (xt+1 − x∗)2] (110)
Combining with Eq.(51) and Eq.(108), we have
E|∆t − √
2λη|2 = 1 2Lη2 [E(xt − x∗)2 + E(xt+1 − x∗)2] ≤ (1− 4λη)t · C + V Ll , (111)
where C = 12Lη2 ·B, B is defined in Eq.(51).
In SGDM case, the angular update can be computed by
∆t = sin(∆t) (112) = √ 1− cos2 ∆t (113)
= √ 1− 〈wt,wt+1〉 2
||wt||22 · ||wt+1||22 (114)
= √ 1− b 2 t
atct , (115)
where (at, bt, ct) = (||wt||22, 〈wt,wt+1〉, ||wt+1||22). According to the proof of theorem 2, when t is sufficiently large, (at, bt, ct) will be close to the fixed point of Eq.(66), (a∗, b∗, c∗), where a∗ =
c∗ = (
√ Lη
λ(1−α)(2− λη1+α ) , b∗ = 1+α−λ1+α a ∗. Then the first order Taylor series expansion of Eq.(115)
at (at = a∗, bt = b∗, ct = c∗) is
∆t =
√ 1− b 2 t
atct (116)
=
√ 2λη
1 + α +
√ 1 + α 2 √ 2λη · (1− λη 1 + α )2[ at − a∗ a∗ − 2(bt − b ∗) b∗ + ct − c∗ c∗ ]. (117)
Now substituting (at, bt, ct) with (ãt, b̃t, c̃t) defined in Eq.(75, 76,77, 78), Eq.(107) is rewritten as
∆t =
√ 2λη
1 + α +
√ 1 + α 2 √ 2λη · (1− λη 1 + α )2 1 a∗ [ (1− α)2 α2 (c̃t − c̃∗) +O(λη)] (118)
where (ã∗, b̃∗, c̃∗) is the fixed point of Eq.(89). Omit O(λη), we have
|∆t − √ 2λη
1 + α |2 = 1 + α 8λη · (1− λη 1 + α )4 λ(1− α)(2− λη1+α ) Lη [ (1− α)4 α4 (c̃− c̃∗)2] (119)
≤ (1− α 2)(1− α)4
4Lη2α4 ||Yt − Y ∗||22, (120)
where Yt, Y ∗ is defined in Eq.(75, 89) respectively. According to Eq.(100), the mean square error E|∆t − √ 2λη 1+α | 2 can be bounded by
E|∆t − √ 2λη
1 + α |2 ≤ (1− α 2)(1− α)4 4Lη2α4 E||Yt − Y ∗||22 (121)
≤ (1− 4λη 1− α )tC + (1− α2)(1− α)4 4Lη2α4 · V η 2(1 + 4α2 + α4) l(1− α)4 (122)
= (1− 4λη 1− α )tC + V (1− α2)(1 + 4α2 + α4) 4Llα4 , (123)
where C = (1−α 2)(1−α)4
4Lη2α4 ·B, B is defined in Eq.(101).

heading: B.5 PROOF OF COROLLARY 3.1
text: Proof of Corollary 3.1. In SGD case, and Eq.(42), we have
||wt+1||22 > (1− 2λη)||wt||22, (124)
which means ||wt+T ||22 > (1− 2λη)T ||wt||22. (125) On the other hand, we know that when η is divided by k, ||wt||22 should be divided by √ k to reach the new equilibrium condition, therefore we have
||wt+T ||2
||wt||2 = 1√ k > (1− 2λη)T . (126)
Since λη 1, log(1− 2λη) ≈ −2λη, thus
T > log(k)
4λη . (127)
.
In SGDM case, by Eq.(87), we have
||wt+1||22 > (1− 2λη
1− α )||wt||22, (128)
Similar to SGD case, we have
T > log(k)(1− α)
4λη . (129)
.

heading: C EXPERIMENTS

heading: C.1 EXPERIMENTS ON SYTHETIC DATA
text: In this section we apply experiments on sythetic data to prove our claim in section 4: theorem 1 only requires expected square norm of unit gradient is locally steady, the expectations do not need to remain unchanged across the whole training process. The proof of theorem implies square norm of weight is determined by the following iterative map:
xt+1 = (1− 2λη)xt + Ltη
2
xt , (130)
where λ, η ∈ (0, 1), Lt denotes the square of unit gradient norm. Hence we simulate xt with different type of {Lt}∞t=1. Results in Figure 4 shows as long as the local variance of square norm of unit gradient is not too much, and expectation of Lt changes smoothly, weight norm can quickly converge to its theoretical value base on expectation of square norm of unit gradient.
We also simulate SGDM case by following iteration map
Xt+1 = AXt + Ltη
2
Xt[0] · e, (131)
where A, Xt, e is defined as Eq.(62), (63), (64). Simulation results is shown in Figure 5.

heading: C.2 COMPLEMENTARY IN MULTI-STAGE LEARNING RATE SCHEDULE
text: In this section we present complementary results in Multi-Stage Learning Rate Schedule experiment.
The plots of weight norm (empirical and predicted values) in multi-learning rate stage is shown in Figure 6. We also present the test performance of resent50/maskrcnn on Imagenet/MSCOCO with multi-stage learning rate schedule mentioned in Section 5.2. We only provide complementary results for reference only. We do not intend to prove the advantages or disadvantages of rescaling strategy here, it is beyond the discussion of this paper.

heading: C.3 RETHINKING LINEAR SCALING PRINCIPLE IN SPHERICAL MOTION DYNAMICS
text: In this section, we will discuss the effect of Linear Scaling Principle (LSP) under the view of SMD. Linear Scaling Principle is proposed by Goyal et al. (2017) to tune the learning rate η with batch size B by η ∝ B. The intuition of LSP is if weights do not change too much within k iterations, then k iterations of SGD with learning rate η and minibatch size B (Eq.(132)) can be approximated by a single iteration of SGD with learning rate kη and minibatch size kB (Eq.(133):
wt+k = wt − η ∑ j<k ( 1 B ∑ x∈Bj ∂L ∂w ∣∣∣ wt+j ,x + λwt+j), (132)
wt+1 = wt − kη( 1
kB ∑ j<k ∑ x∈Bj ∂L ∂w ∣∣∣ wt,x + λwt). (133)
Goyal et al. (2017) shows that combining with gradual warmup, LSP can enlarge the batch size up to 8192(256× 32) without severe degradation on ImageNet experiments. LSP has been proven extremely effective in a wide range of applications. However, from the perspective of SMD, the angular update mostly relies on the pre-defined hyper-parameters, and it is hardly affected by batch size. To clarify the connection between LSP and SMD, we explore the learning dynamics of DNN with different batch size by conducting extensive experiments with ResNet50 on ImageNet, the training settings rigorously follow Goyal et al. (2017): momentum coefficient is α = 10−4; WD coefficient is λ = 10−4; Batch size is denoted by B; learning rate is initialized as B 256 · 0.1; Total training epoch is 90 epoch, and learning rate is divided by 10 at 30, 60, 80 epoch respectively.
The results of experiments(Figure 7, 8) suggests that the assumption of LSP does not always hold in practice because of three reasons: first, the approximate equivalence between a single iteration in large batch setting, and multiple iterations in small batch setting can only hold in pure SGD formulation, but momentum method is far more commonly used; Second, according Theorem 2, the enlargement ratio of angular update is only determined by the increase factor of learning rate. Figure 7 shows in practice, the accumulated angular update ](wt,wt+k) in small batch batch setting is much larger than angular update ](wt,wt+1) of a single iteration in larger batch setting when using Linear Scaling Principle; Third, even in pure SGD cases, the enlargement of angular update still relies on the increase of learning rate, and has no obvious connection to the enlargement of gradient’s norm when equilibrium condition is reached (see Figure 8).
In conclusion, though LSP usually works well in practical applications, SMD suggests we can find more sophisticated and reasonable schemes to tune the learning rate when batch size increases.

heading: C.4 SPHERICAL MOTION DYNAMICS WITH DIFFERENT NETWORK STRUCTURES
text: We also verify our theory on other commonly used network structures (MobileNet-V2 (Sandler et al., 2018), ShuffleNet-V2+ (Ma et al., 2018)) with standard training settings. The results is shown in Figure 9.
1+α . Learning rate η is initialized as 0.5, and divided by 10 at epoch 30, 60, 80 respectively; WD
coefficient λ is 4× 10−5; Momentum parameter α is set as 0.9.

Review summaries:
The main goal of the paper is to establish theoretically some previous known results that for scale invariant networks the weight norm has a fixed point with ||w||^4=eta/lambda ||\tilde{g}|| . They also discuss the angular update, which because of scale invariance is basically equivalent to arccos (1-eta lambda) |w_t|^2/|w_t+1|^2 and it thus comes mainly from the gradient. They have some experiments which they compare with the predicted equilibrium values for the angular update/ weight norm.