Spherical Motion Dynamics: Learning Dynamics of Normalized Neural Network using SGD and Weight Decay
Year: 2022

Abstract:
In this paper, we comprehensively reveal the learning dynamics of normalized neural network using Stochastic Gradient Descent (with momentum) and Weight Decay (WD), named as Spherical Motion Dynamics (SMD). Most related works on this topic focus on studying “effective learning rate" using “equilibrium" assumption, i.e. assuming weight norm has converge to a fixed value. However, their discussion on why equilibrium can be reached is either absent or unjustified. To clarify the mechanism behind, our work directly explores the cause of equilibrium, which should be regarded as a special state of SMD. Specifically, 1) we introduce the assumptions that can lead to equilibrium state in SMD, and prove equilibrium can be reached in a linear rate regime; 2) we propose “angular update" as a substitute for effective learning rate to depict the state of SMD, and derive the theoretical value of angular update in equilibrium state; 3) we verify our assumptions and theoretical results on various large-scale computer vision tasks including ImageNet and MSCOCO with standard settings. Experiment results show our theoretical findings agree well with empirical observations. Furthermore, we provide intuitive interpretations, showing how the behavior of angular update in SMD affects the optimization of neural network, and yields unexpected phenomenon in practice. We believe our findings and theoretical results can deepen our understanding on current training techniques for deep neural network.

heading: 1 Introduction
text: Normalization techniques (e.g. Batch Normalization (Ioffe & Szegedy, 2015) or its variants) are one of the most commonly adopted techniques for training deep neural networks (DNN). A typical normalization can be formulated as following: consider a single unit in a neural network, the input is X , the weight of linear layer is w (bias is included in w), then its output is
y(X;w; γ;β) = g( Xw − µ(Xw)
σ(wX) γ + β), (1)
where g is a nonlinear activation function like ReLU or sigmoid, µ, σ are mean and standard deviation computed across specific dimension of Xw (like Batch Normalization (Ioffe & Szegedy, 2015), Layer Normalization Ba et al. (2016), Group Normalization (Wu & He, 2018), etc.). β, γ are learnable
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
parameters to remedy for the limited range of normalized feature map. Aside from normalizing feature map, Salimans & Kingma (2016) normalizes weight by l2 norm instead:
y(X;w; γ;β) = g(X w
||w||2 γ + β), (2)
where || · ||2 denotes l2 norm of a vector. Though formulated in different manners, all normalization techniques mentioned above share an interesting property: scale-invariance Definition 1 (Scale-invariance). Given loss function L(w), w is scale-invariant w.r.t. L if and only if ∀k ∈ R+, we have L(w) = L(kw).
By definition of scale-invariance, we can directly derive the following properties of scale-invariant weights in Lemma 1 Lemma 1. If w is scale-invariant with respect to L(w) , then for all k > 0, we have:
〈wt, ∂L ∂w ∣∣∣ w=wt 〉 = 0 (3)
∂L ∂w ∣∣∣ w=kwt = 1 k · ∂L ∂w ∣∣∣ w=wt . (4)
Proof is in appendix. Lemma 1 is also discussed in Hoffer et al. (2018); van Laarhoven (2017); Li & Arora (2020); Li et al. (2020), it makes the learning dynamics of normalized neural network exhibit an interesting phenomenon when using Stochastic Gradient Descent (SGD) with Weight Decay (WD): a typical SGD update rule with WD is
wt+1 = wt − η( ∂L ∂w ∣∣∣ w=wt + λwt) = (1− ηλ)wt − η ∂L ∂w ∣∣∣ w=wt , (5)
where η denotes learning rate, λ denotes WD factor. Then dynamics of wt is like a physical process – a satellite’s motion around the earth (see illustration in Fig.1): according to Eq.(3),−η∂L/∂w ∣∣ w=wt
(green line in Fig.1) is always perpendicular to wt, providing “centrifugal effect” to make ||wt+1||2 larger than ||wt||2; while −ηλwt (red line in Fig.1) is always in the opposite direction of wt, providing “centripetal effect” to make ||wt+1||2 smaller than ||wt||2. Due to this “tug of war” between “centrifugal effect” and “centripetal effect”, the norm of weight will change accordingly. Though the norm of scale-invariant weight doesn’t influence the loss at all (by definition), but it will affect the optimizing trajectory by changing the scale of gradient (Eq.4). Therefore, to decouple the relation between weight norm and gradient norm, a common way (van Laarhoven, 2017; Hoffer et al., 2017; Chiley et al., 2019) is to regard the
learning dynamics of scale-invariant weight as a manifold learning process restricted on a unit sphere, using “effective learning rate", defined as η/||wt||22. Akin to this reason, in this paper the learning dynamics of scale-invariant weight is viewed as a motion on a unit sphere, whose motion speed is determined by weight norm (in original weight space), gradient norm, and hyper-parameters. Therefore, we formally name it as Spherical Motion Dynamcis (SMD) and focus on discussing its unique properties.
Concept of “Equilibrium” Since “tug-of-war” in SMD determines the relative sizes of ||wt+1||2 and ||wt||2, a question naturally arises: what will happen if ||wt+1||2 = ||wt||2? van Laarhoven (2017) discuss this question first; Chiley et al. (2019) named this state where ||wt+1||2 = ||wt||2 in SMD as “equilibrium”, and discuss its properties; Li & Arora (2020) derives a lemma about equilibrium in SGD with Momentum (SGDM); Li et al. (2020); Kunin et al. (2021) establish the
equilibrium in continuous approximation model. However, most early literature (van Laarhoven, 2017; Chiley et al., 2019; Li & Arora, 2020; Kunin et al., 2021) do not discuss a fundamental question: “Does equilibrium really occur in practice?” van Laarhoven (2017) intuitively explains that equilibrium is caused by convergence of optimization. But there exists a contradiction between the interpretation of van Laarhoven (2017) and traditional view of optimization: if equilibrium (||wt||2 = ||wt+1||2) is caused by convergence of optimization, then gradient of loss ∂L/∂w ∣∣ w=wt should be 0, which makes the balance of “centrifugal effect” and “centripetal effect” impossible to reach (since centripetal effect comes from ∂L/∂w ∣∣ w=wt
). Therefore, van Laarhoven (2017); Chiley et al. (2019); Li & Arora (2020); Kunin et al. (2021) all essentially regard equilibrium as an assumption, and do not justify its existence in neither empirical nor theoretical aspects. “Equilibrium” was not even a phenomenon observed in practice, but only a concept until recently.
Recent work (Li et al., 2020) successfully exhibits the existence of equilibrium by formulating SGD in Eq.(5) via a Stochastic Differential Equation (SDE) in the continuous time limit. They theoretically prove equilibrium can be reached in SDE settings via convergence of ||w||t, and the convergence time is O(1/(λη)) (λ, η denote WD factor and learning rate respectively). However, due to the gap between discrete formulation of SGD and continuous formulation of SDE, theoretical results derived from SDE model can only provide intuitive understanding on empirical observations, some of which are even incorrect (will be discussed latter); Besides, SDE can hardly take SGD with momentum (Polyak, 1964) into account, which has become default setting in nearly all kinds of deep learning tasks. In summary, a thorough understanding on cause of “equilibrium” and its impact to learning dynamics of normalized neural network is still needed.
In this paper, we comprehensively reveal Spherical Motion Dynamics (SMD), i.e. the learning dynamics of normalized neural network using SGD(M) and weight decay (WD). Our analysis on SMD is directly established on discrete settings. We interpret why equilibrium can be reached in SMD in both theoretical and empirical aspects, and show how SMD affects the optimization trajectory of neural network. Specifically, our contributions are
• We introduce the assumptions which can lead to equilibrium in SMD, and justify their reasonableness by sufficient experiments. We also prove under given assumptions, equilibrium can be reached as weight norm approach to its theoretical value in a linear rate regime. Our theorem show equilibrium is a dynamic state in SMD, convergence of weight norm is neither the precondition nor the consequence of equilibrium. This conclusion almost refutes the assumptions of all previous related work (van Laarhoven, 2017; Chiley et al., 2019; Li & Arora, 2020; Kunin et al., 2021), and the conclusion of Li et al. (2020).
• We define a novel index, angular update, to measure the change of normalized neural network within a single iteration. We also derive its theoretical value in equilibrium. Our results show that angular update is better than norm of weight to indicate if equilibrium has been reached reached in SMD. Our empirical results further show angular update is an important index to reflect the effect of SMD and equilibrium;
• We verify our theorems on different computer vision tasks (including one of most challenging datasets ImageNet (Russakovsky et al., 2015) and MSCOCO (Lin et al., 2014)) with various networks structures. Experiments show the theoretical value of angular update and weight norm agree well with empirical observation. We also show how SMD influence the optimization trajectory of normalized neural network by controlling angular update.
We believe SMD is one of the key reason why learning dynamics of normalized neural network is not consistent with traditional optimization theory (Li et al., 2020). We think it is of great potential to take SMD and its equilibrium state into account while studying leaning dynamics of modern normalized neural network or designing novel efficient training strategy.

heading: 2 Theoretical results
text: In this section, we theoretically formulate Spherical Motion Dynamics (SMD) in discrete SGD/SGDM settings, and provide a precise description on “equilibrium” phenomenon; Then we prove equilibrium can be reached in SMD under specific assumptions; Finally a new index, whose theoretical value in equilibrium can be derived, is proposed to indicate the state of SMD.
A new definition need to be given at first. Eq.(4) implies though norm of scale-invariant weights does not affect the output of neural network, it can influence norm of gradients, thus we define unit gradient in order to decouple weight norm and gradient norm. Definition 2 (Unit Gradient). If wt 6= 0, w̃ = w/||w||2, the unit gradient of ∂L/∂w|w=wt is ∂L/∂w|w=w̃t .
According to the definition of unit gradient, the unit gradient norm is independent of weight norm. Specifically, by setting k as 1/||wt||2 in Eq.(4), the relation among weight norm, gradient and unit gradient is
∂L ∂w ∣∣∣ w=wt = 1 ||wt|| · ∂L ∂w ∣∣∣ w=w̃t . (6)
Now, we can depict equilibrium of SGD and SGDM in theorem 1, 2 respectively. Theorem 1. (Equilibrium in SGD) Assume the loss function is L(X;w) with scale-invariant weight w, denote gt = ∂L∂w ∣∣ Xt,wt , g̃t = gt · ||wt||2. Consider the update rule of SGD with weight decay,
wt+1 = wt − η · (gt + λwt) (7) where λ, η ∈ (0, 1). If the following assumptions hold:
1) λη 1 (o(λη) can be omitted);
2) Let Lt = E[||g̃t||22|wt]. ∃V ∈ R+, ∀t ∈ N+, E[(||g̃t||22 − Lt)2|wt] ≤ V ; 3) ∀t ∈ N+, Lt satisfies |Lt+1 − Lt| < 4 √ V (λη)3/2;
4) ∃l ∈ R+, ∀t ∈ N+, ||g̃t||22 > l, l > 2[ 2λη 1−2λη ] 2Lt.
Then ∃B > 0, ∀t ∈ N+, w∗t = 4 √ Lt−1η/(2λ), we have
E[||wt||22 − (w∗t )2]2 ≤ (1− 2λη)tB + 2V η2
l . (8)
Remark 1. The theoretical value of weight norm w∗t in Theorem 1 is consistent with the magnitude of weight norm (O( 4 √ η/λ)) in equilibrium in van Laarhoven (2017), though van Laarhoven (2017) assumes the equilibrium has been reached in advance, hence van Laarhoven (2017) cannot provide the approaching rate and scale of bias/variance. The vanishing term ((1 − 2λη)tB) in Eq.(8) is consistent with the mixing time O(1/(λη)) presented in Li et al. (2020).
The proof can be seen in appendix. Assumption 1 is consistent with commonly used settings in practice (Goyal et al., 2017; He et al., 2017; Ma et al., 2018); Assumptions 2, 3, 4 all concern unit gradient: unit gradient norm should change smoothly (assumption 3) with bounded variance (assumption 2); besides, unit gradient norm should have a lower bound (assumption 4). We will see these assumptions can easily hold in practice in section 4.1. Remark 2. Note in theorem 1, we assume ||g̃t||22 has a uniform lowere bound l, while E[(||g̃t||22 − Lt)
2|wt] has a uniform upper bound V . But these are too strong assumptions and may not hold in practice. We only use them for ease of demonstration and proof. In fact the lower bound of ||g̃t||22 and upper bound of E[(||g̃t||22 − Lt)2|wt] can be replaced with smoothly varying functions lt and Vt respectively to remove unnecessary assumptions on uniformness. The “trend” to reach equilibirum is actually a local property, which can be inferred from the proof.
True meaning of “equilibrium”: a dynamic state of SMD Recall as we demonstrate in introduction, the concept of equilibrium is originally established on the assumption that weight norm is steady (||wt||2 = ||wt+1||2). But the assumption (||wt||2 = ||wt+1||2) is unrealistic due to the complex dynamics of training process and the variance of stochastic gradients. Now theorem 1 provides a realistic meaning of equilibrium in SGD settings: equilibrium is just a dynamic state of SMD, meaning ||wt||22 oscillates around the theoretical value (w∗t )2 determined by hyperparameters and unit gradient norm. Its variance is bounded by 2V η2/l, which is relatively small comparing with (w∗t )
4 because 2V η2
l /(w∗t )
4 = 4V λη
Lt−1l = O(λη) 1. (9)
Aside form the stochastic behavior of ||wt||22, the “dynamic state” also reflects in the variation of the theoretical value (w∗t ) 2. Because (w∗t ) 2 is determined by Lt, which is allowed to change smoothly across the whole training process in assumption 2 (See more discussion in appendix). In summary, the sign of equilibrium is neither the convergence of weight norm ||wt||22 (van Laarhoven, 2017; Chiley et al., 2019) nor the convergence of ||wt||2 in expectation (Li et al., 2020). The real sign of equilibrium is whether E||wt||22 is close to its theoretical value (w∗t )2. Theorem 1 also shows the dynamic equilibrium can be reached in a linear rate regime when vanishing term is larger than constant term in Eq.(8). The approaching rate is only determined by predefined parameters λ, η. Moreover, based on the proof of theorem 1, the cause of equilibrium is independent of optimization process at all, which implies the possibility that equilibrium can be reached long before the convergence of loss function, which refutes the conjections that “equilibirum” a equilibirum distribution of function space.
Now we extend theorem 1 to momentum case. SGDM is more complex than SGD since momentum is not always perpendicular to the weight, hence we need to modify assumptions.
Theorem 2. (Equilibrium in SGDM) Considering the update rule of SGDM (heavy ball method (Polyak, 1964)):
vt = αvt−1 + gt + λwt (10) wt+1 = wt − ηvt (11)
where λ, η ∈ (0, 1), α ∈ ( 12 , 1). If following assumptions hold:
5) λη 1, λη < (1− √ α)2;
6) Define ht = ||gt||22 + 2α〈vt−1, gt〉, h̃t = ht · ||wt||22, Lt = E[h̃t|wt]. ∃V ∈ R+, ∀t ∈ N+, E[(h̃t − Lt)2|wt] ≤ V ; 7) ∀t ∈ N+, Lt satisfies |Lt+1 − Lt| < 4 √ V (λη)3/2;
8) ∃l ∈ R+,∀t ∈ N+, h̃t > l > 2[ 6λη(1−α)3(1+α)−8λη(1−α) ] 2Lt;
then ∃B,C > 0, C only depends on α, w∗t = 4 √ Lt−1η/(λ(1− α)(2− λη/(1 + α))), we have
E[||wt||22 − (w∗t )2]2 ≤ (1− 2λη
1− α )tB +
V η2
l C, (12)
Remark 3. So far, no other work rigorously prove equilibrium can be reached in SGDM. The most relevant work (Li et al., 2020) only provides a conjecture on convergence rate of weight norm in SGDM. By regarding SGDM as SGD with larger learning rate, they guess that the mixing time to reach equilibrium in SGDM case should be O(1/(λη)), same order as mixing time in SGD case. Their conjecture cannot provide further insight on difference between SGD and SGDM. While our results (vanishing terms in Eq.(8), (12) respectively) clearly reflect the difference: the approaching rate of SGDM should be 1/(1− α) times larger than rate of SGD with same ηλ. α is usually set as 0.9 in practice, hence SGDM can reach equilibrium condition much faster than SGD.
Proof can be seen in appendix. Like assumption 1, assumption 5 also holds for commonly used hyperparameter settings; Assumption 6, 7, 8 concerns not unit gradient norm ||g̃t||22 but an adjusted value h̃t which dominates the expectation and variance of ||wt||22. We empirically find the expectation of 〈vt−1, gt〉 is very close to 0, therefore the behavior of h̃t is similar to that of ||g̃t||22 (see Figure 3(d)). We leave theoretical analysis on h̃t as future work. The experiments on justification of assumptions 6, 7, 8 can be seen in Figure 3. Comparing with Eq.(8) and Eq.(12), we can infer with same η, λ, SGDM can reach equilibrium state much faster than SGD, but it may have a larger variance, our experiments also verify our claim (see Figure 3(b), 3(e)).
We have derived the theoretical value of weight norm in equilibrium, it allows us to check if equilibrium has been reached in practice. But the theoretical value of weight norm still relies on the expectation of unit gradient norm, which is not easy to compute in practice. Besides, weight norm is of little value for studying normalized models since their weight is scale-invariant. Hence we introduce an new and meaningful index, angular update, to reflect the effect of SMD and equilibrium.
Definition 3 (Angular Update). Let wt denote a scale-invariant weight from a neural network at iteration t, then angular update ∆t is defined as
∆t = ](wt,wt+1) = arccos ( 〈wt,wt+1〉 ||wt|| · ||wt+1|| ) , (13)
where ](·, ·) denotes the angle between two vectors, 〈·, ·〉 denotes the inner product.
Angular update has a concrete geometric meaning (see illustration in Figure 1): it is exactly the geodesic distance between w̃t and w̃t+1 on Sp−1, where w̃t = wt/||wt||2, w̃t+1 = wt+1/||wt+1||2. Comparing with the Euclidean distance ||wt+1−wt||2, angular update ∆t better reflects the effective update of the scale-invariant weight wt on its intrinsic domain Sp−1. Angular update is determined by the relative sizes of gradient norm and weight norm, while the relative sizes of gradient/weight norm are influenced by SMD, hence angular update is strongly affected by SMD. The following theorem exhibits the behavior of angular update when equilibrium is reached in SMD.
Theorem 3. (Theoretical value of Angular Update) In SGD(SGDM) case, if assumptions in theorem 1(2) hold, η2 1, t is sufficiently large so that vanishing terms in Eq.(8), (12) can be omitted, then with probability at least 1− 3 √ V Ltl we have
|∆t − √ 2λη
1 + α | < O( 3
√ V
Ltl ). (14)
In SGD case, α = 0.
Remark 4. Results of SGD and SGDM case are summarized in Eq.(14) in order to highlight the connection between SGD and SGDM. Theoretical value of angular update in Theorem 3 is partially consistent with previous works (Chiley et al., 2019; Li & Arora, 2020; Li et al., 2020; Kunin et al., 2021), detailed discussion is buried in appendix. Note bias term in right side of Eq.(14) is of O( 3 √ V/Ltl), which is too large comparing with its empirical value (see Figure 3(c), 3(f)), we leave it as a future work to improve the bound in Eq.(14).
Proof is in appendix. According to theorem 3, the theoretical value of angular update in equilibrium only depends on hyper-parameters: learning rate η, WD factor λ, and momentum factor α. Hence comparing with behavior of weight norm, angular update provides an easier way to check whether equilibrium is reached. Since equilibrium can be reached in a linear rate regime as theorem 1, 2 demonstrate, theorem 3 implies update efficiency of scale-invariant weights within a single step eventually will be determined only by predefined hyperparameters, regardless other attributes of the weights (shape, size, position in network structure, or effects from other weights).

heading: 3 The Role of Angular Update in Sphereical Motion Dynamics
text: Far beyond just an index indicating whether equilibrium is reached, angular update is also a important way by which SMD can affect the optimization process of normalized neural network. In this section, we will provide an intuitive explanation on the role of angular update in folklore view of landscape exploration.
The optimization process of gradient descent method is like an expedition to find the bottom of a basin in loss landscape (i.e. a local minimum). It is well known that gradient descent with constant learning rate can always converge to the minimum in strong convex problem as the gradient norm will converge to zero (Boyd et al., 2004); while SGD with constant learning rate can not converge to the minimum certainly,
because noise of gradients can make trajectory “escape from local minimum” Zhu et al. (2018).
However, the behavior of SGD(M) in equilibrium is unique: due to the scale invariant property, the loss landscape is defined on a unit sphere Sp−1, so does the basin (local mimium). Then based on the geometrical meaning of angular update, angular update denotes a "single step length" on the loss landscape. According to theorem 3, when equilibrium state has been reached in SMD, angular update is approximated equal to √ 2λη/(1 + α). Hence with constant η, λ, and α, the optimization trajectory is “stuck” at the top of the basin, unable to reach the bottom (converge/reduce loss), just like the illustration in Figure 2. The value of angular update in equilibrium is totally determined by hyper-parameters, there is no way the optimization trajectory can move to the bottom of basin in equilibrium due to overly large single step length. Based on the analysis above, we have following conjectures: if the trajectory stays in the same basin with different values of angular update in equilibirum, and momentum factor is fixed, then
1. Larger angular update leads to higher training loss and lower test accuracy in equilibrium;
2. Even with different learning rate and weight decay, same angular update leads to similar training loss/test accuracy in equilibrium;
We will verify our conjectures in following empirical study. Note the influence of momentum is different from learning rate and weight decay: even momentum can decrease the value of angular update as theorem 3 implies, it still can increase the oscillating range of the trajectory in the basin, resulting in larger training/testing loss.

heading: 4 Experiments
text: In this section, we verify our theorems and conjectures on SMD and equilibrium by empirical study. We show the equilibrium phenomenon described in our theorems really occurs in various computer vision tasks including ImageNet (Russakovsky et al., 2015) and MSCOCO (Lin et al., 2014). We also analyze an interesting phenomenon as an example to show how SMD can affect training process in a way different from traditional view on optimization of neural network.

heading: 4.1 Fixed learning rate
text: First we verify our theorems on ImageNet (Russakovsky et al., 2015) with fixed learning rate. We use Resnet50 (He et al., 2016) as the baseline model. In SGDM case, the momentum factor is α = 0.9. Form Figure 3(b), 3(e), 3(c), 3(f), we can see empirical value of ||wt||22 and ∆t differ from their theoretical value respectively at very beginning, because the initialized value of weight norm is handcrafted, far away from the theoretical value in equilibrium. After several iterations, empirical values of weight norm and angular update agree with their theoretical values very well, which implies equilibrium has been reached. We also observe SGDM can achieve equilibrium much faster than SGD. According to Eq.(8), (12), the underlying reason might be with same learning rate η and WD factor λ, approaching rate of SGDM ( λη1−α ) is larger than approaching rate of SGD (λη). Results in Figure 3 also prove our claim that equilibrium is a dynamical state: even equilibrium has been reached, E||g̃||22 (Eh̃t) constantly increase, ||wt||22 increases accordingly, ||wt||22 and ∆t always oscillate around their theoretical values respectively, showing equilibrium state maintains in SMD.
Then we train Resnet50 on Imagenet with different hyperparameters to verify our conejctures. The results are shown in Figure 4. We can see the experimental results strongly support our conjectures: in SGD case, whatever learning rate η and WD factor λ is, they have similar angular update in equilibrium, as well as similar training/testing curves; when angular update is larger, both training loss and test error are larger. These phenomena are even more obvious in SGDM case, when momentum factor α is fixed. Besides, the role of momentum in SMD is exactly as we speculate: it can decrease angular update while increase training/testing loss.

heading: 4.2 Multi-stage learning rate
text: Now we study the behavior of angular update with SGDM and multi-stage learning rate schedule on Imagenet (Russakovsky et al., 2015) and MSCOCO (Lin et al., 2014). In ImageNet classification task, we still use Resnet50 as baseline model. The training settings rigorously follow Goyal et al. (2017); In MSCOCO experiment, we conduct experiments on Mask-RCNN (He et al., 2017) benchmark
theoretical value of angular update by
√
2λη 1+α .
using a Feature Pyramid Network (FPN) (Lin et al., 2017), ResNet50 backbone and SyncBN (Peng et al., 2018) following the 4x setting in He et al. (2019).
There appears to be a mismatch between theorems and empirical observations in Figure 5(a), 5(d): angular update ∆t in the last two learning rate stages is smaller than its theoretical value. This mismatch can be well interpreted by our theory: according to Theorem 1, 2, when equilibrium state is reached, theoretical value of weight norm ||wt||2 satisfies ||wt||2 ∝ 4 √ η λ . However, when learning rate is divided by k, equilibrium state is broken, theoretical value of weight norm ||wt||2 in the new equilibrium state is 4 √ 1/k times smaller. But new equilibrium cannot be reached immediately (see Figure 5(c), 5(f)), following corollary gives the least number of iterations to reach new equilibrium. Corollary 3.1. In SGD case with learning rate η, WD factor λ, if learning rate is divided by k, and unit gradient norm remains unchanged, then at least d[log(k)]/(2λη)e iterations are required to reach the new equilibrium state; In SGDM case with momentum coefficient α, then at least d[log(k)(1− α)]/(2λη)e iterations are required to reach the new equilibrium state.
Corollary 3.1 implies SGD/SGDM with smaller learning rate requires more iterations to reach new equilibrium state. Hence, in second learning rate stage in Imagenet experiments, angular update ∆t can reach its new theoretical value within 15 epochs. But in last two learning rate stages of Imagenet/MSCOCO experiments, SGDM cannot completely reach new equilibrium by the end of training. As a result, we observe empirical value of ∆t is smaller than its theoretical value. Based on our theorem, we can bridge the gap by skipping the intermediate process between old equilibrium and new one. Specifically, when learning rate is divided by k, norm of scale-invariant weight is also divided by 4 √ k, SGDM can reach new equilibrium immediately in new learning rate stage. Experiments((b),(e) in Figure 5) show this simple strategy can make angular update ∆t always close to its theoretical value across the whole training process though learning rate changes.

heading: 4.3 “Pseduo overfitting” caused by learning rate decay
text: “Dropping test preformance(accuracy)” phenomenon is often explained as a result of “overfitting” issue, which refers to the phenomenon where a trained model can fit training data very well, but fails to fit additional data for validation or prediction. To handle such issues, regularization methods
like early-stopping (Prechelt, 1998) and dropout (Srivastava et al., 2014) are proposed. But here we show in some cases “dropping test preformance” phenomena is just a “pseduo” overfitting, caused by learning rate decay.
Let’s see the following experiment in which Resnet18 (He et al., 2016) is trained on CIFAR10 (Krizhevsky & Geoffrey, 2009) with standard settings (see experiment settings in appendix) Figure 6 presents the training curves of the experiments. It can be seen from figure 6(b)(blue line inside red ellipse) that with standard implementation of SGDM, test accuracy severely drops during the 2nd learning rate stage. It seems a typical overfitting issue. However, two other phenomena cannot be well interpreted by overfitting: 1) test accuracy only severely decreases during 2nd learning rate stage, it performs normally in the 3rd, 4th learning rate stages (blue lines in Figure6(b)); 2) When other optimization method like Adam (Kingma & Ba, 2015) is used, test accuracy does not drop apparently in any learning rate stages (green line in Figure6(b)).
Now SMD can provide a more reasonable interpretation: we have demonstrated the conjectures on the relationship between angular update and performance of model (Sec. 3), and the fact that learning rate decay can make angular update increase (Sec. 4.2) above, so it’s reasonable to speculate that learing rate decay is the cause of the temporary “dropping test accuracy” phenomenon. See illustration in Figure 7: after shrinking learning rate, smaller angular update allows the optimizaton trajectory to move towards the bottom of the basin, then angular
update will increase, forcing trajectory to “escape” from the bottom, resulting in dorpping test performance. If this interpretation is ture, then rescaling strategy introduced in Sec. 4.2 should avoid “pseduo dropping test performance” by eliminating increasing angular update phenomenon. Further experiments support our speculation (Figure 6). They also explain why dropping test performance cannot be seen in Adam: we track the angular update with Adam, and no increasing angular update phenomenon is observed. Please see more discussion in appendix.

heading: 5 Conclusion
text: In this paper, we comprehensively reveal the learning dynamics of normalized neural network with SGD/SGDM and weight decay (WD), named as Spherical Motion Dynamics (SMD). With mild assumptions, we strictly prove SMD will reach equilibrium state in a linear regime. We also propose a novel index, angular update, to depict the state of SMD, and derive its theoretical property in equilibrium. Most importantly, we show our theorem is widely valid, they can be verified on challenging computer vision tasks, beyond synthetic datasets. Besides, we show SMD can dramatically effect the optimization of neural network by controlling angular update in practice. We believe our results on SMD make an important step to understand the mechanism of deep neural networks, and can inspire new deep learning techniques.
Acknowledgments and Disclosure of Funding
This work is supported by The National Key Research and Development Program of China (No. 2017YFA0700800) and Beijing Academy of Artificial Intelligence (BAAI). We really appreciate the valuable suggestions on proof of the theorems from Dr. Donghao Wang and Dr. Siyi Yang.

Review summaries:
In this paper, the dynamics of scale-invariant (SI) weights of normalized neural networks trained with SGD(M) + WD are studied. It is known that the scale-invariant parameters' intrinsic domain is a unit hypersphere, which the authors have effectively taken into account by introducing the "angular update" (AU) term, and the gradients w.r.t. SI parameters 1) scale inversely proportional to parameters norm and 2) are orthogonal to the radial direction in parameter space: the first property compels to introduce a unit gradient term and condition the dynamics of SI parameters on the intrinsic domain, the second one, together with weight decay, gives birth to the main concept of the paper &mdash; Spherical Motion Dynamics (SMD). Important attainment of this work is that, in contrast to the prior art, the notion of SMD equilibrium is not static (i.e., when $\Vert w_{t} \Vert_2 = \Vert w_{t+1} \Vert_2$) but represents a dynamic state when parameters norm oscillates around its theoretical value determined by hyperparameters and (generally varying) unit gradient norm. Interestingly, despite the dynamic nature of the equilibrium in terms of weight norm, the authors have shown that AU, in contrast, becomes approximately constant with its value only dependent on hyperparameters: learning rate, weight decay, and momentum factor. The authors corroborate their theoretical findings with experiments involving different computer vision tasks and networks architectures. They also provide an intriguing view on overfitting from the SMD perspective: when the learning rate drops, test error sometimes increases which could be interpreted as escaping from the local minimum due to an increase in the angular update.

This paper investigates the time-evolution of weights in normalized neural networks (e.g. one in which every layer has batch-norm applied). It proposes theorems regarding convergence to equilibrium, in which the norm of the weights approaches a fixed value. Compared to prior work, which established the fixed value, this work additionally gives the approach rate, and the scale of the variance due to stochastic gradients. The paper introduces the angular update parameter, and shows this approaches a fixed value. The paper emphasizes that equilibrium is not reached immediately, but is approached dynamically. It also asserts that equilibrium is not about convergence of stochastic oscillations to zero, but instead convergence to a fixed value, up to oscillations. The theoretical results are then investigated empirically, showing agreement with theoretical results for the weight norm and angular update, and the dynamical approach to equilibrium.