Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper studies the dynamics of Spherical Motion Dynamics (Stochastic Gradient Descent with momentum and Weight Decay), more specifically, the behavior of “effective learning rate" in the “equilibrium" state. The paper (1) gives the assumptions for the equilibrium state and prove the equilibrium can be reached in linear time; (2) describes an “angular update” as a substitute for effective learning rate to depict the state of SMD; (3) verifies the assumptions and theories on ImageNet and MSCOCO.
main_review: Pros:

+ This paper gives solid and rigorous theoretical results on the dynamics of Spherical Motion Dynamics. The study is timely and important. The proofs are given in detail. The results are well motivated. It may shed light in understanding deep learning (while this part is not discussed sufficiently; see below).

+ The theoretical results are well motivated. The authors give sufficient discussion to the idea of the theoretical study.

+ The paper gives clear proof sketch. The proof idea is clearly presented. Readers can easily follow the idea without checking the whole proof details in the appendices.

+ The paper gives comprehensive empirical results on very large-scale datasets, ImageNet and MSCOCO. The empirical results are in full agreement with the theory. This is much impressive for a theory paper. 

Cons:

- The paper does not give a sufficiently comprehensive literature review. The presented contributions are not well positioned in the current state of research. Readers may fill confusing for some of the results. The advantages and difference of this work from the existing results.

- The paper does not give enough practical implications of the presented theory. The authors are encouraged to discussion how this work can contribute to future novel algorithm design.

- The paper has many typos. It seems written in a rush. I recommend the authors carefully proofread the paper in the next version.
limitations_and_societal_impact: Yes. The authors discuss the conflict between the theory and experiments in remark 3 though it is not so serious.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper investigates the time-evolution of weights in normalized neural networks (e.g. one in which every layer has batch-norm applied). It proposes theorems regarding convergence to equilibrium, in which the norm of the weights approaches a fixed value. Compared to prior work, which established the fixed value, this work additionally gives the approach rate, and the scale of the variance due to stochastic gradients. The paper introduces the angular update parameter, and shows this approaches a fixed value. The paper emphasizes that equilibrium is not reached immediately, but is approached dynamically. It also asserts that equilibrium is not about convergence of stochastic oscillations to zero, but instead convergence to a fixed value, up to oscillations. The theoretical results are then investigated empirically, showing agreement with theoretical results for the weight norm and angular update, and the dynamical approach to equilibrium. 
main_review: **Originality**

As stated in the paper, the fixed value of the weight norm was already found in van Laarhoven, so the extra piece this work adds here is the rate of approach and the scale of the stochastic variance. It also has a corresponding result for momentum. For the angular update, the equilibrium fixed value had already been found Chiley and other papers (see also Kunin). The added bit here seems to also be the momentum term. I am not aware of whether the other papers conducted a similar experimental verification of their results. It would be good to clarify if Kunin is to be treated as concurrent or as prior work. The related work section was vague on how this relates to recent work, lines 124-126.

**Quality**

I have not checked the proofs of the statements in the appendix, but the experiments seem to be of high quality overall. My main comment here is that fig 4b regarding the drop in test accuracy could be strengthened by running more seeds etc as this might just be a random variation. 

**Clarity**

The submission is clearly written and well organized. The weakest section is 4.2, where the motivation and explanation of the experiment is not that clear. Also, I’m not sure why a new term ‘spherical motion dynamics’ is necessary, and may also be misleading as the motion may sometimes not be restricted to a sphere (if I've understood correctly, when out of equilibrium, the motion is not restricted to the sphere). 

**Significance**

If the main original contributions are the rate of approach and scale of variance, why are these useful? This is not explained or motivated. Similarly for the extension of existing results to the case of momentum. It would be good to explain further why this is useful to a practitioner.


-----

**Edit**: 

I have updated my score from 6 to 7 based on the originality of this work being more clear to me, following author responses. I now see that the value to which the weight norm square tends to in equilibrium is not a constant value (as in prior/concurrent works), but is a non-constant value, due to the expected squared norm of the gradients being non-constant. This is also empirically demonstrated in Fig 2b,e. 

The author reply also assured me how the clarity of the work will be improved in a possible camera-ready version.
limitations_and_societal_impact: The paper has no discussions of limitations (technical or otherwise), nor of societal impact. Since this is mostly theoretical, the societal impacts are not direct, so this seems ok. But I would like to see discussion of technical limitations - for example, what about networks in which not all weights are scale-invariant (just a subset), eg ones where only a few types of layer are normalzed.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper, the dynamics of scale-invariant (SI) weights of normalized neural networks trained with SGD(M) + WD are studied. It is known that the scale-invariant parameters' intrinsic domain is a unit hypersphere, which the authors have effectively taken into account by introducing the "angular update" (AU) term, and the gradients w.r.t. SI parameters 1) scale inversely proportional to parameters norm and 2) are orthogonal to the radial direction in parameter space: the first property compels to introduce a unit gradient term and condition the dynamics of SI parameters on the intrinsic domain, the second one, together with weight decay, gives birth to the main concept of the paper &mdash; Spherical Motion Dynamics (SMD). Important attainment of this work is that, in contrast to the prior art, the notion of SMD equilibrium is not static (i.e., when $\Vert w_{t} \Vert_2 = \Vert w_{t+1} \Vert_2$) but represents a dynamic state when parameters norm oscillates around its theoretical value determined by hyperparameters and (generally varying) unit gradient norm. Interestingly, despite the dynamic nature of the equilibrium in terms of weight norm, the authors have shown that AU, in contrast, becomes approximately constant with its value only dependent on hyperparameters: learning rate, weight decay, and momentum factor. The authors corroborate their theoretical findings with experiments involving different computer vision tasks and networks architectures. They also provide an intriguing view on overfitting from the SMD perspective: when the learning rate drops, test error sometimes increases which could be interpreted as escaping from the local minimum due to an increase in the angular update.
main_review: **Originality**  
Despite a large volume of the related literature (well cited by the authors), I find the contributions of this paper novel and original. I put a special emphasis on the notion of angular update, dynamic interpretation of the equilibrium, formulation of the main results in discrete setting (instead of continuous SDE approximation), and generalization to the SGD with momentum case.

**Quality**  
The submission is overall technically sound, although there are some concerns about the reasonableness of assumptions, experimental setup, and clarity of proofs (see Weaknesses below). The main claims are supported both by theoretical analysis and experimental results. All in all, I consider this submission a complete piece of work.

**Clarity**  
Errata aside, this paper is clearly written and well organized. The authors provide enough details in the main text and appendix to reproduce the experiments and verify the claims.

**Significance**  
The results of this work are important and provide valuable insight into the dynamics of learning of normalized neural networks significantly differing from previous works (see Originality). Notably, the authors state that the effective stepsize of SI weights will eventually be determined only by hyperparameters, regardless of any other attributes like shape, size, position, etc. While the results are indeed intriguing, their validity can be compromised by the limitations below.

**Weaknesses**  
Below I provide a list of principal concerns about this submission. I would like the authors to address them primarily.
1. Here I question the assumptions made in Theorem 1, though similar concerns apply to Theorem 2 as well.
    1. Assumption 2 requires $V$ to be a *uniform* bound on the conditional squared unit gradient norm variance which can be a large value and lead to vacuous results (see below). Could this uniformness assumption on variance be weakened?
    2. Even more important, in Assumption 4 you set a *uniform* lower bound $l$ on the square unit gradient norm that, in addition, must not differ too much from the conditional expected squared unit gradient norm $L_t$ for *all* iterations. The second part does not allow the derived results to hold when the network converges to a minimum and the unit gradient norm tends to zero, so the lower bound is not valid anymore. I believe that might be the reason why the plots in Figure 2 depict only the beginning of training when the gradient norm is still high. Again, could this assumption be further weakened?
    3. Assumption 3 demands $L_t$ to change as $\mathcal{O}\left( (\lambda \eta)^{3/2} \right)$, i.e., very slowly since in assumption 1 you state that $\lambda \eta \ll 1$. I do not find this "smoothness of mean squared gradient norm dynamics" condition convincing enough, especially considering the possible heavy-tailness of the stochastic gradient noise yielding sudden jumps in gradient norm (see, e.g., https://arxiv.org/abs/1901.06053). Furthermore, a concurrent work (https://arxiv.org/abs/2106.15739) reveals that such jumps can in fact be regularly observed in normalized networks.
2. Considering that $V$ can be larger than one (see Figure 2a) and $L_t l$ is always below one (as a product of mean and min values), the value $\sqrt[3]{\frac{V}{L_tl}}$ can easily exceed one, hence the statement in Theorem 3 becomes vacuous: something can happen with probability larger than a negative value. I appreciate that the authors note in Remark 3 that their theoretical bound is too large compared to the empirical evidence, however, this issue may be of secondary importance. I expect the authors to reconsider the statement in Theorem 3. Would a formulation in terms of the *expected* angular update fit better (especially in light of the empirically observed behavior)?
3. In the process of reading, I encountered several questions about Figure 2, so some additional comments on it in the text would be appreciated. 
    1. Why does the gradient norm start from zero and only grow? If I understand correctly, it does so due to the vanishing gradient effect for the first layers at the beginning of training and a limited number of iterations. However, usually, the total gradient norm (i.e., w.r.t. all network SI parameters) behaves differently, so does the behavior hold for all layers and the whole network? A brief comment on this would help a reader. While the lines for different layers are presented in Figure 3, more examples in the setting with a fixed learning rate in the appendix would also be beneficial.
    2. Does the behavior of angular updates remain stable after gradient norm saturation? Since the paper is about reaching a certain equilibrium state, adding at least one longer experiment would make the empirical results more convincing and even partially address the concerns about assumptions in Theorems 1 and 2.
4. About the “Overfitting” part:
    1. In line 331 of the main text, you write that "Adam has no equilibrium state like SGDM". Why so? Could you, please, comment more on this? A recent work https://arxiv.org/abs/2006.13382 could be relevant.
    2. The explanation in lines 324-328 of the main text seems vague. Rewriting this part using the intuition from Appendix F would make the text clearer.
    3. I suggest putting the bolded text from lines 493-494 of the appendix into the main text to avoid confusion that rescaling strategy helps generalization.

Minor concerns:
* When stating the main theorems, you use different iterations for defining certain important values in the main text and the appendix (compare, e.g., definitions of $w_t^*$ in line 148 of the main text and line 129 of the appendix). Also, heed the conditioning on $w_t$ when defining $L_t$ in lines 145 and 184 of the main text (compare, again, with lines 126 and 185 of the appendix, respectively). While I believe that the main claims are still valid, I would like the authors to fix these inconsistencies.
* I admit that I did not thoroughly follow all the proofs, however, I managed to find several inaccuracies. For instance, please explain what is going on in Eq. (38): e.g., what is $L$? Also, I believe you missed $2$ in Eq. (58), however, I still do not completely understand how it follows from equations (56) and (57), namely, why did you put $(1 - 2 \lambda \eta)$ in the denominator? I also suppose that the inequality signs must be $>$ in the second line of Eq. (149). Considering these and other issues, I advise you to revise your proofs more thoroughly.
* In Appendix E (and partially in the main text), you conjecture that larger angular updates lead to worse performance. However, I suppose that a more accurate formulation would be "*too* large angular updates *may* lead to worse performance" due to the following reasons: first, you consider only two AU values in Figure 3a-c in the appendix, which is not enough for making far-reaching conclusions; second, there exist plenty of works showing that larger learning rates can aid generalization by escaping from narrow minima (see, e.g., https://arxiv.org/abs/2003.03977 and references therein); third, the experiments with momentum, as noted in the text, contradict this conjecture (the explanation that momentum prevents optimization from reaching the bottom of the loss landscape is unconvincing since momentum is specifically tailored for improving convergence); finally, at the end of Appendix F you confirm that growth of AU is not always followed by test accuracy drop. I suggest relaxing the conjecture or provide more experiments supporting it.

---

**UPD**  
After the discussion, I have found most of my concerns addressed and decided to increase my initial score. I also hope that the authors will improve their paper based on our discussion: formulate their theorems more accurately (e.g., modify the unnecessary uniformness of bounds), provide additional longer experiments with saturating gradients (or even periodic behavior), change the demonstration of some of their claims to evade misunderstanding (e.g., about "Adam has no equilibrium state like SGDM"), etc.
limitations_and_societal_impact: I appreciate that the authors have noted some limitations of their work and possible future research directions  (e.g., in Remark 3 and at the end of Section 4.2), however, I find several important flaws in this submission to be addressed (see Weaknesses above).
review_title: null

Meta-review: This is a clear accept.  Aside from the high scores of all the reviewers, I myself have seen versions of this paper in the past on arxiv (it was inexplicably rejected at a previous submission, but is even improved here) and have really benefitted from this work's thinking.  I think the contribution of this work is clear and important.   I think we'd be foolish not to accept this work here this year.  
Decision: Accept (Spotlight)