Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Post-response update: I thank the authors for their response. I updated my score, but still think the paper needs improvement to be of interest to the ICLR community.

---


The submission analyzes the behavior of gradient descent and adaptive variants for scale-invariant models, including batch-normalization, by looking at the trajectory of the iterates when projected on the unit sphere. It contributes a formula for the equivalent learning rate if the gradient step was to be taken on the unit sphere for SGD and Adam and shows an approximate equivalence between gradient steps and normalized gradient steps taken on the unit sphere.

The spherical perspective is an interesting and insufficiently explored aspect of modern machine learning models. As the magnitude of the weights is essentially an irrelevant free parameter, understanding its behavior is likely to yield insights on how we train those overparametrized models.

The submission, however, does not provide a strong motivation for the presented results. The stated contribution that "in the presence of BN layers, standard SGD behaves like Adam without momentum" is also slightly over-selling the results of section 3. The main issue is that the significance of the explicit learning rates derived in section 2 or the equivalence between gradient descent and a normalized variant in section 3 is unclear. The manuscript does not provide sufficient context to understand what the derived formulas change about our theoretical understanding of those methods or what they imply for applications. 

As it stands, I am worried that the submission would have little impact. I believe the manuscript would benefit from major revisions to be of interest to the community and my initial recommendation is a rejection.

The major issues that need to be adressed are:
- The technical definitions of the terms introduced in the work, like "equivalence of order 2", and the significance of the results, need to be stated in the main text. That two methods produce similar updates in terms of the overall model is interesting. But the significance of the result depends on what information it gives for the study or the application of the method, and this is currently unclear from the manuscript. For example; 
  - why does the memory $v_k$ in the AdamG* update increase over time (with $\beta > 1$) if $\eta\lambda >2$? Or is it a setting that is not supported by the assumptions in Thm. 2? 
  - What is the effect of L2 regularization if the complexity of the model is independent of the magnitude of the weight vector? 
  - Why is $\beta = 1$ if there is no L2 regularization but $\beta < 1$ if $\eta\lambda < 2$? 
  - Does it tell us anything about the difference between an Adagrad-style complete sum and Adam-style exponential moving average?
- The claimed contribution that SGD behaves like Adam without momentum is over-selling the results of section 3. Those results show that a gradient step is approximately a normalized gradient step projected on the unit ball. While this version might have similarities with Adam, this is not what the stated contribution would imply to most readers.
- The message of the paper is obscured by excessive formality. It might of course useful for a subset of readers to state the result in terms of topological equivalence to quotient manifolds, isomorphisms or to relate the results to the canonical metric on the sphere. But it makes the paper harder than necessary for the average reader, especially when technical terms, like the canonical metric, do not appear after their introduction. I strongly advise the authors to keep the main message of the paper accessible. 

review_title: Interesting perspective but unclear consequences

final_score: rating: 4: Ok but not good enough - rejection
review_content: This work studies optimization dynamics for neural network models that are scaling invariant with respect to parameters. A general formulation of optimization algorithms is considered, covering many widely used algorithms like SGD and Adam. The projected dynamics (to the unit sphere) is studied, and the effective learning rate and update direction on the unit sphere are derived. Focusing on the projected dynamics, the equivalence is built between SGD and a type of "Adam". Then, different factors in the Adam dynamics that can potentially influence the optimization performance are identified, and empirically studied. 

The paper is overall clear and easy to follow. My major concern is that the results are weaker than that is claimed in the abstract. Specifically,
1. The image optimization on the hypersphere is just a projected dynamics, it is not a close dynamics with respect to u. The radial part, although does not contribute to the loss directly, still plays a role in the dynamics. In this sense, the projected dynamics is not simplified compared to the original one. The effective learning rate is not easy to control. Moreover, the loss function of u is still highly non-convex and hard to study. No evidence is shown that L(u) is simpler than L(x).
2. The adaptive gradient methods are adaptive in at least two ways: a) the learning rate is adaptive; b) the learning rate is different for each parameter. The AdamG* considered in the paper does not use element-wise learning rate. It is more like an SGD with a learning rate schedule, instead of an adaptive gradient algorithm. Hence, the sentence "performing SGD alone is actually equivalent to a variant of Adam constrained to the unit hypersphere" in the abstract is misleading and conveys over-optimistic information. 
review_title: Weaker results than claimed

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper attempts to analyze Adam with BN from a spherical perspective. 

1. Though the authors derived the effective learning rate for Adam in Eq. 13, the analysis still needs to be more thorough. For example, how this effective learning rate is affected by the hyperparameters (i.e. learning rate, batch size). And in Eq. 13, since this term is complicated, involving ck, uk and b, a detailed and intuitive analysis should be provided. 

2. About the second contribution, the paper claimed that SGD with BN behaves like a variant of Adam, AdamG*. However, this AdamG* has significant difference from original Adam: in AdamG* the division is by a *scalar* (not an element-wise division in Adam), this does not change the radially-invariant property of models. Thus, the connection between SGD with BN and AdamG* is obvious. I expect the authors provide some analysis on the connection between SGD with BN and AdamG*. 

So I think is the contributions in this work are slightly below the acceptance bar of ICLR before these concerns are solved appropriately. 
review_title: The analysis might be not sufficient

Meta-review: Three reviewers recommend rejecting or weak reject. The studied problem is interesting, but as one reviewer pointed out, it is not that clear how this work changes our theoretical understanding of those methods or what they imply for applications. Overall, I feel this work is on the borderline (probably it deserves higher score than the current score), but probably below the acceptance bar at the current form. 
Decision: Reject