Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors perform a rigorous study of training regimes for neural networks with normalized weights, which is a common practice. There is a mathematical analysis, using an example problem, and empirical study on standard image classification benchmarks. The paper culminates with a comparison of training regimes on a Resnet18 on cifar10 trained with and without scale invariance. 
strengths_and_weaknesses: The analysis is thorough, and paying attention to normalization methods makes this paper a solid contribution. Practitioners are more familiar with “normal” training techniques, but in the real world, spherical projections are common, but not as well understood. The authors make a good case that the training behavior changes significantly, and the presentation which will help readers build intuition to interpret these in practice. Section 6 which compare-and-contrasts is exceptional at achieving that end.

One weakness is that the paper only focused on CIFAR-10/0 with convolutional nets. These problems are over-optimized that batch norm alone works. It would be interesting to see if the analysis applied to other domains and architectures, such as MLPs or transformers, where spherical projection is more common. That would maximize the paper’s impact, but it is substantial as-is.
questions: # Section 6 questions

The experiment setup in section 6 needs some clarification. What exactly is teh difference between the three cases in Figure 7?
What exactly does it mean to do scale invariance “in the whole parameter space”? How is this different from the models in the previous sections?
To clarify, is spherical projection applied during training for these models? If so, for which?
Why did you vary LR instead of ELR?
How are batchnorms applied in all three cases? Does the ResNet 18 contain batch normalization layers in the conventional training runs?
Figure 7: Why is the test error missing on the rightmost SI figure? Did these networks perform worse? 
Which of the methods in Figure 7 is “the best” in terms of accuracy, training performance, robustness, etc.?
Line 340: How does data augmentation make the problem “too hard”? Shouldn’t it make the problem easier?

# Clarification points

The first line of the paper calls out batch normalization and layer normalization but these methods normalize hidden states, not the weights themselves. Spherical projection normalization is different. To which normalization techniques exactly does the analysis apply? The first paragraph would be better to clarify this.

The text needs better descriptions of architectures. What architecture is “ConvNet” exactly? What are the activations, layer width, etc.

# Discussion questions:

Given these results, what observation can I make in practice to inform decisions on model training? Do the results suggest a way of determining scale invariant weights will improve my model?

Does your work show any advantages to using scale invariant weights? Are there any roadblocks that your analysis removes from using scale invariant weights in practice?

# Minor

Figure 7: I first read “Conv. training” and “convolutional” instead of “conventional”; it’s a confusing overload of an abbreviation.

Extra space in line 3 of Figure 7 caption , .

limitations: The text addresses the limitations of the work. As an empirical understanding paper, there are no major limitations I see, and it is a contribution as-is.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies properties of training scale invariant nets using fixed learning rates. In this setting, it further characterized 3 distinct regimes where qualitative behavior NN training and generalization is different. These properties hold both in an illustrative toy problem and practical neural nets.
strengths_and_weaknesses: Strengths
- Solid empirical results and conclusions follow directly from the evidence
- The paper also does a good job of setting up the background and situating itself with related work.
- Toy problem is not novel but is indeed illustrative

Weaknesses
- Figures are too small, I would encourage the authors to make them bigger for improved reading experience 
questions: I like the paper overall, I'm curious if the authors can suggest future avenues of direction if and how these insights can be used to improve either training or generalization performance of practical (maybe scale invariant) neural nets.
limitations: Yes
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studies the training behavior of scaling invariant neural networks using projected SGD with fixed effective learning rate on the sphere. The topic is meaningful because neural network models become (semi) scale invariant when normalization techniques are applied. Theoretically, the authors point out several relations satisfied by the effective learning rates and effective step sizes of individual parameter groups and the whole parameter vector. With the relations, the dynamics of individual parameter groups' effective learning rates are derive, showing a "negative feedback" principal, which is later used to explain the "chaotic equilibrium" regime of training. Also theoretically, a simple scale invariant function is analyzed to study the different optimization behaviors shown by different effective learning rate. 

Empirically, experiments are conducted on both scale invariant neural networks and regular neural networks. Three regimes are identified for the behavior of the training loss: convergence, chaotic equilibrium, and divergence. The three regimes happen in order when the effective learning rate is increased. Extensive experiments are conducted to observe and compare the test error and sharpness of solutions for the three regimes. Discussions are made trying to explain the three regimes using the theoretical results. 
strengths_and_weaknesses: Originality: The theoretical results obtained in Section 3 is new. Also, the extensive experiments on scaling invariant neural networks with fixed effective learning rate are new. Though, the reviewer does not think the observations dramatically change our understanding on neural networks' training process. It is a common belief that, even for non-scale-invariant networks, the loss experiences smooth descending, oscillation, and divergence when the learning rate is increased. The experiments and resulting observations on the sharpness of solutions, and jump between basins, also exist in previous works. The results for scale invariant networks are not different. Also, the theoretical results are not sufficient and convincing enough to explain the empirical behaviors. 

Quality and Clarity: The paper is well written and easy to follow. The descriptions and discussion on numerical experiments are clear.

Significance: The work has fair significance on our understanding to neural networks with normalizations. It shows some widely known phenomena in training also hold for scale invariant models trained with fixed effective learning rate. 
questions: 1. It there any difference in the training behavior of scale invariant networks and regular networks? If such differences exist, are they able to be explained by the theoretical insights obtained in this paper?

2. For the toy function (7), it seems there no divergence regime by proposition 1. Theoretically understanding, what happens when the behavior moves from regime 2 to regime 3?

3. The first line of equation (1) is confusing. 
limitations: Limitations are discussed in the last section.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper investigates the directional optimization dynamics of training scale invariant neural networks. By fixing the effective learning rate (ELR), the paper seeks to decouple the influence of norm dynamics on the gradient from directional dynamics. The authors observe that training can behave according to three distinct regimes depending on the chosen ELR value. The various regimes are carefully analyzed both theoretically and empirically through the use of toy examples as well as more realistic settings. The authors utilize their framing to offer explanations to various conflicting observations in previous works, as well as share suggestions on how to use their insights for training more conventional neural networks.
strengths_and_weaknesses: **Strengths**
- [**quality**] the main statements presented in the paper are generally well supported by clear, theoretical foundations accompanied by well explained proofs in the appendix. 
- [**clarity**] the paper is clearly written and overall easy to follow. It also provides an extensive appendix containing additional discussion as well as further edge case experimental results to aid replication and completeness.
- [**significance**] the authors carefully analyzed a very specific setting that does not seem to have been explored in isolation before. However, because it touches on quite broad learning settings for neural network training - the (partial) use of scale-invariant modules - it could provide guidance and save time for other researchers meaning to conduct research in this topic, or practitioners analyzing observed learning behavior of their models.


**Weaknesses**
- [**clarity**] the provided plots are not always easy to interpret, as they are quite small with lines strongly overlapping. A possible suggestion to increase the legibility would be to enlarge the most important plots and move others to the appendix.
- [**significance**] while the paper does a great job at completing an isolated analysis seemingly absent from the relevant literature, it is not entirely clear to this reviewer how the provided results, i.e. the existence of three optimization regimes depending on the effective learning rate for (partially) scale-invariant NN, should be utilized in practice. Generally, the paper could be enhanced with more direct take-aways or suggestions as to how the results presented in this work should be used or build upon.

questions: **Questions:**
- Section 3.1, starting line 102: what is the difference of the method described currently versus the statement ‘one learning rate for all spheres is overconstraining’? This difference still appears unclear after reading Appendix B
- Section 3.2, starting line 152: the analysis in this paragraph at some point switches from features of regime 1 to regime 2, yet it’s not quite clear exactly where this happens?
- Section 3.2, line 158/159: ‘is also non-vanishing (we give a more formal argument in the appendix)’ → how should one interpret the relevance of ‘non-vanishing’? Perhaps point to specific appendix?
- Section 3.2, starting line 171: for the toy example, it is clearly demonstrated how constant ELR are chosen to end up in one of the three regimes. Would the authors have a suggestion how this type of reasoning of picking ELR values could be extended to the more realistic settings beyond trial and error?
- Section 4 (related to the previous question): could the authors elaborate as to how the learning rates are chosen for the conducted experiments?
- Section 5.1: in the discussion surrounding picking different ELR values in the first regime, it is not entirely clear how the following concepts all move together: speed, loss, sharpness, robustness. Would it perhaps be possible to include a small table with this? E.g. rows: (speed, loss, sharpness, robustness), columns: 1st regime low ELR, 1st regime high ELR
- Section 6, starting line 336: could the authors elaborate why NaN values would eliminate the third regime in this setting compared to the previous settings?

**Suggestions:**
- [Throughout paper] sphere → hypersphere or n-sphere
- Section 3.1, equation 1: perhaps change R^P to S^(P-1) to indicate hyperspherical space?
- Section 3.2, line 122: ‘it can be shown’ → maybe point to relevant appendix or related work?

**Possible typos**:
- Section 3, line 87: ‘alike normalized neural networks’
- Section 3.2, line 125: ‘from what follows’ → ‘from which it follows’
- Section 3.2, line 156: ‘of individual’ → of the individual

limitations: The authors adequately addressed limitations of their work from a technical point of view. However, as pointed out in the '*weaknesses section*': (1) more practical advice around picking 'good' ELR values/ranges (or when this is possible/impossible), and (2) clearly stating hopes/expectations as to how/where this work could be improved upon would likely strengthen this work. Negative societal impact does not seem relevant.
review_title: null

Meta-review: metareview: All reviewers find the papers analysis on optimization regimes for scale invariant networks interesting and observations novel. The results are clearly presented and well supported by the (limited) analysis. However the reviewers also highlight several drawbacks
1) the paper mainly presents analysis for a scalar function
2) experiments limited to Cifar datasets with ConvNets.

While the results for presented settings are convincing, I think it is hard to judge the universality/importance of the phenomenon from analysis of scalar functions and experiments only on Cifar. Overall I think the paper is on borderline and suggest acceptance as the phenomenon is well presented and can lead to further works building on this. I encourage authors to include more experiments on different datasets and model architectures in the final version.
confidence: Less certain
Decision: Accept