Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The current paper argues that benefits of BatchNorm in regards to generalization have been under-discussed and tries to develop a novel analysis for the same. To this end, the authors prove that under weight decay and a scale-invariant loss, as induced if there is a BatchNorm corresponding to all parameters, the model's training functions in two regimes: stable and edge-of-stable (EoS), which arises because the use of weight decay forces parameter norm to increase. In the EoS regime, under assumptions such as existence of a manifold of minimizers, it is shown that scale invariance will force the model to follow a "sharpness-reduction" flow, i.e., a gradient flow under which the sought solution is trying to minimize sharpness. Sharpness is defined by taking scale-invariance into account.
strengths_and_weaknesses: ### Strengths
I think the analysis in this paper is strong and the proposed tools, though extensions of prior works to some extent, can be of interest beyond the paper's goal. The overall story is very well laid and I really enjoyed reading the paper. 


### Weaknesses

I think the paper has few primary weaknesses in its current form and they should be easy to address. I strongly encourage the authors to do the same and am happy to raise my score accordingly.

* Related work: While the related work is decently broad, I think a few highly relevant papers have been missed. Specifically, Luo et al. (https://arxiv.org/abs/1809.00846) developed an objective that approximates BatchNorm's regularization effects. It would be important to cite the paper and compare if its results relate with the proposed sharpness reduction flow (could they be similar in some settings?). Second, Tanaka and Kunin (https://arxiv.org/abs/2105.02716) developed a detailed relationship between models trained using SGD under normalization layers and adaptive gradient methods such as RMSprop. The authors claim their proposed link is the first, but this paper has a similar relationship derived via analyzing symmetry breaking of scale invariant losses under fiinite learning rate. It would be important to highlight how the authors' results differ.

* Experiments: While I enjoyed the analysis, I think the experiments lack some ablative baselines demonstrating how lack of BatchNorm would have not yielded similar generalization or similar sharpness reduction. For example, in Figure 2 with CIFAR-10 results, one can train the model without BatchNorm and achieve similar or better test accuracy using the initialization strategy proposed by Daneshmand et al. (see section 4 of https://arxiv.org/abs/2003.01652). Would one see a similar reduction in sharpness? In fact, Daneshmand et al. argue in their work that BatchNorm's effectiveness cannot be explained via scale invariance and as long as one gets a rank-maximizing signal flow at initialization, good generalization is achieved. Then, the experiment suggested above will help understand whether the current paper's position is correct.

* Extension to other layers: I think it is important to include discussion on why similar results on sharpness reduction will not be seen with other normalization layers. Technically, most normalization layers have scale invariance at some granularity (e.g., channel-level in InstanceNorm or layer-level in LayerNorm). Several of those layers are unable to match BatchNorm in its effectiveness, but some of them do when they have similar signal propagation characteristics, as shown by Lubana et al. (https://arxiv.org/abs/2106.05956). 
questions: See weaknesses. 
limitations: Primarily, there is scope for improvements in experiments and discussion on relevant related works.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper shows that normalization along with weight decay biases full-batch gradient descent with finite step sizes towards flat solutions.
strengths_and_weaknesses: Strengths and Contributions:

-The paper’s organization is great, and it’s easy to follow.

-This paper explains the EoS phenomenon theoretically, which to my knowledge was previously a puzzling unexplained observation.

-The paper does a reasonable job of providing intuition behind the theory.


Weaknesses

-Throughout the paper, starting in the first sentence of the intro, grammatical errors make the paper a bit hard to read, but I was still able to understand everything after re-reading some sentences a couple times.  Not a huge deal but definitely worth fixing before publication.

-Empirically, the benefits of weight decay are unclear.

-Other works have found that full-batch GD finds sharper mins than SGD in the absence of, for example, gradient clipping.  The difference between sharp/flat properties of full-batch and stochastic training calls into question exactly what this work can tell us about the dynamics of training routines people actually use, which are nearly always stochastic.  It would be interesting to connect your findings to the effects of stochasticity on training.

-The empirical validation is very sparse.  I realize that there has already been a lot of work on the sharp/flat hypothesis and normalization layers from an empirical perspective, but ablations on the assumptions of the theory (i.e. experiments to see if the predictions of the theory still hold empirically under weaker assumptions) would be interesting.

-Minor point: the scale invariance problem for flatness measurements like the spectral norm of the Hessian is a very special (and particularly coarse) case of a broader space of reparameterization problems by which different parameter settings may yield the same function but not the same flatness.  A number of stronger invariant metrics have been proposed, for example invariant to filter-wise rescaling, but it would also be interesting to show function-space properties (e.g. classifier margins, etc) rather than parameter-space ones.

questions: N/A
limitations: The authors do discuss their assumptions in detail, and they do mention the limitations of their work at understanding broader deep learning phenomena.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper concerns on the optimization aspect when training deep neural networks (DNNs) with normalization techniques. Normalized DNNs are prevalent and often have high performance in practice. However, little is theoretically known. This paper analyzes the role of using normalization and weight decay when training by gradient descent (GD). This paper shows that normalization + weight decay can help GD move to flat region, can impose an implicit bias to reduce sharpness. The authors find that when GD reach the manifold of minimizers, it can move to the Edge of Stability regime. Once in this regime, we can move to a less sharp region on the manifold when using more GD steps. Some experiments on VGG-11, ResNet-20, matrix completion, and linear regression using Batch normalization further support their theory.
strengths_and_weaknesses: Strengths:
- This paper provides an interesting view about training normalized DNNs by GD, adding to our knowledge about the behavior of practical DNNs. After reaching the manifold of minimizers, it can move to the Edge of Stability regime, and continue moving to a less sharp region. There is close connection between flat minima and generalization.
- Some experiments on some DNNs, matrix completion, and linear regression are done to further support their theory.

Weaknesses:
- Despite of the main focus on normalization, this paper uses only "scale-invariant" property to derive their results. This property appears in some methods such as layer norm, instance norm, group norm. However, some other normalization methods (e.g., batch norm - BN) do not ensure scale-invariant loss. The reason is that BN computes the variance of an input signal from the whole training data, and therefore an BN operation is data-dependent and will be dynamic over the training course. This fact violates the assumption about scale-invariant loss in this paper.
- Due to the above reason, the scope of this paper is unclear and the theory may mismatch their experiments. BN seems not to follow their theory, but the experiments use only BN. The authors should redesign their experiments and narrow down the scope.

Minor: "reduce the sharpness of the loss" may not reflect the true nature. A loss function is often fixed before training. Therefore, one may not reduce its sharpness when minimizing the loss. Instead, GD may move from a sharp point to another less sharp point. 

======
After reading the authors' response, I now see no more mismatch. I hence slightly increase my score.
questions: - Normalization methods often have their own trainable parameters. Those trainable params are important to DNNs' performance in practice. Why this paper does not take those params into consideration? Ignoring those params may not provide correct understanding of normalized DNNs.
- There is a high fluctuation after 47k training steps in Figure 2. It suggests that when moving out of minimizer manifold, the training loss will be extremely high. Does it indicate that the manifold of minimizers disconnect with the outside or there are some "hole" around that manifold?

limitations: The authors discuss some limitations of this work. I encourage the authors to state clearly the scope of this paper and redesign their experiments.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors presented analysis that Gradient Descent (GD) reduce spherical sharpness in training neural networks with normalization layers.
Their analysis is done by going through the trajectory of GD, from the stable regime into the Edge of Stability (EoS) regime.
In the EoS regime, the trajectory oscillates around the local minimizer manifold, and moves toward the a direction that reduces spherical sharpness.
In that regime the author shows, through inductive proof on two EoS condition, that a periodic behavior moves the gradient through until it minimizes the error.
The authors experimentally verified their findings on CIFAR-10 with batch-normalized, scale-invariant VGG-11 and ReNet-20, using Swish activation function. 

strengths_and_weaknesses: As more of a user of deep neural networks (both Computer Vision and NLU), I feel that the paper can be inaccessible to the general AI community. For one quite a large part of the proof is in the appendix, most reader will just take it to heart that things will work out. However, through that process, reader may lose grounded-ness of what should be kept tracked. 

I would say, making better use of the neural network results of the optimization at certain stages may help. That is to show what more is needed to be learned (which images went from misclassified to classified) and how batch normalization allows for that to happen.
questions: Typo, Grammar, clarification etc.:
Pg.3, ln. 91: generalizaiton —> generalization
Pg.3, ln. 95: with different with different —> with different
Pg.3, ln. 201: moves towards —> moves toward

limitations: The paper is theoretical in nature. Any negative societal impact would have to come from, at the current stage, a hard-to-foresee application of the research.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies the dynamics of sharpness for networks with normalization layers (invariant losses) that are trained using gradient descent with weight decay. The authors define spherical sharpness to account for the scale-invariance of the loss function. They first show theoretically that when neural network parameters get close to a minimizer, after some time, they would enter the "edge of stability" regime where both the learning rate and the network parameters would oscillate. Next, it is theoretically shown that at this edge of stability, the network parameters converge to a sharpness-reduction flow with a period-2 oscillation.  Finally, the theoretical results are showcased for a few experimental settings.
strengths_and_weaknesses: Strengths:

The problem (why normalization helps generalization) is of interest to the community. The theoretical contributions are rather complete and indeed provide a reasonable answer to the problem. The technical contribution with regards to showing the sharpness reduction property for gradient descent (without any stochastic noise) is solid. The practicality of the assumptions has also been considered (lines 209-214). There are even many results that are mentioned mainly in the appendix. 

Weaknesses:

The main point missing from the paper is the motivation behind the particular choice of the setting (BN+WD). Perhaps an empirical example would be nice to motivate this; first showing without BN there is no decrease in sharpness, but then after adding normalization layers there is. 

I would suggest a proofread for the paper to fix typos and rewrite some sentences to make them more clear. 

questions: 1- It is not clear why the combination of BN and WD is interesting to study? It seems a bit random.

2- Why are so many jumps in the train/test accuracy curves of Figure 2? What explains the drops in the training accuracy to below 100%? In typical experimental settings (training a VGG on CIFAR-10) such oscilations don't occur.

3- How are Figures 1 and 2 showing that "settings where normalization provably induces an implicit bias to persistently reduce sharpness in training, which we call the sharpness-reduction bias". It is not clear why the reduction in sharpness is linked to BN? 



limitations: The limitations of the work have been mentioned in the conclusion section.
review_title: null

Meta-review: metareview: 
This papers demonstrates how normalization with weight decay forces GD to converge to flater solutions.
Overall, the reviewers find that the paper does bring some new insights, perhaps with a limited impact but still valuable to the community. There were also a few concerns left after the discussion period, e.g.
- Dicussion of prior work is rather superficial. Some key works are missing or not discussed in appropriate depth
- Limited empirical evaluation

Given the discussion with the reviewers, the first concern seems easilly fixable in the camera-ready version. Regarding the second concern, the paper does however compensate with a solid technical contribution. I therefore recommend acceptance but I strongly encourage the authors to update their manuscript to address the above concerns. The writing quality is sometimes poor and should be improved in the final version.
confidence: Certain
Decision: Accept