Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper tackles the issue of explaining periodic behavior that can occur when jointly training neural networks using batch normalization and weight decay, and that causes training to alternate between phases of instability and stability.
The authors give a theoretically reasonable explanation of why this phenomenon occurs. They derive the notion of delta-jumps in parameter space that are responsible for training instabilities, and derive theoretical bounds based on parameter norm, effective gradient norm, and learning rate.
They investigate this mechanism empirically by training scale-free CNNs on the CIFAR-10 and CIFAR-100 dataset classification task, and also extend their analysis to more practical scenarios, showing that the periodic behavior can still be produced in less artificial settings.
main_review: The paper seems to provide a significant contribution in that it resolves the standing contradicting between the “equilibrium point vs. instability” view of training with batch normalization and weight decay. However, while the authors themselves point out that this effect can in principle also be observed in non-scale free networks, they make clear that it is of limited practical relevance due to the narrow parameter range in which it can occur.
The paper outlines its general ideas in a comprehensible way. However, some of the connections between theory and experiment could be made clearer, e.g. how the theoretically derived size of delta-jumps, convergence times etc. relates to the outcomes of their experiments. The rightmost plot in figure 3 was also a little confusing. The results are also generally shown without any statistics.

Some questions that could be clarified:

1. What happens for optimizers that use adaptive learning rates, such as Adam?
2. With what consistency does the loss achieve a lower minima after a phase of instability? Figure 6 makes it seems like this is the case, while in Figure 4 the result looks different.
3. “Making an SGD step in the direction of the loss gradient always increases the norm of scale-invariant parameters, while WD aims at decreasing the weight norm.” It is not directly apparent what this is based on. However, as this is the theoretical basis for the entire paper, it would be good to make the citation clearer and explain the mathematical reasoning in more detail.

Some minor comments:
- Using either epoch or SGD step (or making clearer how they relate to each other) consistently in the x-axis of figure 2+3
- Warm-up phases can't be seen too well from the plots
- coarse grain choices of learning rate. What happens in the range between 0.01 and 0.001?
- With which probability do the delta jumps that lead to instabilities actually occur? Do they happen with certainty if curve is in given interval, and does it always happen?
- The description of the ensemble method in 5) was a little confusing
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: In this paper, the authors look at scale-invariant networks trained with SGD using both batch normalization and weight decay. They study the contradiction between the equilibrium presumption and the instability presumption and claim that the training process converges to a consistent periodic behavior which includes some instabilities during training but does not lead to failure. They define the notion of delta-jumps for the dynamics of training and conjecture that the delta jumps are necessary for destabilization of the dynamics. Extensive empirical experiments are provided for supporting the claims.

main_review: The paper is written clearly in most parts and the periodic observation is interesting. I also appreciate the extensive experiments. My main concern about this paper is that the observation is limited to a very specific setting and this periodic behavior is usually not observed in practice. Therefore, I believe the problem is not well motivated in the first place and even for this special case, the theory to support the claims is not complete. I am not convinced by the authors’ conjecture that delta-jumps are necessary for training dynamics destabilization. I also have the following comments/questions:

1. It is noted in [1] that destabilization can happen due to the increase in sharpness of the network (the maximum eigenvalue of the training loss Hessian). Have you done any observations regarding this? 

2.  Lines 108 to 121: the explanation is insightful but very intuitive and without any figures or any exact expressions. It would be great if you could write the exact expressions and relations. For example, what do you exactly mean by effective learning rate? \frac{\eta}{\norm{x_t}^2} \proportional \sqrt{eta lambda}? or Maybe you could show the centrifugal and centripetal forces on a plot.

3. It would be great if you could discuss how limiting the assumption of bounded effective gradients is since this is the key element in generalizing proposition 3.

4. There is no Figure 7.  Please fix the number.



[1] https://arxiv.org/pdf/2103.00065.pdf

limitations_and_societal_impact: The authors have discussed the limitations of the work and the is no potential negative societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper shows that commonly used Batch Norm and Weight Decay on networks with frozen last layer exhibit periodic behavior in training. The authors provide theoretical guarantees and also study extensions of their results to more commonly used settings and show where their results break down.
main_review: Building upon some previous work in the theory of deep learning field, the authors show both theoretically and experimentally that for a network trained without last layer, Batch Norm and Weight Decay have opposing actions on control of the norm, leading to a periodic behavior in training. 

It is interesting that despite this rather atypical setting, the authors then expand their experiments and relax the assumptions one by one to reach the more typical regime of training. It is interesting to note that the periodicity of the behavior is broken by momentum, and not visible unless one uses a relatively large learning rate, which is something this reviewer has previously also noticed while training NNs with large learning rates on many occasions.

It would be interesting to see how these results coexist with the setting of all layers being trained, where it is expected that Neural Collapse (see V. Papyan et al. "Prevalence of neural collapse during the terminal phase of deep learning training"), and hence convergence, happens when we train deep nets with both Batch Norm and Weight Decay, see for example:
- A. Rangamani, "Dynamics and Neural Collapse in Deep Classifiers trained with the Square Loss" 
- T. Ergen and M. Pilanci,  "Revealing the structure of deep neural networks via convex duality"

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies an interesting contradiction about neural networks with Batch Normalization (BN) and weight decay (WD): some works show that BN and WD can cause instabilities in training, while other works show convergence to an equilibrium that stabilizes training metrics. After conducting very extensive experiments, the authors claim that the training process converges to a consistent periodic behavior, which regularly exhibits instabilities and restarts a new period of training.
main_review: I really appreciate the authors' work for their extensive experiments across different settings. The experiment setups are stated clearly and the figures are also very clear. I learned a lot from those experiments and I'm convinced that the periodic behavior could be more general than the instabilizing view and the equilibrium view from previous works. I would like to vote for acceptance.

For the theory part, the propositions in the current form are not very satisfactory. Note that the theory is expected to explain the periodic behavior, rather than just the first period. This means $\rho_0$ in Proposition 2 should be interpreted as the initial norm at the beginning of each period and thus cannot be set arbitrarily except for the first period. Thus, a better theorem should upper and lower bounds $\rho_0$ by analyzing the dynamics of previous periods and provide bounds for the time period without $\rho_0$.

Regarding why this work and Li et al. [17] produce different results, I suspect the most important factor could be the noise. When the noise is large and the learning rate is small, the SDE approximation in Li et al. [17] is valid and the convergence to equilibrium should occur. However, this paper turns off the data augmentation in most of the experiments, and thus the noise may not be sufficient.

I believe it could greatly benefit future research if the authors can compare their experiment setups with previous works. Section 6 has already done a very good job, but it could be more helpful if the authors can point out the most important factor for why previous experiments do not show this periodic behavior, specifically for each the works they are directly following (e.g., Li et al. [15], Li et al. [17]).

(The authors sometimes omit many second-order terms like  $O(\delta^2)$ or $O((\eta \lambda)^2)$ in the derivation. I would suggest the authors use $\approx$ and $\lesssim$  instead of $=$ and $\le$ when such approximation is used. E.g., Proposition 1.) 

-----------------

The rebuttal addressed most of my concerns. Although the theory part in the current form is not very strong, the empirical study in this paper is very nice and the period behavior is very interesting.  I would like to keep my score for acceptance and I look forward to seeing the camera-ready version of the theory part.
limitations_and_societal_impact: As stated in Section 6, the main limitation of this work is that periodic behavior mainly occurs on fully scale-invariant neural nets training for a sufficiently long time. Nevertheless, the training dynamics when $t \to \infty$ is a fundamental problem to study in the theory of deep learning, and the authors made a good contribution to this problem in the setting they adopted.
review_title: null

Meta-review: Three knowledgeable reviewers recommend this paper for acceptance and one more knowledgeable reviewer rates it as marginally below the acceptance threshold. After the discussion period, one reviewer concludes that although the theory part in the current form is not very strong, the empirical study in this paper is very nice and the periodic behaviour is very interesting. Another reviewer comments that s/he stands by her/his opinion that the discovery of the periodic behaviour is very interesting and warrants being shared with the NeurIPS community. Another reviewer indicates that although some of her/his main concerns still stand, s/he is convinced the extensive experiments are insightful and could be relevant in more practical cases and increased her/his score. In summary, after the rebuttal and discussion the general consensus among the reviewers is for accepting this paper. Hence I am recommending the paper for acceptance with the request that the authors take the reviewers’ comments and suggestions carefully into account for the preparation of the final manuscript. 
Decision: Accept (Poster)