Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper aims to understand what makes general architectures trainable, specifically what limits the maximum learning rate for deep learning models trained with SGDM, from the loss curvature aspective. They empirically studied the evolution of the loss sharpness as they vary the learning rate, warmup period, initialization, and architectural choices. They show maintaining a sufficiently small \lambda_1 is a necessary condition for successful training at a large learning rate. And different methods such as initialization, learning rate warmup, and normalization all enable higher learning rates to be used by reducing \lambda_1 during training. More specifically:
- Some initialization strategies for architectures without normalization actually operate by reducing curvature early in training, enabling training at larger learning rates.
- learning rate warmup gradually reduces \lambda_1 during training, which is a competitive baseline in comparison with better model initialization methods. 
- large loss curvature can result in poor scaling at large batch sizes and interventions designed to improve loss conditioning can improve the model’s ability to leverage data parallelism.
main_review: The paper presents the necessity of low curvature at the early stage of training for stable neural network training with large learning rate, some notable observations includes “the mid-training conditioning is determined by the learning rate, not on the initialization method used”; “Learning rate warmup can match the performance of recent advances in initialization research”. It seems to suggest that the weight initialization does not matter that much and warmup can be mitigated.

However, the paper is kind of hard to read as there are too many empirical observations presented but not quite well organized. For example, the most measurement \lamda_1 is is not clearly introduced and there is only a small footprint saying lambda_1 refers to maximum eigenvalue of the loss Hessian. The reason why 2/\eta is chosen is also not clear.

The curvature and sharpness: the authors use the term sharpness to denote the max eigenvalues of loss Hessian to denote the curvature or the the loss surface, however, there are other sharpness/smoothness measurements such as [1, 2], which were used to denote the generalization ability of the loss surface at the end of training. It is easy to get confused with those terms for different conditions. Are there any connections? If so, why not study them as well? It would be great to know how those metrics are correlated with the curvature. If not, it would be better to make clear the difference and not use the terms interchangeably.


The authors have shown that many techniques actually made low curvature. However, curvature itself can not reliably determine the trainability. As noted by the authors,  some models can be successfully trained even when they start out in the unstable region and measuring \lamda_1 at initialization is not always sufficient to predict whether or not the model will be easily trained. 

On the other hand, the DenseNet experiments says that the models with Batch Normalization actually start out with higher curvature than the non-BN variants, which suggests that curvature may not be able to explain the effects of BN quite well and no smoothness benefits are observed at initialization. I would hope there are more investigations here as BN is still a critical module and the authors observation contradicts previous belief.


[1] Keskar et al, On large-batch training for deep learning: Generalization gap and sharp minima. ICLR 2017
[1] Jiang et al, Fantastic Generalization Measures and Where to Find Them, ICLR 2020
summary_of_the_review: The authors present investigations regarding the connections between curvature and many initialization methods and techniques. However, no clear conclusions are made and some investigations are not deep enough.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper explores training instabilities of deep learning models by monitoring the max eigenvalue of the Hessian in i) different architectures; ii) different optimization tricks; iii) different stages of training. 

The paper is a mix of several empirical observations: 

(Section 4) One of the main observations is that the necessary (not sufficient) requirement for a "successful" training is to have *relatively* low max eigenvalue through the training process, where *relative* is roughly determined by the inverse of the learning rate. 

(Section 5) Second main observation is specific to the learning rate warmup strategy: the warmup strategy pushes max eigenvalue to the boundary below $2/\eta$, and with a better understanding of this optimization strategy, the authors show comparable performance in multiple datasets compared with other optimization tricks. 

(Section 6) Lastly, the authors explore batch size scaling with different scales of max eigenvalue, focusing on comparing BN (with small curvature only for that specific task), NoBN (larger curvature), NoBN 1.5xinit (highest curvature). The experiment shows for small curvature setting, both batch size and learning rate can scale quite well. 
main_review: Strengths:

* The paper includes quite a lot of experimental results and definitely provides valuable empirical observations (see them in the brief summary above). It gives an insightful and convincing understanding of the learning rate warmup strategy. It also points out several failure cases where good curvature does not guarantee convergence in training, which is equally valuable. 

Here are the questions for each section:

* Section 4.
I see slightly different max eigenvalue at initialization for same architecture with different learning rate -- is it fully due to the randomness in the initialization? Or do I miss something here?

* Section 5. 
Explain MetaLoss briefly. 

* Section 6. 
  1. Given Figure 3, why would the authors not include MetaInit as another baseline to test batch size scaling?
  2. In Figure 6, what learning rate is used for the Top left plot? 
 I'd guess it's the optimal (peak) learning rate per batch size, but it's not fully clear. If not, please clarify and better help me understand how the learning rate is chosen; If yes, it looks like BN with batch size 128, 256, 512 has a log-linear improvement from the top left plot, but not very log-linear if looking at the bottom left plot: improvement from 128 to 256 is much larger than the improvement from 256 to 512. Similar results for NoBN 1.5x init. So I may be missing something here. 
summary_of_the_review: Overall, this is a good empirical paper with lots of experimental results and good insights. The learning rate warmup part is the most insightful among all.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper performs extensive empirical investigation into factors that play important role in making neural network trainable. One of the key quantity that the authors find to give significant insights is the largest eigenvalue of the Hessian matrix. They show that models that train successfully tend to have learning rates close to the well known critical bound for GD - 2 / \lambda. Further investigation to various architectural choices - such as normalizations and initialization techniques indicate that the value of \lambda behave somewhat consistently after the initial training period for well training models. In addition, they demonstrate that learning rate wramup can provide a significant improvement by pushing parameters in regions with lower values of \lambda. 
main_review: The paper seems well written and manages I think to condense a significant amount of empirical work in the short limit we have. I think the analysis that has been done across multiple architectures and models provides a good basis for the authors claims in the text. 

Strengths:
* The paper contains a very solid set of large scale and in-depth evaluation, which is something well needed for this kind of empirical work.
* The different analysis done well presents the vast amount of results and the key takeaways from the experiments

Weaknesses: 
 * Since we do observe models that diverge despite beginning in stable regions (Fig. 3 D) it seems that there the initial warnup has a significant effect of where it takes the model beyond its initialization, which is not too well discussed.
 * In the Dense Net experiment in Table 1, there is no reported results for including warmup



summary_of_the_review: Strong empirical paper which well presents many experiments and observations of the interplay of the maximum eigenvalue of the Hessian and learning dynamics.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a comprehensive set of experiments showing how the maximal learning rate to train modern architectures depends on various algorithmic choices.
main_review: Comments :
- Results are well presented and yield a rather coherent picture. In particular, I like the intuition that warmup brings to a region of smaller curvature, which fits in well with the catapult mechanism. I also appreciate the batch size scaling experiments, which go against the idea that the only quantity that matters is the ratio of learning rate / batch size. 
- The observation that lambda_1 follows 2/lr in presence of warmup could be better connected to Cohen et al. with a sentence such as “Cohen et al. showed that lambda_1 increases up to 2/lr then plateaus at the edge of stability, in the constant lr setup : we show that this holds even in presence of scheduling”
- The figures are extremely long to process for the various PDF readers I tried. This is likely due to an excessive number of datapoints in the logscale figures (happened to me in the past) : consider subsampling the data at large values.

Questions :
- Fig. 3 : Could one try to understand why some of the models diverge even though eta<2/lambda_1 ?
- Fig. 3 : Multiplying the init variance by 1.5 seems to increase the sharpness by a factor between 5 and 100. Why such a variability ? Can one predict this factor from the number of layers in the model ? 
- If increasing init variance increases sharpness, couldn’t one simply reduce the sharpness by reducing init variance, hence enabling larger learning rates ? This could be connected to the question of lazy vs feature learning regimes
summary_of_the_review: Although the results presented are not particularly groundbreaking, the paper is rather well written and easy to follow, and gives some potentially useful insights on an important topic : learning rate scheduling. Therefore, I think this paper could be of interest to the community.
review_title: null

Meta-review: The paper provides a thorough study of the evolution of Hessian depending on a wide variety of aspects such as initialization, architectural choices, and common training heuristics. The paper makes a number of interesting observations. Some of them are not really new but overall, the experimental evaluation of the paper makes it a valuable resource for the community.

The reviewers are overall quite positive. One reviewer notes that more investigation of the behavior of batch-normalization is required. I encourage the author to address this concern in the final manuscript. There is a lot of recent work on batch-normalization that might be worth discussing, e.g.:
Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs
Jonathan Frankle, David J. Schwab, Ari S. Morcos

Batch Normalization Provably Avoids Rank Collapse for Randomly Initialised Deep Networks
Hadi Daneshmand, Jonas Kohler, Francis Bach, Thomas Hofmann, Aurelien Lucchi

A Quantitative Analysis of the Effect of Batch Normalization on Gradient Descent
Yongqiang Cai, Qianxiao Li, Zuowei Shen
Decision: Accept (Poster)