Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper studies the effect of the learning rate's magnitude when training neural networks. I believe this to be an extremely relevant problem since large learning rates are widely adopted in practice due to the their positive impact on the model's generalization, even though we don't understand the reason behind this.

The main contributions come from the analysis of a 2-layer linear network trained on a single data point, which shows the existence of three regimes that are governed by the learning rate's magnitude and the local curvature at initialization. Unlike prior work that also identifies different regimes governed by the learning rate, this submission does not point at noise as the main responsible for the regimes.

Presented experiments are interesting and clearly show the different regimes and how they are connected to the learning rate, but do not seem to provide significant new insights compared to previous work. I believe the plots showing how the curvature behaves as the learning rate varies are novel to the best of my knowledge, but are not surprising given the previous discussions on how the learning rate affects the solution's sharpness.

While the theoretical result does provide a precise characterization of the phenomena, it is restricted to a very constrained setting (2-layer linear net trained on one data point), making it hard to evaluate how the result translates to more complex settings (although the authors do a reasonable job at exploring this via experimental analysis).

I also find the discussion on the presented theory to be somewhat lacking. For example, while the paper seems to downplay the importance of noise in characterizing the three regimes (compared to prior work), the curvature inevitably changes if noise is added to the sample gradients, e.g. adding isotropic noise will decrease \eta_crit and shift the phase transitions, which is also expected to occur if as we decrease the batch size when doing mini-batch SGD. The fact that different batch sizes have been adopted across experiments (even for the same dataset/model) makes the experimental results harder to evaluate when compared to the developed theory. Except for a footnote on page 2, there is no discussion on how the initialization scale affects the phase transitions, which would be useful to better understand how the theory relates to prior work that focuses on the init scale instead of noise or width.

Overall I believe the problem to be very relevant and the approach to be promising, but the theoretical results are overly limited and the contributions are below what I'd expected to suggest acceptance. Since extending the analysis to non-linear networks would seem to be a considerable technical leap, the authors could instead extend it to a less limited training setting where gradient variance is non-zero, in which case \eta_crit would have a dependence on the batch size assuming mini-batch SGD updates.

----

I have read the response and am keeping my score. I agree that the simplicity of the results/model is valuable, but additional theoretical results (even extensions to Theorem 1, with more involved but stronger claims) would greatly improve the paper and make its contributions closer to what is expected of a ICLR submission. Extending the result to consider noise should be straightforward and yield a reasonably simple claim, which can be further verified empirically by adding synthetic noise to the gradients, adding label noise, and/or adopting extremely small batch sizes. As it stands now, the submission is still lacking in terms of contributions.
review_title: Valuable insights on a highly relevant problem, contributions are somewhat limited

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper is a detailed account of how large of a learning rate a given mode can take when it is trained by constant step size SGD. Many papers investigated the effect of learning rate (and batch size and width etc…) to the final accuracy. In general, the findings indicate increased accuracy up to a certain threshold (where the algorithm doesn’t converge any more). Such findings are abundant in the literature. However, the present paper investigates the path that SGD takes to get to find those ‘good’ performing points in the weight space. Two phases emerge: monotonic decay in loss vs catapult regime and the latter appears to perform the best. 
The paper has interesting findings but also some shortcomings, in more detail:
- Similar curve as in Figure 14 of https://arxiv.org/abs/1905.03776 as Fig 1 of the present paper, in that sense the findings of large lr better are certainly not novel, but the authors are well aware of this
- The paper defines the curvature as the max eigenvalue of the Fisher information matrix. The large learning rate values are determined according to that value. Fig 2 c and f show the cross over of the regimes. What would make the cross over sharper? Increasing width? Larger batch size? 
- The large increase in loss value is huge! It is surprising that training doesn’t just diverge from there. What controls the growth and the subsequent decay? How consistent is it? Would the result hold in all random seeds or what fraction would still converge? 
- What’s the max tr loss for the wide res net? (Why not consistent plots for fully connected and resnets in fig2?) (Also, for clarity, Fig 2 label of B would be Max tr loss)
- Fig 3 would be clearer if the test error axis of (a,b) would be aligned, same for (c,d). It appears as if decay actually makes the search for the correct large lr rate redundant. (Perhaps at the expense of searching for the right decay rate). In general, what can we do to maximize the range of hyperparameters that works for the model (to minimize search)? The paper proposes a heuristic based on the initial local curvature yet the findings for practice seem somewhat preliminary. 
- The comparison between lazy training and how large learning rate deviates from lazy training is a very interesting and active area of research. I think the paper would be more complete if the experiments would verify it in a more robust way. Measuring the deviation of the activations or the movement of the kernel would be helpful. Here is one of the references with clear observables to measure in the experimental framework https://arxiv.org/abs/1906.08034 

The characteristic difference in the early time dynamics is claimed to be related to the performance gains. There is a growing body of evidence that points to the role of the initial dynamics of SGD when it comes to identifying the final performance. The paper is further evidence in that direction. Therefore I think it’ll be useful for the ICLR community. Overall, the paper has very interesting bits and pieces but it fails to come together as a coherent whole to provide a consistent story.
review_title: Limits of large learning rate training at fixed learning rate SGD

final_score: rating: 3: Clear rejection
review_content: This paper analyzes the effect of choosing a large step-size on the generalization of deep networks. They suggest that starting with a large learning rate, the loss initially increases before converging to a flatter minimum with improved generalization (catapult effect). When the learning rate is above a certain threshold, the authors observed this catapult effect along with a decrease in the curvature of the landscape. These observations were empirically demonstrated through various experiments and analytically studied for a two-layer linear network.

Pros:
The paper tackles an interesting problem in a vibrant field of research.

Several experiments that demonstrate the authors claims were presented in the paper. These results can be further used to propose algorithms designed to converge to flatter solutions. 

The related material is referenced and well-discussed in the paper.

Cons:
This paper lacks theoretical evidence for supporting the raised claims. The model studied in Theorem 1 is very simple. Moreover, the effect of choosing the step-size and batch-size on the sharpness of the computed solution has been observed in the literature. For instance, Keskar et al. (2017) justified the improved generalization achieved when using small batch sizes by the ability of such methods to converge to flat minima. The choice of the batch-size was formally related to the choice of the step-size in the work of Patel (2017) which provides a learning-rate lower bound threshold for the divergence of batch SGD. This threshold is also a function of the curvature. The higher the curvature around a critical point, the lower the threshold required for divergence. Hence choosing a large step-size tend to escape sharp minima and potentially converge to flatter minimizers. The last paper was not cited in the submitted manuscript.

The paper does not provide any intuition on how to compute the thresholds $\mu_{\mbox{crit}}$ and $\mu_{\max}$, and how is this related to the choice of non-linearity, structure of the network, …

Comments:
1.	For the gradient descent update rule in equation (1): why do we have $f_t$? 

Minor Comments:
1.	What is meant by ``compute budget’’?
2.	In the definition of $\Theta$ in Page 4: I think it should by $\sum_{\alpha = 1}^m$ instead of $\sum_{\mu=1}^p$.
3.	In Figure S1, the images are small and not very clear.
4.	Expression (S12) is missing a term.
5.	S(16), what is $u_{ia}$.
6.	Figure S10, the colors in the label do not match the plots.  


review_title: The Large Learning Rate Phase of Deep Learning

Meta-review: I agree with the reviewers that said that this paper has valuable insights. However, all reviewers ultimately recommended rejection. I think the main reason was that the reviewers did not feel these insights don't accumulate together to a message that would justify a paper. I hope the authors can address these concerns and resubmit. There were additional concerns, like the fact a very simplistic toy model is being used, but I agree with the authors that it makes sense to first explore such phenomena in the simplest model that produces them.
Decision: Reject