Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This work studies the problem of how to define learning rate schedules when training deep models so that the models better generalize.  To this end, the paper proposes and evaluates a learning rate schedule that consists of two stages (knee schedule).  A first stage of exploration adoptes a high learning rate. This initial stage is followed by a second stage where the learning rate decreases in a linear way. Extensive experimental results, both in text and image data, show that the proposed scheme allows one to train faster or to obtain better results with a fixed computational budget. The proposed learning schedule leads to SOTA results on IWSLT’14 (DE-EN) and WMT’14 (DE-EN) datasets.

The work relates the good performance of the proposed knee schedule in the hypothesis that wide minima have a lower density (are less common), therefore, a large learning rate  is required initially (and for some time) to avoid shallow minima. The second refinement stage with the learning rate declining linearly allows one to delve into the minimum found in the exploration stage. Recent works indicate that in fact wide minima are the ones that lead the models to generalize better and this is in agreement with the experimental results of the article.

The main contribution of the article is an exhaustive experimental evaluation in different applications where they analyze different schedules and show how the proposed schedule leads to superior performance. The paper raises a working hypothesis compatible with the success of the LR schedule and in that sense generates an interesting line to continue research. 

Some questions:

1. From reading the article it is not clear to me how it is justified to keep the learning rate high even when the loss stagnates. I understand this is based on conducting experiments and then measuring the power of generalization. But it is interesting that from the training point of view it would seem that after training stagnates the network is not learning but pivoting from one side to the other. What do you think can be a good hypothesis of what is happening during training at this stage? I would like if possible that this point is better discussed. And it would also be useful if the work better discussed why the working hypothesis is the most reasonable explanation.

2. Table 1 shows that reducing the learning rate after the exploration stage helps to better minimize the loss. However, this does not translate into a network that generalizes better. Is it reasonable to hypothesize that during this second period the network overfitted to the behavior around this minimum? Does this phenomenon occur in other experiments? If so, why is the second refinement stage needed?

3. Warmup. Some optimizers use a warm up step where the learning rate starts to rise smoothly. It would be interesting to better discuss how this stage is linked to the exploration stage. How long does the warmup stage need to be? If warmup + exploration + decay is put together, at the end it is a curve with a certain resemblance to a cosine. 

Additionally, if the information is available it would be useful to have the standard deviations of the average values ​​calculated in Table 6.



review_title: New learning rate schedule for training deep models leading to better generalization.

final_score: rating: 5: Marginally below acceptance threshold
review_content: Learning rate schedule plays an important role in DL, which has a large influence over the final performance. Though there have been lots of schedules, achieving SOTA performance still requires careful hand-tuned schedule that may be case by case. Compared with previous learning rate schedules, authors first conjectured that the number of wide minima is significantly lower than the number of sharp minima, and then  proposed to use a large learning rate at the initialization phase for sufficient exploration to achieve a wide minima, which may achieve better generalization performance. Extensive experiments validate the proposed learning rate schedule. 

The observation of this paper looks interesting, and authors have conducted lots of experiments to validate the effects of proposed learning rate schedules. However, the novelty of this paper seems limited. First, authors conjecture that the number of wide minima is significantly lower than the number of sharp minima, but it lacks a thorough investigation of this conjecture, either from related empirical study or theoretical understanding. Second, for the proposed learning rate schedule, it seems not very clear how to set the duration of exploration epochs appropriately across different tasks, as it is still a hand-tuned hyper-parameter. For fixed 50% explore, there is not much difference in terms of the performance compared with previous schedule such as Cosine Decay or linear decay in Table 6.    

Overall, I tend to a weak reject and it would much better if authors could go deeper behind the observation/conjecture.
review_title: Interesting Observations But Limited Novelty

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:
This paper did an empirical study on the learning rate (LR) schedule for deep neural networks (DNNs) training. The authors argue that the density of wide minima is lower than sharp minima and then show that this makes keeping high LR necessary. Finally, they propose a new LR schedule that maintains high LR enough long. 

Pros:
-	The problem this paper studies is import for DNNs training. The proposed LR schedule is simple and has the potential to be used widely.
-	The authors conduct extensive empirical tests to support their claim and the experimental design is reasonable.

Cons:
-	I’m not fully convinced by the hypothesis that wide minima have lower density. The empirical results can be explained by other hypotheses as well. For example, it is also possible that wide minima are farther away from the initialization. I think the authors need to either provide theoretical analysis or come up with new experiments to further verify this hypothesis.
-	The proposed LR schedule does not seem necessary. One could easily achieve the same purpose by existing LR schedules, e.g. use a step decay LR schedule. 
-	The novelty is low. The main novelty of the paper is the above hypothesis, but it is not supported enough. The proposed LR schedule is a slightly modified version of the existing LR schedule.  Thus the contribution of this paper seems incremental. 


review_title: Review

final_score: rating: 3: Clear rejection
review_content: Overview:
Overall I believe the comparisons to baselines seem too problematic to understand the value of the proposed method. Regarding the reduced training budget results (“Knee schedule can achieve the same accuracy as the baseline with a much reduced training budget”), were the baseline schedules also retuned for the reduced training budget? If not, this seems like an unfair advantage to the proposed method. For example, in the MLPerf competition (https://arxiv.org/abs/1910.01500) for ImageNet there have been schedules (consisting of a linear warmup followed by quadratic decay) that have been tuned to reach 75.9% in only 64 epochs, even at massive batch sizes, implying that the baseline schedule in Table 3 could likely do much better than what is reported if it was retuned with the same number of trials as the proposed method, or if a more competitive baseline schedule was used. Some of the results seem misleading as well; in the training curve figures 6, 7, 8, 9, 10 in the appendix, it seems odd that the proposed method only catches up to the (untuned) baselines towards the very end of training, and that this was not mentioned in the main text. For example:
-on CIFAR10 the baseline beats the proposed method until the final *5 out of 200* epochs of training
-on BERT_LARGE pretraining it is unclear from the plots when the proposed method beats the baseline as the curves are so similar
-on WMT’14 (EN-DE) the baseline beats the proposed method until the final 54 out of 70 epochs of training
-on IWSLT’14 (DE-EN) the baseline and proposed method cross each other a few times, the final time being at epoch 41 of 50
-on IWSLT’14 (DE-EN) with the MAT network, the baseline and proposed method cross each other a few times, the final time being at epoch 330 of 400
While it is not invalid for a proposed method to overtake a baseline towards the end of training, these results indicate that perhaps if the baselines were retuned, they could maintain their better performance for the last few epochs of training. Using the same initial LR for the proposed and baseline methods is useful, however it is insufficient to demonstrate that the proposed method could still perform well under different initial conditions. I have additional concerns about the significance of the proposed method over the baselines which I describe below.

Regarding comparing to the sharpness of the baseline LR schedules: “With fewer explore epochs, a large learning rate might still get lucky occasionally in finding a wide minima but invariably finds only a narrower minima due to their higher density.“ it would help to show curvature metrics at frequent intervals during training to confirm this hypothesis, and to also show these for the other learning rate schedules compared to, so that you can demonstrate that the proposed schedule achieves something the baselines cannot. The sharpness values in Figure 2 are interesting, but I am unable to determine how impressive they are given that they are not compared to sharpness values for any other schedules, so I don’t know what the baseline numbers should be.

Finally, it is unclear that the proposed method is novel enough to warrant a standalone paper, without more rigorous theoretical explanations to support the claimed reasons behind its performance.

Pros:
-It is useful to note that definitions of curvature can be problematic, which the authors do discuss (citing https://arxiv.org/abs/1703.04933)
-The breadth of experiments is genuinely impressive, but unfortunately would be more impressive if the breadth was smaller and more careful tuning was done for the proposed method and baselines

Concerns:
-In Appendix C when describing your curvature metric, you say “The maximization problem is solved by applying 1000 iterations of projected gradient ascent”. How was 1000 chosen? Did the sharpness metric stop changing if more steps were used?
-What are the stddevs of the results in Tables 6, 7, 18, 19, 20, 21, 26? The proposed results seem very close to the (untuned) baselines, and so it would be useful to understand how statistically significant they are.
-Toy problems can be extremely useful to empirically demonstrate this wide vs sharp minima selection phenomena would be useful, such as in Wu et al. 2018 (https://papers.nips.cc/paper/8049-how-sgd-selects-the-global-minima-in-over-parameterized-learning-a-dynamical-stability-perspective), or the noisy quadratic model in https://arxiv.org/abs/1907.04164
-The curves in Figure 7 seem extremely similar, it would help to plot the loss on a log scale
-In Section 4.1, “In all cases we use the model checkpoint with least loss on the validation set for computing BLEU scores on the test set.”, is early stopping used in all experiments? If not, why?
-”Thus, in the presence of several flatter minimas, GD with lower learning rates does not find them, leading to the conjecture that density of sharper minima is perhaps larger than density of wider minima.” it is unclear to me how their previous results support this hypothesis; couldn’t one retune the learning rate of SGD to find sharper/flatter minima, independently of how many sharp/flat minima exist?

Writing:
-The experiment details in the intro could be moved to later in the paper (it seems to be repeated in section 2)
-Overall the paper length seems like it could be drastically reduced by removing repeated statements
-Figures 6, 7, 8, 9 would be much clearer to read if it was a single plot per row, possibly on a log scale on the vertical axis when applicable
-For consistency, it would be useful to have “Baseline (short budget)” also be reported in Table 5

Prior work:
There are many previous works on explaining the benefits of large learning rates, the most relevant being https://arxiv.org/abs/1907.04595 which seems to make the same case as this paper, but is not cited. Additionally, https://arxiv.org/abs/2003.02218 has more theoretically explanations for this, using the Neural Tangent Kernel literature, and the authors could likely derive similar explanations. In fact, they use a similar schedule as the proposed method, but do not give it a name: “The network is trained with different initial learning rates, followed by a decay at a fixed physical time t · η to the same final learning rate. This schedule is introduced in order to ensure that all experiments have the same level of SGD noise toward the end of training.” Finally, there are other works that describe how low curvature directions of the loss landscape will be learned first, benefiting from a higher LR, followed by high curvature/high noise directions, which benefits from a smaller LR, described in https://arxiv.org/abs/1907.04164. I believe that a more formal explanation and analysis of the claims on solution curvature density should be provided.

Additional feedback, comments, suggestions for improvement and questions for the authors:
I believe that fairer experimental setup would be similar to the following:
-pick several competitive LR schedules for each problem (not just the “standard” ones)
-identify a similar number of hyperparamters for each*, such as number of warmup steps, decay values, decay curve shapes, etc.
-retune each schedule and the proposed method for the same number of trials, using similarly sized search spaces for each (ideally one would also retune the initial/final learning rates, momentum, and other hyperparameters for each, but this may be too expensive)
-select the best performing hyperparameter setting for each schedule, and rerun it over multiple seeds to check for stability

*it can be problematic to make comparisons across methods with different numbers of hyperparameters even with the same tuning budget, because it is impossible to construct the same volume hyperparameter spaces with different numbers of hyperparameters, see https://arxiv.org/abs/2007.01547 for a more thorough treatment
review_title: Unclear novelty and problematic baseline comparisons

Meta-review: The reviewers are concerned about the novelty of the proposed learning rate schedule, the rigor of the empirical validation, and the relationship between the results and the discussion of sharp vs. local minima. I invite the authors to incorporate reviewers' comments and resubmit to other ML venues.
Decision: Reject