Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: In this paper, the authors analyze training of residual networks using large cyclic learning rates (CLR). The authors demonstrate (a) fast convergence with cyclic learning rates and (b) evidence of large learning rates acting as regularization which improves performance on test sets – this is called “super-convergence”. However, both these effects are only shown on a specific dataset, architecture, learning algorithm and hyper parameter setting. 


Some specific comments by sections:

2. Related Work: This section loosely mentions other related works on SGD, topology of loss function and adaptive learning rates. The authors mention Loshchilov & Hutter in next section but do not compare it to their work. The authors do not discuss a somewhat contradictory claim from NIPS 2017 (as pointed out in the public comment): http://papers.nips.cc/paper/6770-train-longer-generalize-better-closing-the-generalization-gap-in-large-batch-training-of-neural-networks.pdf

3. Super-convergence: This is a well explained section where the authors describe the LR range test and how it can be used to understand potential for super-convergence for any architecture. The authors also provide sufficient intuition for super-convergence. Since CLRs were already proposed by Smith (2015), the originality of this work would be specifically tied to their application to residual units. It would be interesting to see a qualitative analysis on how the residual error is impacting super-convergence.

4. Regularization: While Fig 4 demonstrates the regularization property, the reference to Fig 1a with better test error compared to typical training methods could simply be a result of slower convergence of typical training methods. 
5. Optimal LRs: Fig.5b shows results for 1000 iterations whereas the text says 10000 (seems like a typo in scaling the plot). Figs 1 and 5 illustrate only one cycle (one increase and one decrease) of CLR. It would be interesting to see cases where more than one cycle is required and to see what happens when the LR increases the second time.

6. Experiments: This is a strong section where the authors show extensive reproducible experimentation to identify settings under which super-convergence works or does not work. However, the fact that the results only applies to CIFAR-10 dataset and could not be observed for ImageNet or other architectures is disappointing and heavily takes away from the significance of this work. 

Overall, the work is presented as a positive result in very specific conditions but it seems more like a negative result. It would be more appealing if the paper is presented as a negative result and strengthened by additional experimentation and theoretical backing.
review_title: Large cyclic learning rates for fast convergence, works in very narrow conditions

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper discusses a phenomenon where neural network training in very specific settings can profit much from a schedule including large learning rates. Unfortunately, this paper feels to be hastily written and can only be read when accompanied with several references as key parts (CLR) are not described and thus the work can not be reproduced from the paper.

The main claim of the author hinges of the fact that in some learning problems the surface of the objective function can be very flat near the optimum. In this setting, a typical schedule with a decreasing learning rate would be a bad choice as the change of curvature must be corrected as well. However, this is not a general problem in neural network training and might not be generalizable to other datasets or architectures as the authors acknowledge.

In the end, the actual gain of this paper is only in the form of a hypothesis but there is only very little enlightenment, especially as the only slightly theoretical contribution in section 5 does not predict the observed behavior. 

Personally i would not use the term "convergence" in this setting at all as the runs are very short and thus we might not be close to any region of convergence. Most of the plots shown are actually not converged and convergence in test accuracy is not the same as convergence in training loss, which is not shown at all. The results of smaller test error with larger learning rates on small training sets might therefore just be the inability of the optimizer to get closer to the optimum as steps are too long to decrease the expected loss, thus having a similar effect as early stopping.

Pros:
- Many experiments which try to study the effect
Cons:
-The described phenomenon seems to depend strongly on the problem surface and might never 
be encountered on any problem aside of Cifar-10
- Only single runs are shown, considering the noise on those the results might not be reproducible.
-Experiments are not described in detail
-Experiment design feels "ad-hoc" and unstructured
-The role and value of the many LR-plots remains unclear to me.

Form:
- The paper does not maker clear how the exact schedules work. The terms are introduced but the paper misses the most basic formulas
- Figures are not properly described, e.g. axes in Figures 3 a) and b)
- Explicit references to code are made which require familiarity with the used framework(if at all published).  
review_title: Interesting Phenomenon, but no deep insights

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper discusses the phenomenon of a fast convergence rate for training resnet with cyclical learning rates under a few particular setting. It tries to provide an explanation for the phenomenon and a procedure to test when it happens. However, I don't find the paper of high significance or the proposed method solid for publication at ICLR.

The paper is based on the cyclical learning rates proposed by Smith (2015, 2017). I don't understand what is offered beyond the original papers. The "super-convergence" occurs under special settings of hyper-parameters for resnet only and therefore I am concerned if it is of general interest for deep learning models. Also, the authors do not give a conclusive analysis under what condition it may happen.

The explanation of the cause of "super-convergence" from the perspective of  transversing the loss function topology in section 3 is rather illustrative at the best without convincing support of arguments. I feel most content of this paper (section 3, 4, 5) is observational results, and there is lack of solid analysis or discussion behind these observations.
review_title: This paper shows an observation of “super-convergence” when training resnet with cyclical learning rates but does not provide conclusive analysis or experiment results.

Meta-review: The paper reports unusally rapid convergence of the ResNet-56 model on CIFAR-10 when a single cycle of a cyclic learning rate schedule is used.  The effect is analyzed from several different perspectives. However, the reviewers were not convinced because the effect is only observed for one task, so they question the significance of the result. There was significant discussion of the paper by the reviewers and area chair before this decision was reached.

Pros:
+ Paper illustrates a "super-convergence" phenomenon in which training of a ResNet-56 reaches an accuracy of 92.4% on CIFAR-10 in 10,000 iterations using a single cycle of a cyclic learning rate schedule, while a more standard piecewise-constant schedule reaches 91.2% accuracy in 80,000 iterations.
+ There was partial, independent replication of the results on other tasks reported on OpenReview.

Cons:
- In the paper, the effect is shown for only one architecture and one task.
- In the paper, the effect is shown for only a single run.
- There are no error bars to indicate which differences are significant.

Decision: Reject