Venue: ICLR 2017 workshop

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This work presents a series of observations gleaned from training a ResNet at different learning rates and schedules. While in general this sort of empirical analysis is a good thing, the paper does not put forward any novel explanation or theory based on these observations. Overall the paper is reasonably well written but lacks clear motivation or take-aways. The techniques in this paper are not novel but the analysis is interesting. I would recommend rejection at this time but encourage the authors to see if they can further explore possible insights their experiments may have uncovered. 
review_title: Official Review

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper discusses several interesting phenomena regarding the training and testing error curves over the course of training deep network models on image classification tasks. Among the findings are that test error performance can be nonmonotonic with certain learning rates, and imposing a cyclic alternation between low and high learning rates can speed learning. 

-While these results may point to something deeper, additional control experiments would greatly strengthen the paper. The finding that a cyclic learning schedule can speed learning would be potentially of practical interest, but the experiments compare just one particular cyclic scheme to one particular fixed learning rate. Does a carefully optimized fixed learning rate match the cyclic performance? Is a cycle really necessary, or can the learning rate just decrease monotonically over the course of learning?

-There may be simple standard explanations of these phenomena. The test error spikes up on each cycle as the learning rate crosses some threshold, which seems a straightforward case of SGD becoming unstable and diverging when the learning rate is made too high. After taking a giant bad step, higher learning rates can make progress because the network is terrible and fine adjustments are not necessary. More is necessary to back up the claim that these results provide insight into the "loss function topology."

+The finding of faster convergence with cyclic learning rate schedules, if it remains faster than the optimal fixed or monotonically decreasing schedule, would be very interesting and merits more investigation.

+The suggestion of interpolating many models to yield higher generalization performance is also a potentially interesting direction. 
review_title: Interesting phenomena, but more experiments needed to rule out less interesting explanations

Decision: Reject