Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: This paper proposes an algorithm for automatically tuning the learning rate of SGD while training deep neural networks. The proposed learning rate tuning algorithm is a finite state machine and consists of two phases: the first phase finds the largest learning rate that the network can begin training with for p = 10 epochs; the second phase is an optimistic binary exploration phase which increases or decreases the learning rate depending upon whether the loss is NaN, increasing or decreasing. Empirical results are shown on a few standard neural networks for image classification on CIFAR-10/100 datasets and for adversarial training on the CIFAR-10 dataset.

I recommend rejecting this paper for the following reasons: (i) the algorithm developed here is extremely heuristic, no insight, theoretical or empirical, is provided as to why this could be a general algorithm, (ii) a major claim in the paper is that the automatic learning rate tuning does not have any hyper-parameters but the actual algorithm does have parameters such as patience and successive doubling of the learning rate although they are tuned adaptively using ad-hoc heuristics, (iii) the convergence analysis is not at all rigorous, in particular the optimal oracle for SGD  may not exist, and (iv) the baseline algorithms are not tuned and the minor improvements of the proposed algorithm over them is therefore not significant.

Some questions that I would like the authors to answer:

1. While the first phase of the algorithm seems a reasonable thing to do, the second phase is full of heuristics which I am not sure will work well for all problems. For instance, I do not see why the algorithm performs trains for p epochs twice even if the loss increased after the first stage, or why the learning rate should be increased if the loss decreased after the second stage.
2. Section 4, bullet 3/4 in the definition are problematic: the loss in SGD is not monotonically decreasing with respect to time. What does divergence of training mean here? What does “Any SGD algorithm” mean? Do you instead mean any first-order stochastic gradient-based algorithm?
3. If you imagine a double well potential with one wide minimum and one sharp minimum, both at the same training loss, if OPT starts in the sharp valley, it will not be able to go to the wide valley without the training loss increasing.
4. Have you tried this algorithm on other problems which are sensitive to the values of learning rate, e.g., training optical flow or segmentation networks?
5.  The wordy and heuristic argument in Section 4.1 rests on statements like “AALR and OPT arrive at roughly the same location after so and so epochs and hence reaches similar generalization performance”. This cannot happen in a non-convex landscape, the trajectory of SGD starting from the same initial condition can be very different across two independent runs. Therefore, I also don’t see why the latter half of the statement about generalization should be true.
6. Can you make the development in Section 4 rigorous?
7. Why are some runs for SGDR stuck at 10% accuracy in Table 1-2?
8.  FGSM is a very weak attack for measuring adversarial accuracy. Can you show results with a better attack, say a few steps of PGD?


Some suggestions to improve the paper:

1. A simple experiment to check the automatic tuning would be to increase the batch-size of the same network while maintaining the ratio of batch-size and learning constant (see https://arxiv.org/pdf/1706.02677.pdf, https://arxiv.org/abs/1710.11029, among others). It would be interesting to see whether the auto-tuner finds a learning rate that corresponds to stable learning without degradation in the generalization performance.
review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: The paper considers the problem of automated adaptation of learning rate during (deep) neural network training. The use cases described are standard and adversarial training for image classification. Given the wide use of DNNs in computer vision (and other areas), learning rate tuning is clearly an important problem and is being actively researched.

The proposed learning rate adaptation procedure consists of a straightforward combination of learning rate halving/doubling and model checkpointing. Experimental results from implementing the adaptive learning rule for standard and adversarial training on CIFAR are provided. Multiple architectures are tested in each setting. The paper claims a primary advantage of the proposed learning rule to be that it requires no tuning as opposed to other rules such as SGD, Adam.

My decision is to reject the paper due to methodological issues with the experiments and lack of evidence wrt/ dataset variety. The paper should be considered a work-in-progress that may have potential in a more focused setting, e.g., adversarial training as described in the paper.

***

The major claim of the proposed algorithm not requiring any manual tuning is technically true but misleading. The algorithm does have parameters (SGD momentum, batch size, initial learning rate, patience) with values that were set somehow. In fact, a major methodological issue with the experiments is that the reader does not know if the datasets were used to both set these values and to assess performance, i.e., there are no obvious "held-out" datasets. Also, there is no rigorous or even informal justification of the settings. It could be that the paper is arguing that the specific values will result in competitive, if not better, performance than baselines across a variety of datasets - unfortunately, only two datasets are utilized in the experiments, and one, CIFAR10, is not considered challenging. This leads to the second issue with the paper: the experimental validation is not extensive wrt/ datasets which is significant given that the form of the evidence for the proposed method is almost entirely empirical.

Additionally, I don't agree that competitor algorithms should not be tuned b/c the proposed method does not require tuning. Even if the proposed method does not require tuning (as stated previously, I don't believe this to be accurate), that does not imply a fair comparison precludes tuning competitors via, e.g., cross validation. The only relevant quantities are final test-set performance and total training time/resources required. 

The well-known interdependence between learning rate and batchsize as noted in e.g., Hoffer et al. (2018), is not addressed by the experiments. Batchsizes in the experiments vary, but no justification is provided for how these are selected.

Finally, the paper is unfinished as some experimental runs were not complete at the time of submission.

On the positive side, the general point about the necessity of learning rate tuning for adversarial training (described in the fourth paragraph of the introduction) is a very good one, and there may be an opportunity for a more focused application of the proposed algorithm perhaps among further datasets and considering additional, alternative attacks.

***

Suggestions for improvement / questions (related to decision):

* It should not be a challenge to find more image classification datasets to include in the experimental comparison: SVHN, Fashion MNIST, Imagenet, ... Using these, the paper can either follow the standard train/test methodology *across datasets*, i.e., split the meta-dataset into train/test, and/or provide a more compelling body of evidence for the proposed method. Also, the performance dependence on batchsize amongst the proposed algorithm and competitors should be investigated experimentally.

* The Baydin et al. (2018) algorithm should be added to the set of competitors since it would provide a relatively easy* reference point wrt/ "hypergradient" approaches. I don't agree with the statement in the related work section that this entails "additional computation of gradients." *In the sense that the rule should be straightforward to implement.

* The convergence analysis assumption that the optimal oracle SGD follows typical learning rate regimes motivated by loss plateauing seems to be in direct contradiction to the sentiment expressed in the cited Hoffer et al. (2018) paper that such "rules of thumb" may be misguided. Can the authors discuss the appropriateness of their assumption wrt/ this point? Also, in the convergence analysis, the phrase "in expectation" is used twice. This has a specific probabilistic meaning, but appears to be used heuristically in this section. Can the authors clarify whether this usage is informal or formal? If the latter is true, it would be better to provide a more formal convergence argument that explicitly takes the inherent randomness into account.

***

Editorial comments (not related to decision):

* Introduction: The first two sentences of the second paragraph, particularly the second, would do well to have an accompanying reference or references.

* Proposed method: Even as an informal statement, the second sentence of the second paragraph under the Phase 2 sub-heading is problematic. The proposed method does not "resist" lowering the learning rate "for as long as possible" so much as it doesn't lower the learning rate for a fixed number of epochs (algorithm parameter).

* "Adversarial training" section (5.4): The paper assumes the reader is familiar with the terms "FGSM", "white box", and parameters \epsilon and \alpha since these are referred to w/o description. Perhaps a short (2-3 sentence) description of the adversarial scenario could be added?

* Experiments: It would be good for the paper to include RMSProp and Adagrad results to the experimental tables as these rules are both readily available for use and widely used.

* Experiments: Is the reporting of the peak accuracy standard in the literature?

* Experiments: I want to give the paper credit for performance on CIFAR100, but this is difficult without explicit points of comparison. This can be easily remedied by including SOTA performance values (along with appropriate references) in the tables or text.

* (Potential) Typos:
	Proposed method algorithm description:
		Requirements has a weight decay parameter which seems strange given that the algorithm is performing automated learning rate adaptation...
		The epoch counter is incremented in line 5, but not reset prior to Phase 2. Does this mean that Phase 1 training epochs are counted toward the total (T)?
		Line 7 should be \eta_t <- \eta_0 / 2.
		The patience counter in line 15 is not utilized below.
		Line 23 could/should be an else statement.
review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: This paper proposes a new way of scheduling the learning rate in optimization algorithms such as SGD. It is a stand-alone, parameter-free approach that optimistically doubles the learning rate at every loss improvement between two epochs, until the loss increases too much or diverges, in which case the learning rate is divided by two.
This approach is theoretically proven to converge and to follow an optimal scheduling strategy.
In addition, the authors experimentally tested their approach on two image classification tasks, showing that the proposed algorithm yields similar to baseline results.

I am rejecting this paper because it seems to motivate things with non-related facts, experiments are not robust and thorough enough, and there is no conclusion (not even in the appendix).

- The most important thing in this paper to me is the fact that "adversarial training" is used to motivate this approach a lot. it is mentioned 14 times across the paper: 3 times in the abstract alone. Yet there is no explanation of what it is, and how is it different from "natural training" as mentioned in the paper. I suggest the authors either to clearly explain the difference between the two and explain why their approach may help in one setting or the other; or to simply remove the mentions of "adversarial training" if it is not important to the approach.
- to better motivate the approach, I would suggest the authors include different tasks, rather than different training settings. For instance by having one image classification task (keep one of the two current ones) and one text classification or even generation task. This would show that the proposed approach generalizes well to other network architectures.

- The second concern I have is about the experiments. If increasing the learning rate like the proposed approach is making training to convergence faster, then why are the experiments only measuring test set accuracy and not also runtime to convergence?
- Overall, the experiments are not complete and thorough enough: some table values are missing, the set of adversarial training experiments on CIFAR100 are not reported, and some experiments diverged with the ADAM optimizer. Less than 20% accuracy on a 10-class image classification task seems very far from optimal.

- Eventually, I strongly suggest the authors submit a better closing statement than "We use cross entropy loss in all cases." (especially after having read this same sentence earlier in section 5.1 of the paper). No conclusion is added to the paper, not even in the appendix.

Below are a few minor points not taken into account in the scoring but that could make the paper slightly better:
- Section 1, paragraph #1, first sentence: a few citations here would be nice.
- typo on the first line of page 3: "5The pseudocode ..."
- typo in the 2sn paragraph of section 4: "... the convergence is the fast*est* since the step sizes ..."
- typos in the first line of the second paragraph of section 4.1: "Assuming that *the* loss surface is smooth, *the* loss will continue..."
- page 6: "At this time, p=2^{n-1} at this time."
- Section 5.1, paragraph 2, first sentence: "... with dropout and with both dropout and cutout, ..."

review_title: Official Blind Review #2

Meta-review: This paper proposes an automatic tuning procedure for the learning rate of SGD. Reviewers were in agreement over several of the shortcomings of the paper, in particular its heuristic nature. They also took the time to provide several ways of improving the work which I suggest the authors follow should they decide to resubmit it to a later conference.
Decision: Reject