Title: Improving Stochastic Gradient Descent with Feedback
Authors: Jayanth Koushik
Hiroaki Hayashi
Venue: ICLR 2017
Keywords: Deep learning
Optimization

Abstract: In this paper we propose a simple and efficient method for improving stochastic gradient descent methods by using feedback from the objective function. The method tracks the relative changes in the objective function with a running average, and uses it to adaptively tune the learning rate in stochastic gradient descent. We specifically apply this idea to modify Adam, a popular algorithm for training deep neural networks. We conduct experiments to compare the resulting algorithm, which we call Eve, with state of the art methods used for training deep learning models. We train CNNs for image classification, and RNNs for language modeling and question answering. Our experiments show that Eve outperforms all other algorithms on these benchmark tasks. We also analyze the behavior of the feedback mechanism during the training process.


TL;DR: We improve stochastic gradient descent by incorporating feedback from the objective function

Review discussion:
what about time complexity? could you add two more sub-figures of Figure 1 with wallclock time on x-axis?
We will try to get these results soon, but we do note that since we only perform elementary operations, the running time shouldn’t increase significantly.
We ran the CIFAR 10 experiment 10 times with Adam and Eve for 100 epochs each. Adam took 2233.11 seconds on an average for 100 epochs with a standard deviation of 9.24 seconds. For Eve, the mean was 2265.49 seconds, and the standard deviation was 19.03 seconds. Taking into account the number of samples and the batch size, we compute that per update, Eve takes on an average 0.83 milliseconds more than Adam.
We haven't added this to the paper, since the specific times are very dependent on many external factors, making conclusions hard to draw especially when the results are close.
But we again note that all considered methods, including Eve, are just simple modifications of stochastic gradient descent, so their times are within constant factors of each other.
The algorithm does not seem to be invariant to arbitrary shift in the objective function. If the objective function is shifted by a constant c , r_t would be be a different value from Algorithm 1 and thus the learning will take a different trajectory even though it is the exact same optimization problem.
Hi. Thanks for the comment. This point has been brought up elsewhere; we are working on a proper fix for the issue, but for now we've updated the paper to add an assumption which prevents arbitrary shifts. The assumption holds for all experiments performed in the paper, and the results remain valid.
As you noted for Figure 5 Left, sometimes it seems sufficient to tune learning rates. I see your argument for Figure 6 Right,
but
1) not for all good learning rates make Adam fail, I guess you selected the one where it did (note that Adam was several times faster than Eve in the beginning)
2) I don't buy "Eve always converges" because you show it only for 0.1 and since Eve is not Adam, 0.1 of Adam is not 0.1 of Eve because of d_t.
To my understanding, you define d_t over time with 3 hyperparameters.
Similarly, one can define d_t directly. The behaviour of d_t that you show is not extraordinary and can be parameterized. If Eve is better than Adam, then looking at d_t we can directly see whether we underestimated or overestimated learning rates. You could argue that Eve does it automatically but you do tune learning rates for each problem individually anyway.
Thanks for reviewing! We want to note that the analysis around Figure 5 was only for the simplified case of batch gradient descent on a convex problem. What we mainly wanted to show is that in this case Eve is not equivalent to Adam with a larger learning rate. Figure 5 right does show *all* 10 runs of both Adam and Eve, and it is indeed the case that Adam does sometimes diverge whereas Eve does not. The reason for choosing 0.1 is that d_t converges to 0.1, so the learning rate becomes 10 times in the end - and Eve does become 0.1 of Adam eventually.
Regarding the second point, we note that in the more general non-convex case, it is not sufficient to simply tune the learning rate. This is indicated by Figure 2 left which shows a behavior of d_t which is not easy to parametrize. And because of its non-monotone nature, it would not be the same to simply increase or decrease the learning rate. For the particular case of Figure 2, the learning rate first gets increased, and then decreased.
We hope that answers your concerns.
The paper demonstrates a semi-automatic learning rate schedule for the Adam optimizer, called Eve. Originality is somehow limited but the method appears to have a positive effect on neural network training. The paper is well written and illustrations are appropriate.
Pros:
- probably a more sophisticated scheduling technique than a simple decay term
- reasonable results on the CIFAR dataset (although with comparably small neural network)
Cons:
- effect of momentum term would be of interest
- the Adam reference doesn't point to the conference publications but only to arxiv
- comparison to Adam not entirely conclusive
Thank you for the review. We want to address the 3 indicated cons:
- Could you please clarify what you mean by the momentum term? This point isn't very clear to us.
- Thank you for pointing this out. We will update the paper to fix the citation.
- We conducted experiments across tasks (vision, language) on standard benchmarking datasets, and in all cases, the proposed method outperforms Adam. Could you indicate if there is any specific additional analysis you wanted to see?
> - Could you please clarify what you mean by the momentum term? This point isn't very clear to us.
Gradient descent methods, including Adam, usually become more stable when adding a momentum therm that interpolates between the current update and the previous model parameters. Especially the Nesterov momentum gained popularity when combined with Adam.
> Could you indicate if there is any specific additional analysis you wanted to see?
Not necessarily an additional analysis but maybe acknowledge that the other algorithms are competitive in many cases
The paper introduced an extension of Adam optimizer that automatically adjust learning rate by comparing the subsequent values of the cost function during training. The authors empirically demonstrated the benefit of the Eve optimizer on CIFAR convnets, logistic regression and RNN problems.
I have the following concerns about the paper
- The proposed method is VARIANT to arbitrary shifts and scaling to the cost function.
- A more fair comparison with other baseline methods would be using additional exponential decay learning scheduling between the lower and upper threshold of d_t. I suspect 1/d_t just shrinks as an exponential decay from Figure 2.
- Three additional hyper-parameters: k, K, \beta_3.
Overall, I think the method has its fundamental flew and the paper offers very limited novelty. There is no theoretical justification on the modification, and it would be good for the authors to discuss the potential failure mode of the proposed method.
Furthermore, it is hard for me to follow Section 3.2. The writing quality and clarity of the method section can be further improved.

Meta-review:
role: PC
data: comment: The authors propose a simple strategy that uses function values to improve the performance of Adam. There is no theoretical analysis of this variant, but there is an extensive empirical evaluation. A disadvantage of the proposed approach is that it has 3 parameters to tune, but the same parameters are used across experiments. Overall however, the PCs believe that this paper doesn't quite reach the level expected for ICLR and thus cannot be accepted.

Decision: Reject