Title: On the insufficiency of existing momentum schemes for Stochastic Optimization
Authors: Rahul Kidambi
Praneeth Netrapalli
Prateek Jain
Sham M. Kakade
Venue: ICLR 2018
Keywords: Stochastic Gradient Descent
Deep Learning
Momentum
Acceleration
Heavy Ball
Nesterov Acceleration
Stochastic Optimization
SGD
Accelerated Stochastic Gradient Descent

Abstract: Momentum based stochastic gradient methods such as heavy ball (HB) and Nesterov's accelerated gradient descent (NAG) method are widely used in practice for training deep networks and other supervised learning models, as they often provide significant improvements over stochastic gradient descent (SGD). Rigorously speaking, fast gradient methods have provable improvements over gradient descent only for the deterministic case, where the gradients are exact. In the stochastic case, the popular explanations for their wide applicability is that when these fast gradient methods are applied in the stochastic case, they partially mimic their exact gradient counterparts, resulting in some practical gain. This work provides a counterpoint to this belief by proving that there exist simple problem instances where these methods cannot outperform SGD despite the best setting of its parameters. These negative problem instances are, in an informal sense, generic; they do not look like carefully constructed pathological instances. These results suggest (along with empirical evidence) that HB or NAG's practical performance gains are a by-product of minibatching.

Furthermore, this work provides a viable (and provable) alternative, which, on the same set of problem instances, significantly improves over HB, NAG, and SGD's performance. This algorithm, referred to as Accelerated Stochastic Gradient Descent (ASGD), is a simple to implement stochastic algorithm, based on a relatively less popular variant of Nesterov's Acceleration. Extensive empirical results in this paper show that ASGD has performance gains over HB, NAG, and SGD. The code for implementing the ASGD Algorithm can be found at https://github.com/rahulkidambi/AccSGD.


TL;DR: Existing momentum/acceleration schemes such as heavy ball method and Nesterov's acceleration employed with stochastic gradients do not improve over vanilla stochastic gradient descent, especially when employed with small batch sizes.

Review discussion:
I like the idea of the paper. Momentum and accelerations are proved to be very useful both in deterministic and stochastic optimization. It is natural that it is understood better in the deterministic case.
However, this comes quite naturally, as deterministic case is a bit easier ;) Indeed, just recently people start looking an accelerating in stochastic formulations. There is already accelerated SVRG, Jain et al 2017, or even Richtarik et al (arXiv: 1706.01108, arXiv:1710.10737).
I would somehow split the contributions into two parts:
1) Theoretical contribution: Proposition 3 (+ proofs in appendix)
2) Experimental comparison.
I like the experimental part (it is written clearly, and all experiments are described in a lot of detail).
I really like the Proposition 3 as this is the most important contribution of the paper. (Indeed, Algorithms 1 and 2 are for reference and Algorithm 3 was basically described in Jain, right?).
Significance: I think that this paper is important because it shows that the classical HB method cannot achieve acceleration in a stochastic regime.
Clarity: I was easy to read the paper and understand it.
Few minor comments:
1. Page 1, Paragraph 1: It is not known only for smooth problems, it is also true for simple non-smooth (see e.g. https://link.springer.com/article/10.1007/s10107-012-0629-5)
2. In abstract : Line 6 - not completely true, there is accelerated SVRG method, i.e. the gradient is not exact there, also see Recht (https://arxiv.org/pdf/1701.03863.pdf) or Richtarik et al (arXiv: 1706.01108, arXiv:1710.10737) for some examples where acceleration can be proved when you do not have an exact gradient.
3. Page 2, block "4" missing "." in "SGD We validate"....
4. Section 2. I think you are missing 1/2 in the definition of the function.
Otherwise, you would have a constant "2" in the Hessian, i.e. H= 2 E[xx^T].
So please define the function as  f_i(w) = 1/2 (y - <w,x_i>)^2. The same applies to Section 3.
5. Page 6, last line, .... was downloaded from "pre". I know it is a link, but when printed, it looks weird.
Thanks a lot for insightful comments.  We have updated the paper taking into account several of your comments. We will make more updates according to your suggestions.
Paper organization: we will try to better organize the paper to highlight the contributions.
Proposition 3's importance: yes, your assessment is spot on.
Minor comment 1,2: Thanks for pointing the minor mistake, we have updated the corresponding lines. Papers such as Accelerated SVRG, Recht et al. are offline stochastic accelerated methods. The paper of Richtarik (arXiv:1706.01108) deals with solving consistent linear systems in the offline setting; (arXiv:1710.10737) is certainly relevant and we will add more detailed comparison with this line of work.
Minor comment 3, 5: thanks for pointing out  the typos. They are fixed.
Minor comment 4: Actually, the problem is a discrete problem where one observes one hot vectors in 2-dimensions, each of the vectors can occur with probability 1/2.
So this is the reason why the Hessian does not carry an added factor of 2.
I wonder how the ASGD compares to other optimization schemes applicable to DL, like Entropy-SGD, which is yet another algorithm that provably improves over SGD. This question is also valid when it comes to other optimization schemes that are designed for deep learning problems.
For instance, Entropy-SGD and Path-SGD should be mentioned and compared with. As a consequence, the literature analysis is insufficient.
Authors provided necessary clarifications. I am raising my score.
Thanks for your comments.
We have cited Entropy SGD and Path SGD papers and discuss the differences in Section 6 (related works).
However, both the methods are complementary to our method.
Entropy SGD adds a local strong convexity term to the objective function to improve generalization.
However, currently we do not understand convergence rates or generalization performance of the technique rigorously, even for convex problems. The paper proposes to use SGD to optimize the altered objective function and mentions that one can use SGD+momentum as well (below algorithm box on page 6). Naturally, one can use the ASGD method as well to optimize the proposed objective function in the paper.
Path SGD uses a modified SGD like update to ensure invariance to the scale of the data. Here again, the main goal is orthogonal to our work and one can easily use ASGD method in the same framework.
I only got access to the paper after the review deadline; and did not have a chance to read it until now.
Hence the lateness and brevity.
The paper is reasonably well written, and tackles an important problem. I did not check the mathematics.
Besides the missing literature mentioned by other reviewers (all directly relevant to the current paper), the authors should also comment on the availability of accelerated methods inn the finite sum / ERM setting. There, the questions this paper is asking are resolved, and properly modified stochastic methods exist which offer acceleration over SGD (and not through minibatching). This paper does not comment on these developments. Look at accelerated SDCA (APPROX, ASDCA), accelerated SVRG (Katyusha) and so on.
Provided these changes are made, I am happy to suggest acceptance.
Thanks for the references, we have included them in the paper and added a paragraph in Section 6 providing detailed comparison and key differences that we summarize below:
ASDCA, Katyusha, accelerated SVRG: these methods are "offline" stochastic algorithms that is they require  multiple passes over the data and require multiple rounds of full gradient computation (over the entire training data).
In contrast, ASGD is a single pass algorithm and requires gradient computation only a single data point at a time step. In the context of deep learning, this is a critical difference, as computing gradient over entire training data can be extremely slow. See Frostig, Ge, Kakade, Sidford ``Competing with the ERM in a single pass" (https://arxiv.org/pdf/1412.6606.pdf) for a more detailed discussion on online vs offline stochastic methods.
Moreover, the rate of convergence of the ASDCA depend on \sqrt{\kappa n} while the method studied in this paper has \sqrt{\kappa \tilde{kappa}} dependence where \tilde{kappa} can be much smaller than n.
We group the list of changes made to the manuscript based on suggestions of reviewers:
AnonReviewer 3:
- Added a paragraph on accelerated and fast methods for finite sums and their implications in the deep learning context. (in related work)
AnonReviewer 2:
- Included reference on Acceleration for simple non-smooth problems. (in page 1)
- Included reference on Accelerated SVRG and other suggested references. (in related work)
- Fixed citations for pytorch/download links and fixed typos.
AnonReviewer 1:
- Added a paragraph on entropic sgd and path normalized sgd and their complimentary nature compared to this work's message (in related work section).
Other changes:
- In the related work: background about Stochastic Heavy Ball, adding references addressing reviewer feedback.
- Removed statement on generalization/batch size. (page 2)
- Fixed minor typos. (page 3)
- Added comment about NAG lower bound conjecture. (page 4, below proposition 3)

Meta-review:
role: PC
data: comment: The reviewers unanimously recommended that this paper be accepted, as it contains an important theoretical result that there are problems for which heavy-ball momentum cannot outperform SGD. The theory is backed up by solid experimental results, and the writing is clear. While the reviewers were originally concerned that the paper was missing a discussion of some related algorithms (ASVRG and ASDCA) that were handled in discussion.


Decision: Accept (Oral)