Title: Inefficiency of stochastic gradient descent with larger mini-batches (and more learners)
Authors: Onkar Bhardwaj
Guojing Cong
Venue: ICLR 2017
Keywords: Deep learning
Optimization

Abstract: Stochastic Gradient Descent (SGD) and its variants are the most important optimization algorithms used in large scale machine learning. Mini-batch version of stochastic gradient is often used in practice for taking advantage of hardware parallelism. In this work, we analyze the effect of mini-batch size over SGD convergence for the case of general non-convex objective functions. Building on the past analyses, we justify mathematically that there can often be a large difference between the convergence guarantees provided by small and large mini-batches (given each instance processes equal number of training samples), while providing experimental evidence for the same. Going further to distributed settings, we show that an analogous effect holds with popular Asynchronous Gradient Descent (\asgd): there can be a large difference between convergence guarantees with increasing number of learners given that the cumulative number of training samples processed remains the same. Thus there is an inherent (and similar) inefficiency introduced in the convergence behavior when we attempt to take advantage of parallelism, either by increasing mini-batch size or by increase the number of learners.

TL;DR: We theoretically justify that increasing mini-batch size or increasing the number of learners can lead to slower SGD/ASGD convergence

Review discussion:
Your theoretical results suggest that for SGD, not using mini-batch (i.e., mini-batch size = 1) is the best. On asynchronous SGD, your results suggest that using only 1 learner is best. These seem contradictory to the common practice. Can you please explain?
What our analysis tries to show is that in terms of convergence in the finite-time regime, using larger minibatch sizes in SGD or having more learners in ASGD is likely to require more samples being processed. The two issues are  related.  Larger minibatch and more learners may still bring practical advantages (as the reviewer commented) on current systems as they can better tolerate noise and/or better utilize the computing resources (e.g., multicore machines, a cluster of machines, and GPUs). Suppose for 2 learners it takes 1.5x samples to reach convergence.
Then ignoring communication cost, having 2 learners is be better than 1. In the parallel setting, our analysis suggests that we do not expect to see linear speedup (even without communication cost). It is not known yet how exactly the number of samples required increases with the number of learners. It is an important question to ask, especially considering the explosion of data. If we want to use thousands of learners, the increase in the number of samples needed for convergence (combined with communication cost) may make such parallel deployment impractical. Perhaps new paradigms are needed.
To add to the previous reply by Guojing Cong, it is well-known that although ASGD provides computational advantages, it often suffers from slower convergence.
For example, see the following:
1. "However, permitting this asynchronous behavior inevitably adds “staleness” to the system wherein some of the workers compute gradients using model parameters that may be several gradient steps behind the most updated set of model parameters.
Thus when fixing the number of iterations, ASGD-trained model tends to be much worse than SSGD-trained model."
from "Staleness-aware Async-SGD for Distributed Deep Learning" by Zhang, et al.
2. "Despite the above mentioned problems, Async-Opt has been shown to be scale well up to a few dozens of workers for some models.
However, at larger scales, increasing the number of machines (and thus staleness of gradients) can result in poorer trained models."
from "Revisiting Distributed Synchronous SGD" by Chen et al.
3. Section 3.2 of http://learningsys.org/papers/LearningSys_2015_paper_13.pdf
In this work, we provide theoretical analysis to support slower convergence (in terms of cumulative number of samples processed by all learners). We show with our analysis that even after we ignore communication cost, ASGD does not scale well in finite-time regime.
Thus even with optimized (or zero-cost) communication, one cannot expect for ASGD to have linear speed-up with increasing number of learners. This is an important issue especially when in the future, when with exploding data we may need to use thousands of learners.
I can't seem to find the complete proof of Theorem 2. The main body of the paper shows only the outline and states the complete proof is in the appendix, whereas the appendix also only shows the outline and states: "See the appendix for complete proof". Is the appendix showing the complete proof? If not, the proof needs to be added.
Minor: page 1: "...often it is often...", page 3: "alarger"
Thank you very much for pointing out the source of confusion and the typos.
1. Appendix is showing the complete proof of Theorem 2 (we removed the additional "see the appendix for the complete proof" from the appendix). In that proof, a certain coefficient is shown to take values \sqrt{2} + M_l/\alpha in the first bullet item and 2M_h/\alpha in the second bullet item. Taking ratio of these gives us the desired conclusion.
2. We have corrected the other typos.
Thanks again, and we have uploaded the corrected pdf.
What is the improvement or difference between the result in this paper and Lian et al. 2015?
In Lian et al. , the authors showed the "asymptotic" convergence rate of SGD to Asynchronous SGD (ASGD), i. e. , it was shown that ASGD converges at the rate of O(1/\sqrt{S}) asymptotically where S is the cumulative number of samples processed. The difference is that we are primarily concerned with the convergence behavior prior to the asymptotic regime. ASGD is known to be slower to converge when we use a large number of learners. We justify it theoretically by showing that prior to the asymptotic regime ASGD's convergence can get slower as we increase the number of learners.
Note that here slower convergence theoretically means worsening of the convergence guarantee, reached at the end of cumulatively processing a fixed number of samples (i. e. , the convergence guarantee with all N learners together processing S number of samples, and each learner processes S/N number of samples). This slow convergence corresponds to the practical observations of lack of linear speed-up often observed while using ASGD. In the first part of the paper, we do similar analysis for SGD with respect to the mini-batch size.
Thanks for your comments and let us know in case more clarifications are needed.
This paper addresses the problem of the influence of mini-batch size on the SGD convergence in a general non-convex setting. The results are then translated to analyze the influence of the number of learners on ASGD. I find the problem addressed in the paper relevant and the theoretical part clearly written. The experimental evaluation is somehow limited though. I would like to see experiments on more data sets and more architectures, as well as richer evaluation, e.g. N=16 is a fairly small experiment. It would also enhance the paper if the experiments were showing a similar behavior of other popular methods like momentum SGD or maybe EASGD (the latter in distributed setting). I understand the last evaluation does not directly lie in the scope of the paper, though adding these few experiments do not require much additional work and should be done.
Thanks for your comments and encouragement. We will surely add more experiments in the future versions of this paper.
This paper shows that when a larger mini-batch is used (in the serial setting), the number of samples needed to be processed for the same convergence guarantee is larger. A similar behavior is discussed for using multiple learners in asynchronous SGD. This behavior has been known in convex optimization (e.g., "Better Mini-Batch Algorithms via Accelerated Gradient Methods", NIPS 2011). There, the convergence is of the form O(1/\sqrt{bT}+1/T), and so using bT samples only lead to \sqrt{b} time improvement. This paper extends a similar result to the nonconvex case (but the underlying mathematics is mainly from [Ghadimi & Lan, 2013]).
However, this behavior is also known and indeed has been summarized in the deep learning textbook (chapter 8).
Hence, the novelty is limited.
The theoretical results in this paper suggest that it is best not to use mini-batch.
However, using mini-batch is often beneficial in practice. As discussed in the deep learning textbook, using mini-batch allows using a larger learning rate (note that this paper assumes the same learning rate for both mini-batch sizes).
Moreover, multicore architectures allows parallel execution of mini-batch almost for free.
Hence, the practical significance of the results is also limited.
Other comments:
- Equation (4): since the same number of samples (S) is processed and S=MK, where M is the mini-batch size and K is the number of mini-batches processed (as mentioned in the first paragraph of section 2), when two different mini-batch sizes are considered (M_l and M_h), their K's should differ.
However, the same K is used on the LHS of (4).
- Figures 1 and 2: As convergence speed is of main interest, why not show the training objective instead?
Thanks for your comments and suggestions.
Although similar behavior is known earlier, we tried to come up with mathematical justification (in non-asymptotic regime).
Although the reviewer is correct in suggesting that using mini-batch is often beneficial in practice because of parallelization, this approach can only work with a smaller number of learners.
Afterwards, using larger mini-batches can often lead to a much slower of convergence speed.
Thanks for the suggestions regarding changes in the manuscript. We will make the changes in the future versions.
This paper theoretically justified a faster convergence (in terms of average gradient norm attained after processing a fixed number of samples) of using small mini-batches for SGD or ASGD with smaller number of learners. This indicates that there is an inherent inefficiency in the speed-up obtained with parallelizing gradient descent methods by taking advantage of hardware. This paper looks good overall and makes some connection between algorithm design and hardware properties.
My main concern is that Lemma 1 looks incorrect to me. The factor D_f / S should be D_f/ (S*M) for me. Please clarify this and check the subsequent theorem.
Thanks for your comments and encouragement. I verified that the factor is correct (unless I missed something again). S already has M factored in (S has been defined as cumulative number of samples processed, which for Lemma 1 would be equal to MK).

Meta-review:
role: PC
data: comment: The work addresses the question of whether mini-batching improves the convergence of stochastic gradient methods, in terms of the number of examples, in the general non-asymptotic/non-convex setting of Ghadimi and Lan. Similar results are already known (at least as folk theory) in simpler regimes, but this result is novel. The reviewers are mixed in their opinions of the paper, but AnonReviewer4 brings up a point that deserves further attention: the analysis assumes that the same step-size is used for different mini-batch sizes. Because of this assumption, the conclusion of the paper is misleading. One of the effects of increasing the mini-batch is to decrease the variance of the estimator, which allows a larger step-size. With this larger step-size the net effect of increasing the mini-batch can actually be positive even in terms of the number of examples. Fixing the step-size removes the chance for the mini-batch strategy to do better in theory, and there is indeed a large amount of empirical evidence that mini-batches can lead to practical improvements. I do think the authors should continue this work, but I think accepting the current version is problematic because of the current conclusion.

Decision: Reject