Title: On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
Authors: Nitish Shirish Keskar
Dheevatsa Mudigere
Jorge Nocedal
Mikhail Smelyanskiy
Ping Tak Peter Tang
Venue: ICLR 2017
Keywords: Deep learning
Optimization

Abstract: The stochastic gradient descent (SGD) method and its variants are algorithms of choice for many Deep Learning tasks. These methods operate in a small-batch regime wherein a fraction of the training data,  say $32$--$512$ data points, is sampled to compute an approximation to the gradient. It has been observed in practice that when using a larger batch there is a  degradation in the quality of the model, as measured by its ability to generalize. We investigate the cause for this generalization drop in the large-batch regime and present numerical evidence that supports the view that large-batch methods tend to converge to sharp minimizers of the training and testing functions---and as is well known, sharp minima lead to poorer generalization. In contrast, small-batch methods consistently converge to flat minimizers, and our experiments support a commonly held view that this is due to the inherent noise in the gradient estimation. We  discuss several  strategies to attempt to help large-batch methods eliminate this generalization gap.

TL;DR: We present numerical evidence for the argument that if deep networks are trained using large (mini-)batches, they converge to sharp minimizers, and these minimizers have poor generalization properties. 

Review discussion:
If I understand correctly, the paragraphs above Eq (3) claim the sharpness criterion is scale invariant. Does that happen when the denominator in Eq (3) is "1+f(x)". f(x) could have any scale/unit but "1" is a unitless constant. How the proposed criterion is scale invariant with "1+f(x)"?
Specifically, if I replace the objective function f(x) with a scaled version c*f(x), does your criterion produces the same value?
You are right, this metric is not scale invariant, and we will remove that statement from the paper in our next update. The inclusion of the “1+” in the denominator is of course to guard against the case when f(x) is zero, and as such it is useful for our purposes.
Nevertheless, in response to your comment, we experiment with the denominator changed to f(x0) - f(x), where x0 is the initial point, and observed similar order-of-magnitude results, indicating that our numbers reported in the paper are not misleading.
true "local minima" or just "local minima" of one-dimensional slices of a unimodal (within the search boundary) nonconvex problem?
Agreed, we cannot guarantee that the solutions obtained by the SB and LB methods are indeed local minima of the problem (as we did not attempt to verify second-order optimality). To be cautious, we will use the terms “LB solution” and “SB solution”, as appropriate, in our updated manuscript.
I think that the paper is quite interesting and useful.
It might benefit from additional investigations, e.g., by adding some rescaled Gaussian noise to gradients during the LB regime one can get advantages of the SB regime.
Thank you for your review. We experimented with additive random Gaussian noise (both in gradients and in iterates), noisy labels and noisy input-data.
However, despite significant tuning of the hyperparameters of the random noise, we did not observe any consistent improvements in testing error.
Overall, our feeling is that this needs deeper investigation and that LB methods may need to be modified in a more fundamental way to achieve good generalization.
The paper is an empirical study to justify that: 1. SGD with smaller batch sizes converges to flatter minima, 2. flatter minima have better generalization ability.
Pros and Cons:
Although there is little novelty in the paper, I think the work is of great value in shedding light into some interesting questions around generalization of deep networks.
Significance:
I think such results may have impact on both theory and practice, respectively by suggesting what assumptions are legitimate for real scenarios for building new theories, or be used heuristically to develop new algorithms with generalization by smart manipulation of mini-batch sizes.
Comments:
Earlier I had some concern about the correctness of a claim made by the authors, which is resolved now. They had claimed their proposed sharpness criterion is scale invariance. They took care of it by removing this claim in the revised version.
Thanks for your review and your suggestions on how our work could be used in further investigations.
Interesting paper, definitely provides value to the community by discussing why large batch gradient descent does not work too well
We thank you for your review and your kind assessment.

Meta-review:
role: PC
data: comment: All reviews (including the public one) were extremely positive, and this sheds light on a universal engineering issue that arises in fitting non-convex models. I think the community will benefit a lot from the insights here.

Decision: Accept (Oral)