Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper discusses large batch training at its limit -- full-batch cifar10 training when no stochasticity is introduced. This setting allows authors to examine and discuss the common conception about the generalization benefits of SGD. The authors show that replacing implicit bias of SGD with explicit regularization can eliminate the generalization gap.
main_review: The paper has strong merits on the technical side, offering several well-executed novel experiments:
- First full-batch training regime of cifar10 with competitive validation accuracy
- Removal of any noise affecting training (full-batch, fixed batch normalization slices)
- "Augmentation aware" full-batch training -- by pre-enlarging the dataset with fixed augmentation, thus making sure comparison to practice is fair.

The authors also offer some practical methods that may prove useful in the future, by offering gradient-clipping and regularization to combat large batch induced accuracy degradation and failures. 
The paper is also well written and easy to follow with a good cover of past efforts on this topic. 

My main concern with the work is that several choices seem arbitrary and not well explained:
- Number of iterations: as seen in past works (Hoffer et al. 17',  Shallue et al. 19'), adapting the training regime with more training steps can close the gap between large and small batch training. This is also apparent in this work, as authors increase the number of steps to 3000 (10x the number of epochs) to reach the comparable accuracy milestone. However, this number is not justified in any way and no further exploration in that regard is made. Some possible questions that arise: (1) can we apply the same number of iterations as in small batch (117K) to close the gap completely with no need for additional regularization? (2) what is the tradeoff in terms of accuracy? (3) in terms of computational effort?
- Regularization - the authors choose a specific regularizer to replace the implicit bias offered by SGD. However, several other choices can be offered instead (e.g SAM, preconditioning). More importantly, does the regularization really replaces SGD? what accuracy do we get when the same regularizer is applied to baseline SGD training?
- Learning rate adjustment - authors use x2 learning rate instead of previous scaling methods (sqrt/linear scaling), in what seems like a completely arbitrary choice. As previous works used signal-to-noise ratio of gradients to suggest scaling of the learning rate, I expect authors to suggest and demonstrate why (and if) these break at the full-batch limit, when no noise is introduced.

summary_of_the_review: As explained in the review, the paper has several technical merits and interesting experiments but is clouded with ill-justified and what seems like arbitrary choices and solutions. Currently, I feel these hurt the potential value of the paper to the field. 

My view is that the paper in its current form should be rejected, but I suggest the following improvements:
- Discuss regime adaptation (increasing number of steps) similar to Hoffer et al. by performing additional experiments with a varying number of epochs. I understand that a complete 117K steps training in full-batch is too cumbersome, but the current single datapoint of 3000 epochs seems arbitrary and not justified in any way.
- Regularization on baseline SGD -- does it help? will be very interested to see a discussion either way.
- LR adjustment and gradient clipping justification -- perhaps additional measurements of gradient noise may show why these are needed over "traditional" scaling techniques.

If authors work to lift these concerns in their rebuttal, I will consider raising my score.



----- Post rebuttal update -----
The authors answered most of my concerns, so I'm updating my score as promised. I still feel some of the choices are arbitrary (or at least not substantiated enough) and may apply to the specific case presented only (dataset, model), so I will raise my score 5->6.
review_title: null

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: Paper presents empirical results that show that full-batch training of neural networks can still generalize with appropriate explicit regularization. Authors propose explicit regularization to minimum norm solution and discusses techniques for efficient implementation as well as relation to prior work.
main_review: == Updated score ==
 
I am satisfied by the results of the control experiments, and the explanations of the authors. I have read the other reviewer comments, and I believe the work will be quite valuable for the ICLR community and thus have changed the score to a strong accept. 

========

Authors do thorough experiments to bring out the nuances around the topic with the CIFAR-10 dataset, and Residual Networks. 

Strengths:
+ Very thorough set of experiments ablating over various sources of stochasticity
+ Detailed explanation of experimental protocol and reproducibility
+ Shows explicit regularization can bridge the gap, and shows efficient implementation and also discusses prior work (SAM)

Weakness/Questions:
+ Does explicit regularization help SGD (stochastic mini-batch training) (This is an important control experiment)? If it changes the target solution quality - how does one reason about matching generalization of stochastic mini-batch training. 
+ Do the results hold across wider range of datasets, While I can see how much larger datasets could become more expensive - it would have been very satisfying to verify.

Some other comments: Relevant citations that are missing, which hopefully authors can easily address:
Second order optimization that seem to work well for Neural Nets:
https://arxiv.org/abs/1503.05671 (KFAC)
https://arxiv.org/abs/2002.09018 (Shampoo)
https://www.bmvc2020-conference.com/assets/papers/0479.pdf (L-BFGS)

Recent work on curvature and gradient clipping (and other techniques): 
https://arxiv.org/pdf/2110.04369.pdf


summary_of_the_review: This paper provides convincing contrary evidence that shows stochastic mini-batching by itself is not unique, and can be substituted with explicit regularization, and techniques such as gradient clipping.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper examines the role of SGD noise in the generalization performance of neural networks. In particular, the paper examines if the generalization performance of SGD-trained models can be replicated with explicitly regularized models trained with full-batch gradient descent. The authors show that this is indeed possible for a number of different architectures (ResNet, DenseNet, VGG) trained on CIFAR-10 dataset. Based on these observations, the authors conclude that any theory that relies exclusively on stochasticity of training to explain generalization in neural networks is unlikely to capture the true phenomena responsible for the success of deep learning.
main_review: In the deep learning community, it is widely *believed* that the noise in SGD is a necessary component of any performant neural network. Over the past few years, theory papers have emerged that explicitly attribute the success of neural networks to the special properties of SGD. As most of the effort in the community has been focused on models trained with SGD-type algorithms, the conclusions of these papers so far have been mostly unchallenged. By showing that simple full-batch gradient descent can indeed achieve results similar to SGD, the paper provides an impactful contribution to the theory community. The paper is well-written and the experimental set-up is carefully designed and communicated in detailed. The fact that the authors can replicate their findings across a number of different architectures (Table 2) increases my confidence in the presented results.  

I list my questions / concerns below:

* One can attribute variations in the validation accuracy to either issues in the optimization or generalization aspects of the model. When going through Tables 1 & 2 and Appendix C.2, it was not clear to me where exactly the improvements are coming from. I strongly suggest reporting metrics such as training loss & accuracy with the validation metrics in order to avoid confusion. 

* I wonder if adding the explicit regularization provides the same kind of regularization as the noise in gradient descent. In particular, do SGD and GD models make similar predictions (and similar mistakes)? Will adding the explicit regularization on top of SGD improve the model?

* Figure 2 compares the loss flatness across a random direction in parameter space for different models. Can we reliably state anything regarding the flatness of the loss landscape by using 1-D projections of a O(1M) dimensional loss manifold? Moreover, note that the curvature of these models depend on the normalization parameters of the BN layer. By playing with these normalization parameters, one can change the curvature without changing the predictions of the model. This makes the cross-model curvature comparison that Figure 2 is attempting highly unreliable. 
summary_of_the_review: The paper provides a strong contribution to the community. The main conclusions of the paper are supported with strong empirical evidence. There are a number of improvements / clarifications that can be added to the paper (which I have listed above).
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper shows that it is possible to train CIFAR 10 models with full batches, and still obtain get test accuracy.  While training with SGD leads to accuracy 95+ percent, training with baseline full batch degrades accuracy in a range of 30-77 percent.  However, introducing a number of regularization discussed in section 3 can close the validation accuracy gap between SGD and FB.

This paper spends about half the time discussing theory heuristics for SGD regularization, and half the time discussion regularization heuristics which are compatible with full batch optimization.   

As an empirical paper, there are a number of regularization techniques discussed in section 3: baseline sgd, stabilizing training, finite difference regularization of the Hessian of the loss equation (7), learning rate schedules, gradient clipping, gradient penalty, data augmentation. However, CIFAR-10 is no longer a challenging dataset. 

From the paper: "A number of authors have studied relatively large batch training, often finding trade-offs between batch size and model performance (Yamazaki et al., 2019; Mikami et al., 2019; You et al., 2020). However, the goal of these studies has been first and foremost to accelerate training speed (Goyal et al., 2018; Jia et al., 2018), with maintaining accuracy as a secondary goal. In this study, we seek to achieve high performance on full-batch training at all costs. Our focus is not on fast runtimes or ultra-efficient parallelism, but rather on the implications of our experiments for deep learning theory."
main_review: The theoretical implications of the experiments are not significant. SGD is an empirical regularization technique.  So models trained with SGD *may* empirically overfit less than models trained in other ways.  However, SGD is neither necessary nor sufficient for models to generalize.

Some theory papers argue that SGD is sufficient for generalization.  For example, the paper, Stability and Generalization, Bousquet and Elisseeff,  shows that a technical form of stability is sufficient for generalization.  The paper, Train faster, generalize better: Stability of stochastic gradient descent, Moritz Hardt,  Benjamin Recht, Yoram Singer, argues that SGD introduces stability.  However, no one has argued that SGD is *theoretically necessary* for generalization.

The paper offers "A MORE SUBTLE HYPOTHESIS *we can modify and tune optimization hyperparameters for GD and also add an explicit regularizer in order to recover SGD’s generalization performance without injecting any noise into training.*"

To me this is an empirical technique, not a theory hypothesis.  A theory hypothesis would be something along the lines of : perform this type of regularization, and we can prove a bound on the generalization gap of the model.  An empirical theory hypothesis would be something like: perform this type of regularization, and in practise, we obtain similar generalization performance to SGD.  However, this would require more substantial empirical evidence, than is presented in the paper. 


summary_of_the_review: The achievement of the paper is an empirical result: good validation accuracy on CIFAR 10 using full batch training and a number of regularization techniques.  But the paper makes a claim of implications for deep learning theory which is not substantiated.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper shows what the title says; to paraphrase, the paper shows that GD with a bunch of non-stochastic tricks can be as good as SGD with minibatch noise 
main_review: I think the paper reveals an important message to the field. Recent works on SGD are more or less all based on the idea that minibatch training improves generalization and is worth studying on its own, and this work makes the surprising discovery that the minibatch sampling process is, in fact, unnecessary for a good generalization. The same effect can be reproduced by a series of deterministic tricks on GD

However, I feel that the way the authors present the conclusion is a little too negative against SGD. In fact, from a practitioner's point of view, I become much more convinced of the advantage of minibatch training after reading the paper (instead of finding it unnecessary). The reason is this: the paper essentially shows the following:
 - GD + minibatch sampling = GD + learning rate schedule + large learning rate + gradient clipping + gradient penalty

Namely, disregarding the computational efficiency that the minibatch technique brings, simply using mini batching alone already gives me the advantage of all the later four tricks taken together! Not to mention that a simpler technique tends to be more robust in more complicated settings. 

Therefore, I think one additional conclusion that should be reached is that the minibatch sampling technique in SGD is indeed a really good technique -- but this point is not pointed out in the paper, and missing this point I think can be highly misleading for the general audience, given that the paper mainly has a negative tone against SGD. I thus think the authors need to revise their conclusion to reflect this point (or convince me why this additional conclusion is incorrect).

Additionally, I think that one crucial experiment is lacking: if one adds minibatch sampling on top of GD + learning rate schedule + large learning rate + gradient clipping + gradient penalty, what will happen? The authors are obliged to show that when the minibatch sampling is added, there is no further improvement



summary_of_the_review: Update: I am satisfied with the author's reply. I think the message is sufficiently novel and important to be published at ICLR. Though I actually would like to give a 7, due to ICLR's scoring constraints, I can only give a 6. I do not feel that giving 8 is appropriate because this paper only suggests the possibility of SGD noise being non-essential, but does not give a definitive answer. 

To be specific, the main speculation/conjecture is only validated for a single example, it is completely unclear whether these results are generalizable or not. While I do find meaning and insight in the single example the authors give, I feel that more convincing empirical evidence is required to obtain a score higher than 7 and, more importantly, to reduce the speculative nature of the present conclusion.

--------------

I lean towards acceptance because I find the discovery surprising and important. However, I give a 5 because I feel that the authors missed one crucial conclusion, which might mislead the readers, and missed one crucial control experiment. I will change the score if these two points are addressed satisfactorily
review_title: null

Meta-review: This paper carefully shows how all the stochastic elements in neural network training could be removed (by using full batch, and a dataset with fixed augmentation) and still maintain good performance, by adjusting hyper-parameters and adding explicit regularization. 
All reviewers were eventually positive and recommended acceptance, except one reviewer, who was initially not aware of the recent theoretical interest in this question, and was therefore less surprised.

There are three remaining issues with the current version: 
1) Operations in cuDNN are, by default, non-deterministic, but can be made deterministic. Though I believe this will not affect the final results, without it, the title and conclusions of this paper are technically unjustified. The authors have agreed to add this to the camera ready version of the paper.
2) Deterministic training was only shown on CIFAR10. I understand ImageNet would be too heavy for this task, but there are many other small and medium-sized datasets, and I think showing that this on several such datasets would strengthen the message of this paper, and convince more readers. 
3) The question of how to achieve good performance with deterministic training is still mostly unknown, as it seems to require significant hyperparameter search (with unknown sensitivity), and no conclusion was reached regarding the how to properly adjust them. However, I agree that a good answer to this question might not exist.

Lastly, I recommend the authors to mention in the main paper the new baseline experiment, where the explicit regularization is added to SGD. I saw it in the appendix, but I didn't see it mentioned in the main paper (maybe I missed it). I think without it, many readers will not be fully convinced (as several reviewers requested it).
Decision: Accept (Poster)