Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper provides an empirical study comparing a range of proposed mechanisms by which small batch SGD enhances generalization. 

To achieve this, the authors split large mini-batches into a set of "micro-batches". They then evaluate a range of proposed implicit regularization terms on each micro-batch before summing across micro-batches to form a single large batch update. They show that applying the gradient norm/Fisher trace regularizers proposed by Smith et al. [2021] and Jastrzebski et al. [2020] to these micro-batches consistently closes the generalization gap between small and large batch training, while Jacobian regularization does not. Additionally, as predicted by prior work, they confirm that the benefits of gradient norm regularization diminish as the micro-batch size rises.
strength_and_weaknesses: Strengths:

1) The micro-batch concept enables the authors to directly minimize the modified loss proposed in Smith et al. while increasing the batch size. This is also practical, since large batch training is usually achieved by parallelizing across multiple devices.
2) The authors provide experiments across a range of models/datasets, which consistently show that gradient norm and Fisher trace regularization closes the gap between small and large batch training, thus supporting the claim that small batch SGD implicitly regularizes these quantities.
3) The authors confirm empirically that these regularizers perform poorly when the micro-batch size is too large. 

Weaknesses:
1) The novelty of the work is quite limited, however I think this is fine for an empirical evaluation.
2) It would have been nice to include an ImageNet experiment.
3) I didn't understand how the grafting experiments connected to the main theme of the paper?
4) A longer background section or appendix describing the relevant prior work in detail would help readers less familiar with the topic.
clarity,_quality,_novelty_and_reproducibility: Clarity:
The paper was mostly clear

Quality:
The quality of the work is reasonably high

Novelty:
The novelty of the submission is limited, but this is reasonable for an empirical study

Reproducibility:
It would be relatively straightforward to replicate the results.

Other comments:
1) I do not understand point three in the list of main contributions. Please could the authors clarify what they mean?
2) Note that all of the mechanisms in the first paragraph of section 2 are closely related.
3) Section 2.1 first paragraph: Barrett and Dherin study full batch gradient descent, the correct citation here is Smith et al., which extended Barrett and Dherin's results to SGD.
summary_of_the_review: The authors present some useful experiments clarifying which proposed mechanisms for the implicit regularization of small mini-batches are most effective in practice at closing the generalization gap between small and large batch training.

I am currently recommending marginal accept since I think this is a helpful contribution. The experiments are reasonably thorough but no large scale datasets are included, and the novelty/originality of the submission is not very high.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work studies the implicit bias of SGD by investigating which kind of explicit regularization can help large batch SGD/GD match the performance of (small batch) SGD. Several new insights are drawn from a systematic set of experiments. 

Firstly, the effect of several explicit regularizer are studied, including (1) micro-batch gradient norm regularization, (2) micro-batch fisher trace regularization, (3) micro-batch average Jacobian regularization and (4) micro-batch random and unit Jacobian regularization. It is shown that the first two regularizers are more effective than the last two. 

Secondly, the work turns to looks at some extreme use cases of micro-batch gradient norm regularization: (1) when the micro batch size approaches the scale of the batch size of "large batch SGD", the regularizer no longer works well; (2) when the micro-batch gradient norm regularization is approximated by a sample micro-batch gradient norm regularization, the latter regularizer works well when he regularization strength is property turned. 

Finally, the paper considered a very interesting set of experiments that involve gradient grafting, which mixes the direction and magnitude from two separate (or related) optimizers. It is shown that, for some CIFAR-10 settings, gradient grafting can recover the performance of SGD, indicating that perhaps the direction of large batch SGD update is good enough but its magnitude could have been improved. Yet, this conclusion does not carry over to harder settings, where improving the magnitude of large batch SGD update might not be sufficient for it to generalize well. 

Based on the last result, an inconsistent performance of one algorithm in easy/hard deep learning application, the paper discusses some limitations of current scientific study of deep learning, where most experimental evidence is from relatively easy tasks and might not migrate to harder tasks.  
strength_and_weaknesses: # Strength
+ The main contributions are summarized in the Summary section. Some other strength of this work includes:
+ The paper is overall very well organized. I have a good time readying most of the paper (except for a few places please see below). 
+ As a scientific paper, more experiments are always welcome. With that being said, I think the conclusions presented in this work are well backed up by its experimental design. At this point I did not find a major shortcoming in terms of experimental design. 
+ I really like the design of Section 5 which disentangles the effect of the magnitude and direction of the update. Such a perspective is novel to my knowledge for understanding SGD (despite it has been studied from a more theory work), and I think it may inspire a deeper understanding of SGD. 


# Weakness
- A large portion of the work focuses on verifying which type of explicit regularization for large batch SGD/GD can recover the performance of small batch SGD. I would like to see some discussions on why finding the right explicit regularization (that corresponds to the implicit regularization of SGD) is important or useful. 
- Note that the explicit regularization term introduces several new hyperparameters. For example the penalty strength $\\lambda$. I would like to see some discussion on how hard it is to tune the penalty strength $\\lambda$. Partly due to the new hyperparameters, I somehow feel the finding of this work might not be very useful for practitioners. 
- I find Section 3.2, paragraph (ii) is very hard to interpret. Would you mind elaborate a bit? Perhaps written down some math could help. 
clarity,_quality,_novelty_and_reproducibility: Clarity is good. 

Quality is solid. 

Novelty is fine. Sections 3 and 4 might not be very novel. But Section 5 proposes a new perspective to me. 

Reproducibility is good. 
summary_of_the_review: Please see above. At this point I would like to vote for a weak acceptance. I will consider increasing the score based on the authors reply to my questions. I might also consider decreasing the score if I or other reviewers find a major vulnerability in the experimental design that cannot be satisfactorily addressed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper does a thorough simulations to clarify and verify the existing hypothesis about why small batch SGD outperform its large batch counterpart. They derive many interesting results. First, they show that adding regularizer based on gradient norm or the Fisher Information Matrix trace can recover small-batch SGD generalization, while Jacobian-based regularizations fail. On the other hand, they show that the gradient norm, in some case, is not a good indicator of generalization, i.e., there exists some algorithms achieving good generalization but with large gradient norm. Last, they show that some tricks work on small dataset like CIFAR10 might fail on large dataset like CIFAR100, Anti-PGD, for example.
strength_and_weaknesses: ## Strength
- Though this paper is purely empirical, it provides many important intuitions for future work. For example, the failure of the Jacobian-based regularizations shows that this might not the reason for the good generalization of small-batch SGD. On the other hand, the counterexample in Figure 4 shows that we should seek other explanations beyond gradient norm regularization. The last observation that some tricks might fail on large datasets calls for verifying all the previous tricks only verified on small datasets and rethinking the importance of dataset structure. 
- The paper is well structured. I enjoy the process of reading this paper.  


## Weaknesses
- The performance reported in the paper is not the SOTA. I suspect the reason is that they do not use weight decay and momentum, as mentioned in the appendix. I can understand that weight decay or momentum can be regarded as some kind of regularization. Moreover, I did not find the stepsize regime. Do you use constant stepsize? In the reality, we always add weight and momentum, together with stepsize decay (or cosine learning rate) to improve the generalization. I will be very happy to increase my score if you can provide simulations showing that gradient norm regularization can also recover the generalization after adding all these common tricks.
clarity,_quality,_novelty_and_reproducibility: The paper has good writing and is easy to read. Some of its results are novel, which I believe will broadly impact the community.
summary_of_the_review: Please see the strength and weaknesses part.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper considers micro-batches, which are disjoint smaller subsets of mini-batch. Empirical evidence is provided to suggest that, although a large mini-batch does not lead to generalization as good as a small mini-batch, the introduction of micro-batches together with a large mini-batch helps recover the generalization corresponding to a small mini-batch.
strength_and_weaknesses: Strength: using micro-batch as an estimator of the gradient norm and a regularizer is a very interesting idea. It seems to have a lot of potential.

Weakness: 

(1) very little theoretical justification is provided, but the empirical results are not comprehensive enough to support the claims. 

(2) if I didn’t misunderstand, one central goal of this paper is to suggest that small-batch SGD has an implicit bias that regularizes gradient norm. I’d like to better understand why this wasn’t investigated directly, but rather via an indirect route based on the extra complication of micro-batch. I’m relatively convinced that micro-batch has this implicit bias, but curious about why this serves a demonstration of the small-batch SGD story.

(3) a traditional belief is that smaller batch size leads to behaviors similar to those under larger learning rate. The implicit bias of large learning rate, in terms of regularizing the gradient, for example, has been studied already, not only empirically but also theoretically. In addition, the implicit bias of small batch size has also been studied, see e.g., [Smith et al. ICLR 2021], which was already cited. Therefore, it is not completely clear to me what’s new in terms of SGD’s implicit bias. The micro-batch part is still interesting though.

clarity,_quality,_novelty_and_reproducibility: I wish how this work positions in the literature could be clarified more. It could just be a clarity issue, but without better understanding this, I’m incapable of assessing the originality.

I also hope the authors could clarify about whether/how micro-batch is better than small mini-batch in practice? Is it computationally more efficient? It could just be me but I feel this part is not so clear to me.
summary_of_the_review: Overall, I feel this paper definitely proposed an interesting idea. However, I hope to understand better what is the gain, as well as to see more evidence of its effectiveness (if not theoretically, at least empirically, and more comprehensively).
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper undertakes an empirical investigation of several mechanisms that have been proposed as explanations for the implicit regularization effects of small-batch SGD. By applying different regularization techniques to subsets ("micro-batches") of the full batch, the authors conclude that some techniques (like gradient norm/Fisher trace regularization) can close the generalization gap between large and small batch training, whereas other techniques (like Jacobian regularization) do not perform as well.

The reviewers appreciated the overall motivation and experimental setup, particularly the use of micro-batches, and found the conclusions interesting and relevant to the community. The reviewers generally found the experiments to be thorough, although larger-scale experiments would have been appreciated. Some reviewers noted that the novelty is limited relative to prior work, and that that prior work was not discussed clearly or sufficiently.

Overall, the reviewers and I agree that the work is interesting and, despite some of its limitations, will likely be of interest to the ICLR community. Therefore I recommend acceptance.
justification_for_why_not_higher_score: The reviewers highlighted several shortcomings of the paper, including low overall performance of some models, lack of large-scale experiments, and an insufficient discussion of related work. These limitations are either understandable given the scope and goals, or were addressed in the discussion phase. The major weakness that persists is the lack of novelty relative to prior work, as the paper is basically performing an improved/collected empirical investigation of ideas that have appeared previously. Therefore, while this paper makes important contributions, it does not break much new ground and that is why I do not recommend a higher score.
justification_for_why_not_lower_score: The reviewers and I agree that the empirical investigation is interesting with useful conclusions that are relevant for the ICLR community. It addresses salient questions and is written clearly. There is significant value in this work and for that reason I do not recommend a lower score.
Decision: Accept: poster