Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes two simple modifications of the SAM optimizer that allow to significantly improve its computational efficiency without sacrificing performance.
main_review: This is a nice paper, that is well written and addressing a very clear problem: reducing the computational complexity of SAM. It proposes two simple tricks that both make sense, and shows that they improve efficiency without hurting performance.

## SWP Implementation

I wanted to ask a question about SWP: how do you implement it in practice? I am a bit surprised that not computing the gradients for a random subset (e.g. 50%) of weights leads to a significant improvement in performance. Do you have to do any tricks to achieve this improvement?

## Alignment of SSP gradient with full-batch gradient

You mention in page 5 that the gradient on the subset selected by SSP is well aligned with the gradient on the full batch. Did you try to actually measure this alignment, e.g. in terms of cosine similarity. It could be interesting to compare it to e.g. a random subset and $\mathbb{B}^{-}$.

## Why does ESAM outperform SAM?

In some of your experiments ESAM outperforms SAM in accuracy. Do you have ideas for why this could be happening? 

## ImageNet results

One concern I have with the paper is that the original SAM paper [1] seems to report better results for SAM in some of the same settings than you do. In particular, on ImageNet in Table 2 you report 76.7 for ResNet-50 with SAM and 77.05 with ESAM. However, [1] reports 77.5 with SAM in Table 2. Similarly, their results for ResNet-101 are better, and the results for PyramidNet on CIFAR are better.

Is there a difference in the setting you use? 

## References

[1] Sharpness-Aware Minimization for Efficiently Improving Generalization
Pierre Foret, Ariel Kleiner, Hossein Mobahi, Behnam Neyshabur
summary_of_the_review: Overall, this is a very nice paper. My only concern is about the results relative to the original SAM paper. If the authors address this concern in the rebuttal, I am happy to recommend this paper for acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Paper proposes techniques to improve the efficiency of Sharpness aware minimization method. They are Stochastic Weight Perturbation (Select subset of the parameters at any step) and Sharpness-sensitive Data Selection.  Results demonstrates efficacy over SAM at small batch sizes on multiple models.


main_review: Strengths
+ simple modifications that improves over existing SAM results
+ Wide range of baselines

Weakness
- lack of ablations across batch size (especially with the data-selection algorithm)                          

== On cost of SAM ==
Cost of SAM is mentioned as 100% in the text of the paper. Typical SAM application when batch sizes are larger, involve data parallel training across a flock of GPUs. The efficient version of SAM (that works in practice) do not communicate the gradients across the cores, and hence cost is not exactly 100% but lower. It would be good to know results against this standard version of SAM.

== Hyperparameter tuning == 
It would be ideal if the authors normalized for the training cost. What would be the accuracy of SGD with 40% more epochs, with appropriate tuning of learning rate schedules?

== Choice of gamma and batch size==
Is the gamma=0.5 mean 50% of examples in a batch are discarded from the gradient step? This is quite fascinating that this method was able to improve accuracy while discarding half the batch! It would be extremely valuable to know (see comment on cost of SAM) to much larger batches (for example 2048 for CIFAR10/100) if this continues to remain true.

summary_of_the_review: Modifications are simple and efficient in practice, and may improve the practical usage of SAM across workloads.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper investigates the crucial efficiency issue of the sharpness-aware minimizer (SAM). SAM improves the generalization of DNNs but results in a double training time compared to vanilla training. By analyzing the min-max procedure of SAM, the authors observe the computational redundancy and then propose a method ESAM to improve the efficiency from the data and parameters perspectives. The authors argue that SAM can be approximated properly with fewer computations. Empirical results show that ESAM can reduce the extra training time in CIFAR10/100 and ImageNet datasets with improved accuracy compared to SAM. 
main_review: Pros: 
1.	The proposed approach is well-motivated and practical in real-world applications. The enhanced efficiency does not lead to a degradation of accuracy. The experimental setup is satisfactory and convincing. The proposed method is verified on multiple datasets to show the effectiveness. The ablation studies and the supplementary experiments in the appendix are appropriate.
2.	The idea of picking representative samples for updating parameters is interesting and effective. Why not use the $\alpha$ directly as the hyperparameter for data selection?
3.	The paper is well-organized. Figures 3 and 4 are good for demonstrating. 
Cons:
1.	It is mentioned that the computations of SWP slightly increase in the deeper DNNs. What is the exact impact of using SWP in large-scale deep neural networks? 
2.	The explanation of SWP's computation in Section 2.2 is not clear and may be made shorter. For example, L8 "Decreasing $\beta$ ... may degrade" is superfluous; The claimed positive correlation between saved computations with $\beta$ is vague and should be more precise. 
3.	The gaussian perturbations used in Figure 4 are not representative enough. There is no difference between the x-axis perturbation and the y-axis perturbation. Consider visualizing the loss landscape with the adversarial perturbations. 
Additional comments:
1.	Better to use "hyperparameters of SGD" to replace "parameters of SGD" in sec 3.1 line 6. 
2.	Better to add "(Training speed 140.3% vs. SAM 100%)" for the efficiency comparison in section 3.1 P3L4 "(140.3% vs. SAM 100%)".

summary_of_the_review: This work raises an important question of SAM where the improved training algorithm requires doubled training cost. This paper addresses and improves the efficiency drawback of SAM. The proposed approach is well-motivated and has been verified to improve the efficiency and accuracy of SAM. The authors also provide codes for reproducibility. However, the degradation of SWP's effectiveness in large-scale neural networks, as well as the justifications for the improved accuracy contributed by SWP and SDS, have not been clearly addressed in this paper. I expect the author can answer the points above, and then I can adjust my final score.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a method called ESAM for improving the efficiency of SAM. ESAM has two components: SWP and SDS. SWS accelerates the estimation of $\epsilon$ via random sampling a subset of parameters in backpropagation. SDS further improves the efficiency via sampling a subset of data points that is enough for calculating the upper bound of $L$. Combining these two techniques, they achieve a speed improvement for SAM while yielding comparable or even better performance.
main_review: Strengths:
* This paper is well written and easy to follow.
* While SAM has proven to be helpful for many applications, this technique can be helpful in practice.

Weakness:
*  For SWP, is the gradient mask the same for the samples in one batch, or different gradient masks are applied for each sample in the batch?
* For SDS, an interesting baseline to be compared is estimating $\epsilon$ with a subset of randomly sampled data, which could match the origin $\epsilon$ in terms of empirical estimation.
*  Recent paper [1] points that SAM can be very useful in ViT and MLP-Mixer. It would be more convincing if the author could show ESAM can also work on those two architectures.

Detailed comments:
* First row in Page 8,  "As shwon in Table 3, SWP improves". shwon -> shown

[1] Chen, Xiangning, Cho-Jui Hsieh, and Boqing Gong. "When Vision Transformers Outperform ResNets without Pretraining or Strong Data Augmentations." arXiv preprint arXiv:2106.01548 (2021).
summary_of_the_review: This paper is the pioneer in exploring a more efficient SAM method. This technique can be useful since SAM has been proven to be useful for many networks. Therefore, I would recommend this paper as weak accept. I would be more convincing the practicability of this paper if the authors can show the proposed ESAM works on ViT and MLP mixer, where the SAM has been proven to be very effective.
review_title: null

Meta-review: This paper focuses on improving the efficiency of sharpness-aware minimization method for training neural networks. The proposals are stochastic weight perturbation, namely selecting subset of the parameters at any step, and sharpness-sensitive data selection. The philosophy behind sounds quite interesting to me, namely, sharpness-aware minimizer can be approximated properly with fewer computations after analyzing the min-max procedure. This philosophy leads to a novel algorithm design I have never seen.

The clarity and novelty are clearly above the bar of ICLR. While the reviewers had some concerns on the significance, the authors did a particularly good job in their rebuttal. Thus, all of us have agreed to accept this paper for publication! Please include the additional experimental results in the next version.
Decision: Accept (Poster)