Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Sharpness-aware minimization (SAM), which searches for flat minima by min-max optimization, has been shown to be useful in improving model generalization. However, SAM requires two gradient evaluations per iteration which makes it cost twice more expensive as SGD. One way to reduce this cost is mixing SAM with SGD in an adaptive manner. This paper proposes an adaptation based on the stochastic gradient’s norm.  The main idea is to keep an adaptive threshold (updating it per iterate), and then compare the gradient norm with this threshold: if it is bigger take a SAM update, otherwise take a SGD update.  They show the convergence of their proposed method. 

strength_and_weaknesses: Strength: Adaptive thresholding is a new approach to mixing SAM and SGD/ERM 
Weaknesses: choosing this threshold is not well justified. 
clarity,_quality,_novelty_and_reproducibility: The paper is well written and easy to follow and understand 
summary_of_the_review: Comments: 
1- Assuming that the stochastic gradient follows a normal distribution is not well justified. For example, there exists research in the literature that shows this norm follows heavy-tailed distribution and not normal distribution. 

2- In your analysis, the adaptivity of the threshold doesn’t show up and the analysis is very similar to the method which takes SAM step randomly. Specifically, in the deterministic setting, there is no stochastic gradient and hence normal distribution assumption is incorrect. Besides in the empirical result, there is no significant difference between AE-SAM and SS-SAM. 

3- It is not explained in the paper why using AE-SAM can improve the robustness against label noises.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this work, the authors propose a new variant of SAM optimizer, called AE-SAM by mixing SAM and SGD with a carefully designed condition. Moreover, the theoretical convergence of the proposed algorithm is also provided. Preliminary experiments demonstrate the efficacy of the proposed approach. 
strength_and_weaknesses: The authors propose a new variant of the SAM optimizer. Preliminary experiments demonstrate its effectiveness. However, before this work is accepted, several concerns should be resolved:  
(i)	The proposed AE-SAM falls into the scope of mixing SGD and SAM. Several existing works adopt a similar idea to improve the efficacy of SAM, including SS-SAM, and Gradient penalized SAM (Zhao et al, ICML 2022). The idea is not new, which limits the novelty of the proposed approach.
(ii)	About the practicality. Several hyperparameters are introduced in AE-SAM. In general, SAM-type methods are sensitive with respect to the perturbation parameters. More hyperparameters may make the tuning process more difficult. 
(iii)	About the theory. In general, mini-batch SGD could achieve the linear speedup property with respect to the mini-batch size. (see Efficient Mini-batch Training for Stochastic Optimization, KDD). However, in Theorem 3.4, the linear speedup property is missing. In addition, can the author prove that the proposed AE-SAM theoretically outperforms randomized SAM (SS-SAM)? 
(iv)	About the sensitivity of the adaptive policy. The proposed adaptive policy is highly related to mini-batch size. When training a deep neural network with a different mini-batch size, it shows different gradient variances. (see Not All Layers Are Equal: A Layer-Wise Adaptive Approach Toward Large-Scale DNN Training, WWW2022). The authors may conduct more experiments to show the consistency/sensitivity of the adaptive policy with respect to mini-batch size. 
(v)	More experiments verification on advanced neural network architecture are needed, such as vision transformer (such as Deit). 


clarity,_quality,_novelty_and_reproducibility: This paper is well written. However, mixing SGD and SAM is not new.  
summary_of_the_review: See the comments above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed an adapt scheme for sharpness-aware minimization (SAM), in which the algorithm simply perform ERM when landscape is flat and perform SAM which landscape is sharp. This method is based on couple of approximation strategies. The author also establish the theoretical guarantee for their proposed algorithm. Some empirical results are provided to verify the effectiveness of the proposed algorithm.
strength_and_weaknesses: Strength:
(1) The proposed method is new and intuitive
(2) The effectiveness of proposed algorithm is supported by both theoretical and empirical evidence


Weakness:
(1) It is not clear how the author come up with (4) (5) for mean and variance derivation. It seems that this formulation enables establishing convergence result but it is hard to see the intuition behind this formulation.
clarity,_quality,_novelty_and_reproducibility: As I mentioned in the "weakness" in the last section, the author is suggested to clarify how and why they choose the formulation in (4) and (5) for adaptive SAM.
summary_of_the_review: Overall this paper is well-written and easy to follow. The high level idea of the algorithm proposed in this paper is intuitive and nature and should be interesting to the community. Since i didn't see obvious weakness in both theoretical and empirical results, I will give the score of 6 for now.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents a framework for reducing the computational cost of SAM algorithm. The latter is state-of-the-art optimizer, but the computational cost of it is almost twice as that of SGD. The paper proposes a simple and interesting strategy for dynamically switching between SGD and SAM through the optimization trajectory, and provides a theoretical convergence guarantee for it. The rationale behind the idea is that SAM is not much useful at places that the landscape is already flat. In such regimes, one can apply SGD to navigate in the loss landscape. However, SAM gets critical in regions when the landscape is sharp, where it is needed to take the trajectory in directions that reduces this sharpness. The notation of sharpness used by the paper is based on norm of the gradient, which is a sound proxy for measuring sharpness and can be related to the notion of sharpness used by SAM. However, since the estimate of gradient norm is computed from minibatches, the estimates are noisy. In order to reduce this noise, the paper refines the estimate by exponential moving average.

In my opinion, the paper is very well-written, has a good balance between theory and experiments, presents a sound, interesting and effective idea, and has a great coverage of related works. Some concerns were raised by the reviewers, but the authors did a great job in handling those questions, as a result of which, the reviewers increased their score. Among these was an issue related to the convergence proof (not considering adaptivity of the threshold value), which authors clarified and modified their proof in Section 3.3 to address that. Overall, the paper is borderline accept, but I think it has interesting contributions to be an accept. Please make sure the final version of the paper addresses the clarity issues raised by the reviewers.

justification_for_why_not_higher_score: While the paper has new and interesting ideas with promising results, the ideas are not deep enough to be considered for a spotlight or oral.
justification_for_why_not_lower_score: The scores are overall on the accept side, the authors did a great job with rebuttal (which resulted in increase of score by reviewers).
Decision: Accept: poster