Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents new training methods called RST and G-RST to extend the geometry inspired training method SAM and improve its computational efficiency.
Based on randomized gradient boosting RST randomly selects between SGD and SAM with probability based on parameterized Bernoulli distribution.
The authors explore different parameterization schemes for such a scheduling function and analyze their effect on computations and performance trade-off.
The authors also develop RST’s convergence properties for non-convex stochastic cases where the classes of objective functions are smooth and strongly convex.
The paper evaluates RST and G-RST on multiple image classification tasks showing that the proposed methods can save 50% computations while performing on-par or even better than the original SAM.

strength_and_weaknesses: - The paper is written in a very clear and professional way.
- The paper is very solid with balanced views and analysis.
- The paper provides convergence analysis which somehow lacks in SAM literature.
- The resulting algorithm is simple and effective.
- I literally didn’t find any flaws in the paper but thought adding more experiments on large scale and different domains could make the paper even stronger.

clarity,_quality,_novelty_and_reproducibility: This paper is very well written overall. The paper is also original to some extent in the sense that although there exist SAM variants attempting to improve on SAM’s computational aspect, unlike most of these works this work develops based on matured techniques and optimisation characteristics and provides very well thought-out and reliable results.

summary_of_the_review: Highly recommended for interested readers on SAM literature.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Recently, sharpness-aware training such as SAM has drawn large attention, because of its provably guarantee its significant performance improvement. However, SAM requires a huge additional computation cost, and it is not easy to adopt SAM on a large-scale model or a real-time analysis system. This paper provides an efficient computation for SAM, RST, and G-RST. They adopt randomized sharpness-aware training. The idea is simple, and it works well.
strength_and_weaknesses: Strength points
1. This paper aims to improve the efficiency of SAM, the important problems.
2. This paper raises interesting ideas such as randomized sharpness and its general extension G-RST that adjust regularization degree freely for any scheduling function.

Weakness points
1. The concept of this paper is similar to the other papers [1]. This paper should cite and discuss the difference between this paper and another paper.
2. This paper is necessary to include diverse related works and efficient computation of SAM. For example, there are three research works, [2], [3], [4]. This paper only cites and discusses the [4]. However, there is no experimental comparison with [4]. I suggest that this paper should discuss and compare the performance with [2], [3], [4].

[1] Zhao, Yang, Hao Zhang, and Xiuyuan Hu. "SS-SAM: Stochastic Scheduled Sharpness-Aware Minimization for Efficiently Training Deep Neural Networks." arXiv preprint arXiv:2203.09962 (2022).
[2] Du, Jiawei, et al. "Sharpness-Aware Training for Free." arXiv preprint arXiv:2205.14083 (2022).
[3] Liu, Yong, et al. "Towards efficient and scalable sharpness-aware minimization." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
[4] Du, Jiawei, et al. "Efficient sharpness-aware minimization for improved training of neural networks." arXiv preprint arXiv:2110.03141 (2021).
clarity,_quality,_novelty_and_reproducibility: This paper is well-written, and it is easy to follow the contents.

There is an implementation code, and it is easy to reproduce.
summary_of_the_review: This paper raises an interesting concept, but similar ideas were already suggested in other venues. 

Discussion and additional experimental comparisons are necessary.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Sharpness-aware minimisation (SAM) has been shown powerful to train high-performance deep learning models, but it also incurs at least double computational cost due to the extra back-propagation for the sharpness estimation. To improve the efficiency of the vanilla SAM, this paper proposes an training scheme, dubbed Randomized Sharpness-Aware Training (RST). RST performs a Bernoulli trial at each iteration to choose randomly from base algorithms (SGD) and SAM. The probabilities of Bernoulli trials at each time are  determined by a predefined scheduling function $p(t)$. The average extra time is reduced from 1 to $\sum p(t) / T$.
strength_and_weaknesses: Strength:
- The proposed RST can reduce the extra computational cost of the vanilla SAM from 1 to $\sum p(t) / T$ in average, and preserves similarly good performances.
- The paper is easy to follow.

Weakness:
- The novelty of this paper is my only concern. The idea of applying SAM to a subset of parameters or iterations has been explored in several papers [Mi et al. 2022, Liu et al. 2022, Du et al. 2022]. For example, Liu et al. 2022 propose to only periodically calculate the inner gradient ascent across the training iterations; Liu et al. 2022 and Du et al. 2022 propose to select a subset of parameters to calculate the inner gradients in each iterations. Over expectations, randomly selecting a subset of parameters in iterations is same as randomly select interactions. And the later can be considered as a special case of the former --- alternatively masking out all parameters. 


Mi et al. 2022, Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approach

Du et al. 2022, Efficient Sharpness-aware Minimization for Improved Training of Neural Networks

Liu et al. 2022, Towards Efficient and Scalable Sharpness-Aware Minimization
clarity,_quality,_novelty_and_reproducibility: Please see the weakness part. 
summary_of_the_review: The paper is easy to follow. The empirical experiments are enough to support the arguments. However, the novelty of this paper is the concern. The idea of selecting a subset of parameters or iterations for implementing SAM has been explored in several recent works.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper targets at improving the efficiency of sharpness aware optimizer (SAM) by assigning a probability where a normal SGD optimizer is used. The decision is based on Bernoulli trial. The authors further propose a general framework to make the proposed optimizer schedule usable for different architectures. The idea is intuitive and clearly expressed. The proposed method is simple. This would not be an issue if it is extremely effective. However, the paper has missed many efficient SAM baselines such as GSAM [1] and SAF [2]. Besides, the experiments on large dataset ImageNet is not solid. The selected baseline models are different from the experiments on CIFAR-10 and the are not representative enough.


[1] Juntang Zhuang, Boqing Gong, Liangzhe Yuan, Yin Cui, Hartwig Adam, Nicha Dvornek, Sekhar Tatikonda, James Duncan, and Ting Liu. Surrogate gap minimization improves sharpness-aware training.

[2] Du, Jiawei, et al. "Sharpness-Aware Training for Free." arXiv preprint arXiv:2205.14083 (2022).
strength_and_weaknesses: Strength:
1. The motivation of the proposed method is clear. SAM is indeed suffering from low computational efficiency.
2. The proposed method is clearly presented. The reproducibility is not an issue.

Weakness:
1. The proposed algorithm is intuitive and not novel enough. It is adding a probability to replace some of the SAM steps with normal SGD optimization steps. This would not be an issue if the proposed method is verified to be super-effective. However,
2. The experiments do not show the superiority of the proposed method. The paper has missed some relevant baselines as mentioned in the summary section. Also, I was confused that why the selected architectures are different on CIFAR and ImageNet dataset?
clarity,_quality,_novelty_and_reproducibility: The paper is well written. The proposed method is clearly presented. However, the proposed method is not novel enough and the experiments are not solid enough.
summary_of_the_review: In summary, the proposed method is not well supported by the presented experiments mainly due to two points:

1. The missing baselines on efficient SAM algorithms and 
2. The missing architectures on ImageNet dataset.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The main contribution of this paper is to accelerate SAM. This is a worthy endeavor. 

This paper was perceived to be borderline and so as the AC, I spent a significant amount of time discussing it with the reviewers and looking through the paper. Unfortunately, my recommendation is a reject. This is because the main contribution is very similar to LookSAM (Liu et al., 2022), which is to do SAM once every x iterations. The remainder of the paper is to explain why the method works theoretically; which is good. 

I understand that it is impossible to compare the method to recently proposed variants of SAM (which we published a few days before the ICLR deadline). However, the main idea overlaps significantly with LookSAM, so this has an adverse effect on the novelty of the whole contribution. 
justification_for_why_not_higher_score: I have explained this above. 
justification_for_why_not_lower_score: I cannot go any lower.
Decision: Reject