Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: It is a good modification to the SAM method, which may further improve the performance.
main_review: Strengths
1. This submission provides a more reasonable way to find the flatten minima rather than the heuristic way utilized in SAM. 
2. With negligible additional computation, this method improves the generalization performance.

Weaknesses
1. It is well known that SAM-based methods may lose their superior on some more complex DNN architecture, such as EffeicientNet for CNN-based model, Swin for vision transformer. This submission still performs the experiments on the vanilla architectures as the SAM did. We may wish the more exciting results on the advanced DNNs.
2. Eq.5 seems to do not fit the updating rule provided in this submission.  Although the authors alert the potential caveat for optimizing $f_p + \lambda h$, the proposed GSAM still aims at minimizing this objective. A minor wise modification is to exclude the component in the gradient of the second term, which may degenerate the values of the first term. Hence, the convergence guarantee may require more efforts to deal with the proposed modification; however, no more discussions are given.
3. The convergence guarantee is a classical conclusion for non-convex stochastic optimization. If the authors cannot provide the details that show how to treat $f_p$ technically, there is no need to provide the guarantee in the main part, which cannot offer any superiority of GSAM for readers. So does the generalization guarantee, which is a direct application of general PAC-Bayesian bound. Thm. 5.3  is more interesting than the previous ones.

summary_of_the_review: It is a good submission that provides an interesting way to improve the updating rule of SAM. However, some modifications for the technical part of this submission are still needed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a new method for improving generalization in deep learning called GSAM which builds upon and extends recent works on adversarial weight perturbation and sharpness-aware minimization (SAM). The paper argues that minimizing the loss function used in SAM (which replaces the loss by its maximum in a neighbourhood) does not always lead to flat minima. Based on this insight, the paper puts forward the following contributions:
1. A new loss function which does not suffer from these problems, along with a new algorithm derived from the loss function
2. Theoretical results (generalization bounds and convergence theorems) supporting the algorithm and new loss function
3. An extensive numerical evaluation, showing that the proposed GSAM method leads to significant gains over SAM

main_review: ### Strengths
* The paper is well written, and I like the clear motivations and illustrations shown in Figure 1-3. 
* The proposed method is simple, elegant and well-motivated yet appears to work extremely well across a wide range of experiments.  

### Weaknesses
* The main optimization objective presented at the beginning of Section 4.1 is not entirely clear to me and seems not mathematically rigorous. The quantity (f_p(w), h(w)) is a vector, so is this to be understood as a multi-objective optimization task?  What is the definition of a solution of this problem?  
* Later it is mentioned that GSAM solves min(f_p, h), but it is not clear whether the convergence theorem actually shows that GSAM finds a "solution" of this objective.  Instead of the decomposition of the gradient into two components, one could for example consider an algorithm which directly computes gradients of the min(f_p, h)-function.
* I am a bit unsure whether "surrogate gap" is a good name for h(w). Isn't h(w) simply the sharpness as considered eg. in Foret et al?
* An important aspect of SAM is what is called "m-sharpness" in Foret et al, i.e., computing the perturbed loss on mini batches.   I could not find any mentioning of the "m"-parameter in the paper, and since the performance of SAM greatly depends on it, it would be great to confirm that a reasonable value is used in this work. 
* Overall, I am not sure whether the paper really needs to include the theoretical results presented in Section 5.2.  They appear very hard to digest for the reader and I am unsure whether anything can be concluded from it. I believe this is a strong paper even without the theoretical analysis.
summary_of_the_review: The paper has some smaller weaknesses (see main review) but if these are addressed in the rebuttal I am inclined to increase my score and recommend acceptance of the paper.  Overall, the work proposes a simple and elegant method, which is well motivated and gives state-of-the-art performance in an extensive numerical evaluation.  Therefore, I believe it will be of great interest to the ICLR community.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper builds on the main idea of SAM. Namely, the authors noticed that minimizing the perturbed loss $f_p$ is insufficient for guaranteeing flat minima. The authors then defined the notion of surrogate gap $h$, which they propose to minimize together with the perturbed loss. By using GSAM which consists of two gradient descent/ascent steps that include the minimizing $f_p$ and $h$, the authors show theoretically and experimentally that this results in a model with better generalization than SAM.
main_review: Strengths

+ I really like the idea of this paper that notices that both $f_p$ and $h$ are crucial in boosting's SAM's success. 

+ The empirical findings are nice to see. 

+ The theoretical findings are also nice to have, but perhaps not as convincing as the empirical findings. 

Weaknesses

- I perused some of the proofs of the theoretical results. For the most part, they look fine. However, the authors can be rather sloppy at times. One cannot use $\approx$ in a proof without quantifying what it means. For example, in the transition from (39) to (40), the authors justify the change of $\sin\theta_t$ to $\tan\theta_t$ by mentioning that $\theta_t$ is small. I believe that the authors would do better if they quantified the remainder terms carefully. 

- One other deficiency is that the performance plots/tables do not include error bars. 
summary_of_the_review: The idea is nice. The empirical performances look promising and the theoretically analysis generally sound (though sloppy). I recommend a weak accept.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper,the authors proposed a new method for sharpness aware training. The idea is to note a surrogate gap as a measure of sharpness, which motivates to minimize the perturbed loss and the surrogate gap simultaneously. The authors proposed to use a gradient descent to minimize the perturbed loss and gradient ascent in an orthogonal direction to minimize the surrogate gap. The use of the orthogonal direction is to avoid the influence of the perturbed loss in the ascent step. The paper gives theoretical results to show the connection between the proposed surrogate gap and the sharpness. Convergence rates and generalization bounds are also presented. Experimental results are reported which seem to be convincing.
main_review: The idea of minimizing the surrogate gap while maintaining the perturbed loss with gradient ascent along an orthogonal direction is interesting. The experimental results are comprehensive and seem to be convincing. Here are some issues.

- In the proof of Thm 5.1, the smoothness parameter is $L_p=L+L^2\rho/\epsilon$. Note $\epsilon$ is often very small. Indeed, the default choice of $\epsilon$ is $10^{-12}$. Then the convergence bound would be very crude since $L_p$ is very large, which may not apply well in practice.

- Eq (40) and (41) use inaccurate expressions such as $\approx$. Therefore, these inequalities are not precise. The proof based on these inequalities are therefore not rigorous. Furthermore, in the proof there are some $\beta_{\min}$ and $\beta_{\max}$. These quantities are unknown in practice. They depend on the practical implementations of the algorithm and the authors do not give an upper bound on $\beta_{\max}/\beta_{\min}$. If the term $\beta_{\max}/\beta_{\min}$ is large, the derived bound can be crude. The authors should also be careful about the expectation operators. There are missing expectation operators for Eq (37)-(40), Eq (43)-(54).

- The proof of Thm 5.3 is also not rigorous since the authors use several inaccurate inequalities, e.g. ($\approx$ in (70) and (71))

- In Corollary 5.2.1, the authors tried to show that GSAM can find a smoother minimizer by involving the surrogate gap in the generalization bound. However, this actually does not convey this message. The underlying reason is that both sides of eq (7) involve $C$. Therefore, Corollary 5.2.1 actually gives a bound on the risk in terms of the perturbed training loss if one removes $C$ from both sides: the perturbed training loss is small then the model has a small risk. It does not show the benefit of a small surrogate gap in achieving a better generalization, which is a claim of the paper.

Typos:

$g^{(t)}$ in eq (37) should be $g^{(t)}_\bot$

there are two $++$ in eq (49)

there is a mixing $E_x$ in Thm 5.2
summary_of_the_review: The idea of minimizing the surrogate gap and the perturbed loss simultaneously is interesting. The experimental results are sufficient. However, the theoretical analysis is not quite convincing.
review_title: null

Meta-review: The paper proposes an interesting and well-motivated improvement of Sharpness Aware Minimization.  Overall the AC and reviewers are satisfied by the author feedback in improving the solidity and rigor of the theoretical results. 

The points made by the authors in response to the reviewers initial concerns are essential, especially those regarding interpretation of Corollary 5.2.1, making the proofs rigorous, and fixing the potential for crude convergence bounds. It is therefore critical that the authors incorporate them into their manuscript.
Decision: Accept (Poster)