Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper proposes to modify the Sharpness-Aware Minimization (SAM) algorithm to include a random perturbation before taking the gradient ascept step. The paper presents some justifications to why it could be beneficial, based on the smoothing argument. The experimental results show the effectiveness of the proposed method.

strengths_and_weaknesses: - **Originality.** Low. The idea of adding noise before taking the gradient ascent step of SAM is very simple.
- **Quality.** Medium. The experimental evaluation of R-SAM is quite convincing. The findings that SAM/R-SAM can quite noticeably boost the OOD performance (ImageNet-R, ImageNet-C, etc) is also nice. On the other hand, the motivation of the proposed method requires significant improvements (see below).
- **Clarity.** Medium. Most of the content is clear.
- **Significance.** Overall, I appreciate the empirical effectiveness of the proposed method which could be of interest to practitioners. However, the paper requires multiple improvements listed below.

Things that need improvement:
- A crucial baseline, Bisla et al. (AISTATS’22), is cited but isn’t compared to or discussed in sufficient depth. A comparison to their method (which is **not** using worst-case perturbations) would be beneficial, especially since they leverage random perturbations with the same covariance as suggested in this paper. I wonder if it’s key to the performance of this method.
- *“That means the gradient of smoothed weight $w + \delta_0$ can better maximize the loss value in a large region.”* – But do we really need such large weight perturbations? For $\rho=1.0$, the training loss is increased to $\approx 6$ (Fig. 2) which is even higher than the loss of a random classifier ($\approx ln(100) \approx 4.6$).
- Fig. 1 and 2: is it done for test-time BatchNorm? If yes, then it should be remade for training-time BatchNorm or for a network trained with a different normalization method. In practice, the difference between sharpness estimates of test-time and training-time BatchNorm is very significant (see, e.g., https://arxiv.org/abs/2206.06232).
- The loss values in Fig. 2 are checked for the algorithm described in Eq. (6) while the proposed algorithm of R-SAM presented in Eq. (7) differs from Eq. (6) by the presence of the projection and a special covariance matrix of the noise. In order to properly motivate R-SAM (and not some other possible variation), one has to perform the experiment in Fig. 2 specifically for R-SAM. Intuitively, the projection step can make a lot of difference and I’m not sure if the algorithm in Eq. (7) will lead to a more accurate maximizer compared to the vanilla SAM.
- In addition, the role of the noise covariance is not sufficiently discussed. It’s unclear whether it’s crucial or not. For such a simple proposed idea, I’d expect much more rigorous ablation study for the key elements of the proposed method.
- The original SAM paper already pointed out that multiple steps of projected gradient ascent aren’t helpful (see Table 11 in https://arxiv.org/abs/2010.01412). That would be a more direct way to more accurately solve the inner maximization.
- Error bars are necessary to claim that the observed improvement is indeed significant.


In multiple places throughout the paper, the writing is not precise and some claims don’t seem to be justified:
- “However, the model usually locates in the sharp minima [31] where the unstable gradient, to a large extent, makes one-step gradient ascent performs poorly.” – This statement is surprising given that SAM aims to minimize sharpness and converge to a flat region. Moreover, this sentence leads to the impression that [31] demonstrated that this is somehow the case, although [31] has nothing to do with this.
- “Therefore, how to improve the approximation of the inner maximization procedure and obtain a more aggressive weight is an important problem” – What is meant by the “aggressive weight”?
- “However, the loss landscape of neural network is usually sharp and non-linear [30, 34].” – Surprising to see statements like this as the loss landscape of a **linear** model is also non-linear and may be considered sharp depending on the data covariance.
- “we observe that models can still locate in a sharp region, leading to poor performance on one-step gradient ascent.” – This statement isn’t complemented by sharpness measurements.
- “we are able to apply the one-step gradient ascent on the smoothed weights for a much more accurate measurement of the inner maximization” – There is no evidence to this in the paper. Fig. 2 is done for a different algorithm (see my concern about Eq. (6) vs. Eq. (7)).
- Eq. (2): there should be a division by the gradient norm, not multiplication.

questions: - Was a grid search done for SAM (e.g., Table 1 and 2)? And how are the hyperparameters tuned for each dataset for the proposed method, R-SAM?

limitations: The proposed method has three hyperparameters: the perturbation radius $\rho$, $\gamma$ for the noise standard deviation of the noise step, and $\lambda$ for the step size of the gradient. In comparison, SAM has only one hyperparameter. The hyperparameter sensitivity presented in **4.6 Sensitivity Analysis** is not very convincing as it shows only one-dimensional grid search (while keeping the other two dimensions fixed to the optimal values) and, for example, the range of $\rho$ presented in Fig. 3 is in fact quite narrow (from 1.2 to 1.6).

---

**Update after the rebuttal**
*Overall, I'd say that the improvements coming from R-SAM are consistent across multiple settings which makes the proposed method practically useful. However, the core idea of simply adding random noise (although with specific covariance) before the ascent step of SAM is extremely simple and lacks novelty. Moreover, I think the paper could've been implemented more carefully in various aspects (mentioned above) such as careful ablation studies and better motivation of the proposed method. I strongly feel that since the proposed method is so simple, the paper has to compensate it with very careful and rigorous empirical validations. I increase my score to 4 in light of the improvements of the paper presented during the rebuttal but still feel that the paper is below the acceptance bar.*
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: Sharpness-Aware Minimization (SAM) is an emerging deep learning technology in recent years, which aims to find parameters lie in a "flat" region of the energy landscape. Its purpose can be formulated as a minimax problem as shown in (4): A parameter vector w_adv is "optimal" in the sense of SAM if the loss of worst solution in the neighborhood of w_adv is minimized. The vanilla SAM solved this problem by approximating the landscape around a solution w as a linear function. This paper investigates this approximation thoroughly by experimental and theoretical analysis, and shows that this approximation is possibly poor in applications. Inspired by recent works in the field of adversarial training, authors improve the vanilla by introducing noise wo weight parameters.

strengths_and_weaknesses: Strengths:

Both theoretical and numerical analysis is thorough. The idea is simple and easy-to-use. The paper is clearly written and the contribution is significant.

Weakness:

See Questions.
questions: 1) In Fig. 1, the "flatness" of w is measured by the cosine value between g(w) and g(w+rho*g(w)/||g(w)||). But is it a suitable measure for all w? If w is close to a local minimum and g(w) is close to zero, then directions of both g(w) and g(w+rho*g(w)/||g(w)||) are "random", and the cosine value could be small even in a very flat region. The detailed settings of experiments in Fig. 1 are not provided. If the w is obtained after training, we need to consider this case. 

2) There are some typos in figures. In Fig. 1, \hat g(w) should be g(\hat w). In Fig. 2, L(w+rho*g(w)/g(w)) should be g(w+rho*g(w)/||g(w)||).

3) Do authors miss the mathematical expectation notation somewhere? For example, in Theorem 1, L_s(w)=L(w+delta_0) or E[L(w+delta_0)]? (If it is the former one, we cannot say that L_s(w) must have a smaller Lipschitz constant.)

4) Baseline methods include SGD (and modified SGD) without SAM and existing SAM methods. Certainly it is impossible to compare the proposed method with all regularization techniques, but it would be better to consider some advanced non-SAM strategies here.

5) As stated by authors, "deep learning is usually sensitive to hyper-parameters". So the small improvement of the accuracy cannot strongly prove that RSAM significantly outperforms the other SAM methods. I suggest to add some numerical results to show that RSAM can find "flatter areas", which is more meaningful than 0.1% improvement in accuracy.
limitations: Authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work analyzes the approximation quality of the inner maximization term in Sharpness-Aware Minimization (SAM). It is shown that the unstable gradients can hurt the inner maximization. A novel method called Random Sharpness-Aware Minimization (R-SAM) is proposed that smooths the landscape, enabling a better approximation of the inner maximization. This is done by the addition of Gaussian Noise before the computation of the inner maximization term. The results are shown on CIFAR-10, CIFAR-100, and ImageNet datasets. R-SAM improves upon the performance of SAM in most settings.
strengths_and_weaknesses: Strengths:
1. The analysis of the effect of Gaussian noise on the perturbed loss-value (Figure 2) is interesting. The analysis of approximation of gradient in SAM has not been studied and is a good direction.
2. The proposed method, R-SAM, is simple, has no additional cost compared to SAM, and allows for stable training with large $\rho$.
3. The results on combining G-SAM with R-SAM, resulting in R-GSAM, show that the R-SAM can be integrated with other variants of SAM.

Weaknesses:
1. Relationship between stable gradient and approximation of inner maximization term is not very clear. Also, the term stable gradient is not well-defined. If a stable gradient were the goal, then $\rho$ = 0 (SGD) or a low $\rho$ would lead to the most stable gradient according to the cosine definition. However, it is observed that a very low $\rho$ value does not lead to a significant gain in accuracy over SGD.
3. LPF-SGD [1] is a recently proposed method that aims to reach flat optima. The algorithm of LPF-SGD also adds Gaussian Noise to the parameters with the variance equal to the norm of filter weights, similar to R-SAM.  
4. The increase in accuracy of R-SAM in comparison to SAM is minimal in most settings (Table 1, Table 2, Figure 3a).

[1] Devansh Bisla, Jing Wang, and Anna Choromanska. Low-pass filtering sgd for recovering flat optima in the deep learning optimization landscape. arXiv preprint arXiv:2201.08025, 2022.
questions: 1. In Figures 1 and 2, it would be clearer if the authors also mentioned the model's accuracy in a Table. For instance, $\rho$ = 1.0 leads to the max loss value in Figure 2. Is that observed in the performance as well? 
(Also, why is the scale of the rho value in both the figures very different?)

2. The experimental details of the CIFAR-100 experiments, including the $\rho$ value, seem to be missing. A note comparing the $\rho$ value with SAM and R-SAM can be added. It is to be observed whether training with high $\rho$ is also stable for ResNets. With ViTs (Table 6), the optimal $\rho$ value is higher for R-SAM.

3. The details regarding the value of $\gamma$ have not been mentioned. A sensitivity analysis of the value of $\gamma$ can also be added.

4. Question 3c of the checklist is answered as, "It would be too computationally expensive for us. We conduct all the experiments at least three times and report the average value." This is a bit confusing. I just wanted to clarify whether the authors have run all the experiments three times. This is significant because the increase with R-SAM is small in many experiments and within the error bound (as mentioned in SAM).

4. In the results section, it is unclear why the authors are discussing the improvement of the original SAM over SGD+M and AdamW (L233-238, L246-250, L261-264). These results have been discussed in the SAM work.  

5. The method proposed by Chaudhari et al. is called Entropy-SGD and not "Energy-SGD," as mentioned by the authors. (L53)
limitations: The limitations of the work is not mentioned. This work has no potential negative societal impact.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors argue that the approximation of SAM's perturbation step, which requires solving an inner maximization objective, is prone to unstable gradients. To improve this approximation step without adding significant additional computational steps, the authors propose adding Gaussian noise to the perturbation initialization. 
strengths_and_weaknesses: # Strengths 

* Significance: Efficient stochastic optimization is fundamental for deep learning, and sharpness-aware methods often (but not always [3]) lead to significantly better generalization performance than previous optimizers. Hence, improving methods like SAM can be very relevant to many optimizer users.
* Clarity: the paper is well-written and easy to follow. 
* Simplicity: The proposed method is simple, easy to implement, and effective for the shown experiments shown.  

# Weaknesses


While I'm open to being convinced otherwise and raising my score, I'm not fully convinced yet that the main premise of the paper -- randomly smoothening (RS) the loss function **significantly** **improves** over a non-smoothed loss function for approximating the gradient ascent step -- is true. RS seems very heuristical to me; hence, I'd appreciate either more clarification on why it always improves (if so) or more discussion on when it may not hold and what its trade-offs are. 

SAM's inner objective's gradient ascent step involves the same gradient computation as standard SGD. Yet, when using standard SGD, we typically do not additionally smooth the loss function. For example, [4] recently proposed doing so, however, with the motivation of finding flatter minima (akin to SAM) and by sampling multiple perturbations instead of just one. If RS leads to more "stable" or "accurate" gradients, why not apply it to the descent step too? For example, in Algorithm 1, "Compute SAM gradient", why do we not additionally perturb $\mathbf w_{\text{adv}}$  as well if such smoothening is always beneficial? 

Figure 1 shows Vanilla's cosine similarity between the standard and perturbed gradient decays faster than when RS gets applied. However, it is not clear to me why this matters and can't be simply resolved by re-scaling rho.

Figure 2 shows that the RS loss function increases faster in loss when rho is increased. Again, it is unclear why reaching higher losses by perturbing the weights further and further with increasing standard deviations is strictly better or important for ultimately reaching flatter (and therefore, better generalizing) minima.

The authors claim that Vanilla can not reach loss values as high as RS reaches (although this is not fully proven as rho's range ends at $\rho=1.0$). However, first perturbing the weights before approximating the sharpest loss in their neighborhood effectively moves the neighborhood ball in a worse-performing region -- it is not centered around $\mathbf w$ but at $\mathbf w + \mathbf \delta_0$. Since $\mathbf \delta_0 \neq \nabla L(\mathbf w)$ , almost surely $L(\mathbf w) < L(\mathbf w + \mathbf \delta_0)$ . Therefore, it is not surprising that an ascent step starting from $L(\mathbf w + \mathbf \delta_0)$ will reach even higher losses. 

Further, we may note that the authors generated Figure 2 using the checkpoint of the 200th epoch on CIFAR-100. Using a decent learning rate/schedule, 200 epochs are enough for training a Resnet-18/WRN to (almost) convergence on CIFAR100. Hence, it is not clear why we want to reach higher-loss regions at this point during training. 

Lastly, the loss function/y-axis of Figure 2 is not precisely specified. Assuming that it is sth like the cross-entropy loss and based on my experience, my guess would be that parameterizations with losses above 3 (which is what vanilla still reaches) resemble the performance of randomly initialized networks or within the first few optimization steps epochs during training. That is when the parameters are far from fruitful valleys and hence, not where we want the gradient ascent step jumping towards.  

Despite these issues, I'd be easily convinced of RSAM's utility if the empirical results showed its superiority against other SAM variants and versatility across different tasks. Unfortunately, such results are largely missing.

* While the empirical performance gains over SAM are consistent across the shown experiments, they are less than 1% on average, which resembles the performance gains of other SAM variants, such as GSAM (cited in the paper), ASAM [1], SAM+SWA[3].  However, only for one task (model/dataset combination), the authors compare against GSAM, in which their performance differs by 0.1% and is identical (as the authors say themselves). 
* I appreciate the additional R-GSAM experiment, but again, I find the number of tasks (which is only one here) too limited. It would be great to see how R-SAM compares against GSAM, ASAM, SWA+SAM and on more tasks, and even if they reach similar performances; it would then be interesting e.g. to combine these and see if they are complementary.
* This method introduces another hyper-parameter $\gamma$, which may incur additional tuning overhead for practictioners. While Figure 3c) shows that it is rather stable for three different values on one particular dataset, it would be nice to see a more thorough sensitivity analysis with e.g. more values or a default value used on a few more datasets/architectures (like SAM's rho=0.05).
* The paper includes thorough image classification results; however, it lacks empirical analyses on other tasks such as NLP or graph learning [2,3]. 
* All empirical results seem to be the result of one training run. It would great to see averaged results across at least three random seeds $\pm$ std. 

Minor: 
* Figures 1 and 2 show the effect of random smoothing (RS) on the loss surface. Since RS is a stochastic operation, it would be great to see some empirical confidence intervals (e.g., $\pm 1$ std across 3 seeds).   



[1] ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks, Kwon et al, ICML 2021.
[2] Sharpness-Aware Minimization Improves Language Model Generalization, Bahri et al, ACL 2022.
[3] A Fair Comparison of Two Popular Flat-Minima Optimizers: Stochastic Weight Averaging vs. Sharpness-Aware Minimization, Kaddour et al, arXiv:2202.00661.
[4] Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscape, Bisla et al, 2022. 
questions: Let me rephrase my previously mentioned weakness as a question here. 

It is not entirely clear why performing the inner maximization gradient ascent step on a smoothed loss function necessarily leads to a more "accurate" solution or exactly what that means. Following the same argument, we would expect that we want to apply SGD (solving a minimization objective) on a smoothed loss function too. Yet, this is not standard practice. Works like [1] recently proposed to explore such smoothening; however, with the motivation of finding flat minima (akin to SAM), not "gradient stability" or more "accurate" solutions. It would be great if the authors could clarify why "gradient stability" is important for inner maximization. 

I'm happy to raise my score if this gets sufficiently addressed. 

[1] Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscape, Bisla et al, AISTATS 2022. 
limitations: As far as I can see, the authors do not discuss any limitations of their method.
review_title: null

Meta-review: metareview: All reviewers except one agreed that this paper should be accepted because of the strong author response during the rebuttal phase. Specifically the reviewers appreciated the significance of the problem being addressed, the clarity of the paper, the simplicity of the method, and the analysis. Authors: please carefully revise the manuscript based on the suggestions by the reviewers: they made many careful suggestions to improve the work and stressed that the paper should only be accepted once these changes are implemented. Once these are done the paper will be a nice addition to the conference!
confidence: Certain
Decision: Accept