Random Sharpness-Aware Minimization
Authors: Yong Liu
Siqi Mai
Minhao Cheng
Xiangning Chen
Cho-Jui Hsieh
Yang You
Venue: NeurIPS 2022 Accept (2022)
Keywords: Sharpness-Aware Minimization
Generalization
Adversarial Training

Abstract: Currently, Sharpness-Aware Minimization (SAM) is proposed to seek the parameters that lie in a flat region to improve the generalization when training neural networks. In particular, a minimax optimization objective is defined to find the maximum loss value centered on the weight, out of the purpose of simultaneously minimizing loss value and loss sharpness. For the sake of simplicity, SAM applies one-step gradient ascent to approximate the solution of the inner maximization.  However, one-step gradient ascent may not be sufficient and multi-step gradient ascents will cause additional training costs.  Based on this observation, we propose a novel random smoothing based SAM (R-SAM) algorithm. To be specific, R-SAM essentially smooths the loss landscape, based on which we are able to apply the one-step gradient ascent on the smoothed weights to improve the approximation of the inner maximization. Further, we evaluate our proposed R-SAM on CIFAR and ImageNet datasets. The experimental results illustrate that R-SAM can consistently improve the performance on ResNet and Vision Transformer (ViT) training. 

Random Sharpness-Aware Minimization
Yong Liu', Siqi Mai", Minhao Cheng', Xiangning Chen', Cho-Jui Hsieh', Yang You
*Department of Computer Science, National University of Singapore HPCCAT Technology Inc. Department of Computer Science and Engineering, HKUST *Department of Computer Science, University of California, Los Angeles {liuyong, youy)@comp.. nus edu. sg. minhaocheng@ust. hk, (xiangning, chohsieh)@cs. ucla. edu,
Abstract
Currently, Sharpness-Awar Minimization (SAM) is proposed to seek the param- eters that lie flat region to improve the generalization when training neural networks. In particular, minimax optimization objective is defined find the maximum loss value centered on the weight, out of the purpose of simultaneously minimizing loss value and loss sharpness. For the sake of simplicity, SAM applies one-step gradient ascent to approximate the solution of the inner maximization. However, one-step gradient ascent may be sufficient and multi-step gradient ascents will cause additional training costs. Based on this observation, we propose novel random smoothing based SAM (R-SAM) algorithm. To be specific, R-SAM essentially smooths the loss landscape, based on which we are able to apply the one-step gradient ascent on the smoothed weights to improve the approximation of the inner maximization. Further, we evaluate our proposed R-SAM on CIFAR and ImageNet datasets. The experimental results illustrate that R-SAM can consistently improve the performance on ResNet and Vision Transformer (ViT) training.
Introduction
Deep learning has remarkable progress in variety of areas, with advanced development in over-parameterized neural networks. However, over-parameteriiztion usually leads to overfitting and suffers poor generalization for unseen data [31]. To this end, many studies have investigated the generalization of deep neural network to narrow down the gap [7. 17, 26, 29, 42]. In particular, they try analyse the local minima problem and connect with loss geometry [17, 27, 31]. Amongst them, Foret et al. propose SAM, with an aim to solve a minimax optimization problem. SAM initially finds the maximum of loss function in the region of radius p centered on the w, regarding which to minimize the loss value. This will prompt the model to converge to flat region, where the entire neighborhood tends to have uniformly low training loss. Inner maximization a vital part of SAM procedure as it essentially describes the loss landscape and plays an important role in whether we can reduce the loss sharpness effectively. The solution for inner maximization NP-hard, and for simplicity, SAM uses one-step gradient ascent directly to approximate the maximum loss. However, one-step gradient ascent may not be sufficient. In addition, multi-step gradient ascents are efficient and will cause additional training costs. Therefore, how to improve the approximation of the inner maximization without additional computation cost an important problem.
Based on our empirical analysis in Figure and Theorem 1, we find that random initialization can smooth the loss landscape and obtain more stable gradient direction. Noted that more stable gradient can make a good approximation to the true maximum value. Therefore, in this paper, we propose a novel random smoothing based sharpness-awand minimization algorithm (R-SAM). Our proposed R-SAM consists of two steps. First, we use a Gaussian noise to smooth the loss landscape and escape from the local sharp region obtain a stable gradient for gradient ascent.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Second, we perform the one gradient ascent on the smoothed weights to eventually enhance the approximation quality of inner maximization.
We summarize our contributions as follows:
this paper, we investigate the approximation quality of this one-step gradient ascent. Our evaluations show that unstable gradients might hurt the inner maximization of SAM and adding random smoothing can improve gradient stability.
Based on observation, we propose simple but novel algorithm, R-SAM, which leverages a Gaussian noise in perturbation initialization and successfully enhance the inner maximization without introducing additional cost.
We conduct empirical studies ResNet and VIT. The experimental results demonstrate that R-SAM can help model generalization ability and yield more remarkable performance compared with SAM across a wide range of computer vision tasks (e.g., CIFAR-10/1000, ImageNet).
2 Background
2.1
Sharpness-Aware Minimization
Many studies focus analyzing the generalization of deep neural networks. For example, motivated by the local geometry of the energy landscape, Chaudhari et al. propose Entropy-SGD to construct local-entropy- based objective function that favors eelereenerll solutions lying in large flat regions of the energy landscape, to address generalization problems in the training. Izmailov et al. illustrate the benefits of Stochastic Weight Averaging (SWA) procedure in solving generalization problems and well apply SWA to generate much flatter solutions than SGD.
Keskar et al. shows that deep neural network training is easy to converge to sharp local minima and causes generalization gap. Therefore, how make the model converge to flat local minima and improve the generalization is important problem. Foret et al. propose SAM to seek the parameters that lie in a flat region through optimizing loss value and loss sharpness. Then we will overview Aware Minimization (SAM).
Consider the training dataset as S {a y:) J where each sample (2,, yi) follows the distribution D. Define f fz; w) as the neural network model with trainable parameter w E R?, with which the loss function w.r.t. an input 2, can be given by (x,; w) M.) E R+ shortened ((x,) for convenience. The empirical training loss is L = 7/ E [[(x;;; w), 1.). The objective of SAM [18] is to find the
parameters with uniformly low loss value within the l, ball. The objective function is proposed as follows.
L SAM (w) = max L(w + 8), 81.550
where p 2 defined as the radius of the l, ball. For simplicity, we ignore p when using & norm. As calculating the optimal solution of inner maximization is infeasible, SAM uses one-step gradient ascent approximate it:
8(ur) pV tn (ww)/I
arg max ||
T(un + 5).
Finally, SAM computes the gradient regarding the perturbed model w + for the update:
V S.MM (w) R V ((w))
Although SAM can achieve a great performance in traditional computer vision [8] and NLP tasks [3]. there still exists several drawbacks yet be resolved. To further improve its performance, Kwon et al. introduce the concept of adaptive sharpness of loss function to solve the scale-dependency problem. Zhuang et al. propose GSAM seek the region with both small loss value and low sharpness based on novel optimization objective. Zhou & Chen try to learn dynamically reweighted perturbation for each batch to obtain a better approximation to per-instance perturbation. Noted that SAM needs to calculate two sequential gradients. To improve its efficiency, Du et al. propose ES AM, which use stochastic weight perturbation and cearpness-sensitiiee data selection to reduce the computation of vanilla SAM. Liu et al. propose LookSAM reuse the projected gradient that target to the flat region

---

every k steps to accelerate ViT training. SS-SAM is proposed to improve the efficiency by randomly selecting SGD optimization or SAM each step [51].
2.2 Generalization
There is venerable line of important works the problem of generalization in deep learning[5, 12, 21, 28, 30J. For example, Hardt et al. focus on generating generalization bounds for models learned with stochastic gradient descent. Bisla et al. investigate the sharpness of loss landscape and proposed systematic mechanisms underlying the generalization abilities of deep learning models. Jin et al. analyse the importance of perturbed gradient descent in efficiently escaping saddle points. Further, Jin et al. investigate the use of a Hamiltonian function and propose new framework to track the long-term behavior of gradient-based optimization algorithms. In addition, Dinh et al. argue that the concepts of flatness might be problematic and can not be directly applied to explain generalization, which requires rethinking what flatness actually means.
Adversarial training has achieved a great success in improving the robustness of model [13, 20, 39, 40, 45]. Our approach was also inspired by recent works in random smoothing in adversarial training[1, 9, 44, 47, 50]. Wong et al. demonstrate that applying random initialization in FGSM adversarial training Andriushchenko & Flammarion study the questions of when and why FGSM adversarial training works, and how to enhance the solution of the inner maximization problem. Zhang et al. propose novel bi-level optimization-ban
in fact be just as effective as the more costly PGD adversarial training. Further,
fast adversarial training framework, FAST-BAT to address the issue of overfitting and achieve better performance in generalization ability.
3 Proposed Method
3.1 Empirical Observation on Approximation of Inner Maximization
As shown in Section 2.1, SAM defines min-max optimization objective to find the maximum loss value LSAM (w) near the current weight ur and then minimize the value L,AAM (wr) However, for the inner maximization problem, finding the optimal solution is NP-hard, which means that it's difficult to obtain the true maximum value. Therefore, SAM uses a one-step gradient ascent to approximate the optimal solution and obtain the maximum loss value. In this paper, we investigate the approximation quality of this one-step gradient ascent and propose a simple way to improve the inner maximization without introducing additional cost.
Inner maximization of SAM may be hurt by unstable gradient. The adversarial weight Wade with the optimal solution of inner maximization can be defined as:
w adv w
arg max L(w + 6)- |0
(4)
SAM uses one-step gradient ascent to approximate the optimal adversarial weight:
(ww(w) | n L(will
w wade = w + arg max L(w + 8) w |8
(5)
However, the loss landscape of neural network is usually sharp [31, 35]. That means the direction of (w) = ((w) cannot represent the direction maximize its neighbourhoods' loss value. However, a stable gradient direction in a continuous parameter space can narrow the approximation gap and help SAM obtain a better approximation of w² wadv- Therefore, we try to conduct the experiments in ResNet and WideResNet to analyse the smoothness of gradient (w)) As shown a(w) in Figure 1, plot the cosine value between the g(w) and g(w p [[s(("]] ) to visualize the smoothness. From this figure, we can find that the cosine value decreases significantly with the p value increasing (the red line). Intuitively, when the gradient is constant in a certain region (cosine similarity equals to 1), the one-step gradient ascent will get the exact maximizer; while when the gradient becomes unstable in certain region, the one-step gradient ascent will lead to a poor solution. Therefore, how to smooth the loss landscape and make the gradient stable during the gradient ascent process is important problem.
3

---

1.0
1.0
Vanilla
Vanilla
0.9
0.0075
y² 0
0.01
0.8
1' =.01
0.8
0.0125
Y - 0.0125
Cosine
0.7
0.7
0.6
0.5
0.5
0.4
0.4
0.3
0.01
0.02 0.03 0.04 0.05 0.06 0.07 0.08
0.09
0.10
0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.10
(a) R8 on CIFAR-100
(b) WideResNet on CIFAR- 100
Figure 1: Analysis about the Approximation of Inner Maximization. We plot the cosine value between g(w) and g(w + P g(w)
[[g(w)]] (the red line) and between g(t) and g( + P (|((())11 (the green, blue and orange line), where g(w) = (ww) is the vanilla gradient, and g = (wb V L(w + 66) represents the gradient on the weight with Gaussian initialization, where u = w + 5g. Based on the analysis in [2], we use training-time BatchNorm to conduct the experiments. y is the standard variation of ōg-
Random smoothing improves gradient stability. Some studies illustrate that random smoothing can smooth the loss landscape [22, 46]. That motivates us to analyze the effects of random smoothing
for the smoothness of gradient g(w). ). In Figure I, we sample Gaussian noise dn from N(0, >²)) and obtain the weight = w + d0. Next, we calculate the cosine value between g ((0) and
g( ((²)) (1))(())) to compare the smoothness with the gradient without random smoothing (g( (g(w)). From Figure 1, we find that the cosine value with random smoothing (the green, blue and orange line) is higher than the vanilla cosine value on (ww) (the red line), which means that random smoothing can reduce the decay rate of cosine value. In addition, the cosine value will be more stable with the y value increasing when using random smoothing (from y² = 0.0075 to 72 = 0.0125) in Figure 1. The above observation illustrates that random smoothing can smooth the loss landscape and obtain more stable gradient. We also provide the theoretical analysis for this observation.
Theorem 1. Let L(w) be the vanilla loss function and Ls(w) = L(w + 80) be the smoothed loss function. Assuming vanilla loss function L(w) is a- a-Lipschitz continuous and the gradient T 3-Lipschitz continuous. Given the random noise do is from Gaussian distribution ~²),.²1). We can obtain that the gradient V LwL((w) = V w L(w + 5o) is min a B 3}-Lipschitz continuous, where y is the standard deviation of do-
Theorem is based on [5, 16], which implies that Ls(w) L(w + 6o) is more smooth than the original loss function L(w) when _ < B. In addition, with y value (standard deviation of Gaussian noise) increasing, the loss function Ls(w) become smoother. That is also consistent with our observation in Figure 1, where the change of cosine value will be reduced with 7 increasing. Therefore, based on the above intuition about random smoothing, we can use the Gaussian noise ôn to initialize the one-step gradient ascent process. That means we can firstly use Gaussian noise to escape from the local sharp region and smooth the loss landscape (initial perturbation). and then conduct the one-step gradient ascent step. More specifically, the random initialization for inner maximization do- After that, we the smoothed weight v0 to achieve the one-step
can be defined i gradient ascent process and obtain the approximation of optimal adversarial weight:
( + 5o)
W Wadv ②
+
W 60 + p (w + do)ll
(6)
Finally, we investigate whether the smoothed gradient can benefit the inner maximization and obtain better approximated solution in practical applications. We try to plot the perturbed loss value 9(w+6u)
((w) and L(w + 50 + P (1((w+66)) )) in Figure 2. From this figure, we can find that the loss value is very close for different 72 value when the P value is small (p S 0.2). However, with
(I(w + P (1))(())))

---

Vanilla
Vanilla
6
y 0.0075
6
y 0.005
5
yd 0.01
5-
0.0075
4
2-
1-
0.1 0.2 0.3
0.4
0.5
0.6
0.7
0.8 0.9
1.0
0.1 0.2
0.3
0.4
0.5
0.6
0.7
0.8 0.9
1.0
(a) ResNet-18
(b) WideResNet
Figure 2: The effect of Gaussian noise for loss value. The red line (vanilla) represents the loss
function L(w g(w) with value increasing. The other three lines show the change of loss
function with random initialization L( - Noted that do is from the Gaussian distribution with three different standard deviation (T² 0.0075, 0.01, 0.01 We select the checkpoint of 200-th epoch on CIFAR- as the weight us to plot the figure.
+ 80- (()(())] when increasing P value, where
the p value gradually increasing, the loss value grow slowly for vanilla weight w. That means g(w) cannot efficiently increase the loss value and is not applicable for solving the inner maximization with the region expanding. That limited region size that vanilla weight can select. addition, we observe that the weight with Gaussian noise (w do) shows a more stable pattern for the increase of loss value to solve inner maximization problem. Applying one-step gradient ascent based on the gradient of smoothed weight + 5o can better maximize the loss value in the neighhouring region. The above analysis motivates us to propose a random smoothing based SAM algorithm to obtain better solution for inner maximization. We will introduce proposed Random Sharpness-Awee Aware Minimization algorithm (R-SAM) the next section.
3.2 Random Sharpness-Awee Aware Minimization (R-SAM)
Based the observation and intuition in section 3.1, propose novel random initialization based Aeappe
Sharpness- Aware Minimization algorithm (R-SAM) to benefit the solution of inner maximization in SAM. For R-SAM, smoothed weight v = w + 5g. After that, we use the smoothed weight ú to calculate the gradient ( for gradient ascent process: ( V L(t)) V L w ōu). In this way, a more stable gradient to solve the inner maximization problem. More specifically, we will use the gradient g ((Ø) finish the gradient ascent step and obtain adversarial weight Wadv- A simple way to obtain Wado is that we can directly add the gradient-based perturbation ( = V. L(x)) to the smoothed weight v
Gaussian noise initialize the perturbation for weight and obtain the
obtain
dg, which is defined Wade
+ 5o + P||a(*))11 However, that will
_(w))
cause the norm of total perturbation 5 + + 9(1h)
[[(())]] larger than vanilla p. To constrain the norn of
total perturbation, we try to project the perturbation into the ball with radius The final perturbation can be defined as:
do (g
Wadv w + 1100 A
where do represents the Gaussian noise for random initialization and is used control the ratio between 6o and (
Noted that the selection of Gaussian noise plays an important role in R-SAM. However, unlike the input image data that each pixel is from the same distribution, distribution of weights each layer or kernel shows different distributions. That makes the noise design challenging problem. Related
5

---

Algorithm Random-SAM (R-SAM)
Input: x E IRd, w represents vanilla weight, zù is the smoothed weight, learning rate ne. l the number of kernels in model.
for t + to do
Sample Minibatch B (x) (aa7)) ((8))
from X.
60
N(0, 72 diag(1) wyy (|l))11))
Obtain a stable gradient g V L Vw + 60) on minibatch B.
Compute gradient ascent Wadv w + 60 ( (g+)
Compute SAM gradient: Jsam V Vw ((1) wode
Update weights: 01++1 = We 719amm end for
work has illustrated that the weight in each kernel is from an independent Gaussian distribution [25]. Inspired by this work, in this paper, use kerne-wwie Gaussian noise do (0,0,>² d fll))) as the initialization of perturbation in R-SAM, where | represents the &g no-m of k-th kernel in the neural network.
The pseudo-code of R-SAM can be found in Algorithm 1. Firstly, need to sample minibatch data B = {(z,, 34),. ....................... ( } from dataset X. After that, we will calculate the norm of each kernel ||w زززا and sample kernel-wise Gaussian noise from the distribution N(O.>² diag(( [djag(//w)]]]) [|wr|l/()))))))))))))))))) as the initialization of perturbation. Then we can obtain ( = V C. un + 80) as the stable gradient to better approximate the optimal adversarial weight wadv More specifically, we can use one-step gradient ascent to calculate Wadv on Wade: Jeam = L..L(w)lm Finally, following the process vanilla SAM, we can use the gradient Jaam to update the vanilla weight w. As shown in Algorithm 1, R-SAM is very easy to reproduce.
(+)) In this way,
P +
calculate the gradient
Experiments
4.1 Experimental Settings
In this section, try to evaluate the performance of proposed R-SAM optimizer ResNet [23], WideResNet [48] with CIFAR [33] datasets and Vision Transformer (ViT) [14] with ImageNet [11] dataset. First, compared with vanilla SAM, the experimental results illustrate that our proposed R-SAM can improve the accuracy in CIFAR and ImageNet. In addition, the ablation study and sensitivity analysis show that our proposed R-SAM robust for hyper-parameters.
4.1.1 Datasets
To validate the performance of proposed method R-SAM, we firstly try to conduct the experiments on widely used CIFAR-10, CIFAR-100 [33] and ImageNet- 1 [11] datasets. To further validate the performance and robustness of R- R-SAM, we also report the experiment results ImageNet- Real [4], ImageNet-V2 [41], ImageNet-R [24] and ImageNet-C [19].
4.1.2 Models
We firstly evaluate the performance of our proposed R-SAM in CNN-based models, such as ResNet- 18. ResNet-50 [23] and WideResNet (WRN-28-20) [48]. Currently, -ransformer-based models receive great deal of attention in the field of computer vision [37, 43]. After that, we also try to evaluate our proposed algorithm R-SAM in Vision Transformer (ViT) [14].
4.1.3 Baselines
To illustrate that the random initialization in R-SAM can help to escape the non-smooth vicinity and benefit to the approximation in inner maximization, the main baseline of this paper is vanilla SAM [18]. In addition, show that SGD+M is more suitable for CIFAR training, we also try to compare

---

the accuracy of SGD+M with SGD, RMSProp and Adam W [38]. To further show the generality of R-SAM, we also try combine it with GSAM [53] and propose R-GSAM algorithm. The experiment results illustrate that R-GSAM can obtain a better performance compared with vanilla GSAM in ViT training. That also illustrates the effectiveness of random initialization in R-SAM and R-GSAM.
4.1.4 Implementation Details
The experiments in this paper implemented with JAX [6] Google TPU-V3 chips. The hyper- parameters and experimental setting is based on vanilla SAM [8, 18). For ResNet training, we use SGD with Momentum the base optimizer of SAM and R-SAM. For VIT training, the base optimizer of SAM and R-SAM is AdamW [38]. As for data augmentation, we select mixup [49] and RandAugment [10] to conduct the experiments. Noted that we do not use stochastic depth for ViT training. The implementation details can be found in Appendix 7.4.
4.2 Training from Scratch on CIFAR-10
To evaluate the performance of our proposed R-SAM, we conduct experiments of ResNet-18, ResNet- 50 and WideResNet (WRN-28-20) training on CIFAR-10 dataset. Following the experimental setting vanilla SAM [18]. we use basic augmentation to preprocess the input image data and the base optimizer for SAM and R-SAM is SGD with Momentum (SGD+M).
The
experimental results shown in Table 1. We can find that SAM-based optimizers (SAM and R-SAM) can achieve a better performance than traditional first-order optimizers (SGD and AdamW). For example, the accuracy of traditional first-order optimizers around 96%, and the best accuracy is 96.5% from SGD with Momentum (SGD+M). However, SAM can improve the accuracy from 96.5% 97.3% for WRN-28-20. That illustrates the power of SAM improve the performance of neural networks. In addition, R-SAM can further improve the performance compared with vanilla SAM. More specifically, R-SAM can improve the accuracy of SAM from 97.3% to 97.5% on WRN-28-20 20 for CIFAR-10. Noted that the accuracy of SAM for CIFAR-10 is pretty high and the improvement of R-SAM is also convincing.
Table Accuracy of ResNet and WideResNet on CIFAR-10 for 200 epoch. The base optimizer for SAM and R-SAM is SGD with Momentum (SGD+M), We select batch size as 128.
Model
SGD
SGD+M
RMSProp
AdamW
LPF-SGD
SAM
R-SAM
ResNet-18
95.411..1
95.610.2
95.410.2
95.110.11
95.910.1
96.410.2
96.5.1..1
ResNet-50
95.8+0.11
95.7+0.1
95.7++0.1
96.0+b0.1
96.3+0.2
96.7+t..1
96.9
WRN-28-10
96.410.1
96.5#0.1
96.420.2
96.0+0.1
96.810.11
97.3#0.1
97.5=1.11
4.3 Training from Scratch on CIFAR-100
To further validate the performance of our proposed R-SAM, we also try to evaluate its accuracy on CIFAR-100 dataset. We also follow the setting in SAM [18] and use basic augmentation to preprocess the image data. We select SGD+M as the base optimizer of SAM and R-SAM.
Table 2: Accuracy of ResNet and WideResNet on CIFAR- 100 for 200 epoch. The base optimizer for SAM and R-SAM SGD with Momentum (SGD+M). We select batch size as 128.
Model
SGD
SGD+M
RMSProp
AdamW
LPF-SGD
SAM
R-SAM
ResNet-18
78.010.1
78.910.2
79.410.11
77.710.1
80.210.11
80.910.2
81.410.2
ResNett-0
80.9-0.2
81.410.2
81.410.1
80.8+0..1
82.110.22
83.310..1
84.0-10.1
WRN-28-10
81.1+0.1
81.740.1
81.740.11
80.1
82.6+0.1
84.6#0.1
85.2+0..1
The experimental results are shown in Table 2, we notice that the results show a similar pattern with CIFAR-10 and can achieve a higher improvement. In particular, SAM-based optimizers can obtain better performance compared with first-order optimizers. For example, the best accuracy of first-order
7

---

optimizers is 81.7% for WRN-28-20 model. SAM can improve the accuracy from 81.7% to 84.6% for WRN-28-10 and from 81.4% to 83.3% for ResNet-50. In addition. R-SAM can obtain higher accuracy compared (from 84.6% to 85.2% for WRN-28-20 and 83.3% to 84.0% for ResNet-50). The above experimental results also illustrate that our proposed R-SAM can further help SAM to obtain better performance.
4.4 VIT Training from Scratch on ImageNet
Currently, Vision Transformer has been widely used in a variety of areas [37, 43]. Chen al. illustrate that SAM can significantly improve the performance of ViT training. Therefore, evaluating the performance of our proposed R-SAM on ViT is pretty important for verifying our contribution. In this section, we select ViT-B-16 and ViT-S-16 to evaluate the performance of R-SAM. The experimental setting follows the introduction in [8, 18]. Noted that the base optimizer of SAM and R-SAM are both AdamW. We select batch size as 4096 for VIT training.
VIT Training. The experimental results of ViT training on ImageNet are shown in Table 3. From this table, we can observe that SAM can also significantly improve the accuracy of ViT training. For example, the accuracy of ViT-B-16 is improved from 74.7% (AdamW) 79.8% (SAM). In addition, when using SAM, the accuracy of ViT-S-16 can also be improved from 74.9% to 77.9%. However, our proposed R- R-M can further improve the accuracy of ViT training compared with vanilla SAM and achieve better performance. More specifically. R-SAM can improve the accuracy of ViT-B-16 from 79.8% (vanilla SAM) 80.7%. Besides, ViT-S-16 can also obtain benefits from R-SAM and the accuracy is increased from 77.9% (vanilla SAM) to 78.7%. The above experimental results further validate the effectiveness of our proposed R-SAM, especially when compared with vanilla SAM.
Table 3: Accuracy of ViT ImageNet-Ik 1k for 300 epoch. The base optimizer for SAM and R-SAM is AdamW. Batch Size is 4096. We use Inception-style preprocessing method for input image.
Model
Resolution
Mixup
RandAug
AdamW
SAM
R-SAM
VIT-B-16
224
74.7
79.8
80.7
ViT-S-16
224
74.9
77.9
78.7
ViT-B-16
224
75.7
80.4
81.1
ViT-B-16
224
79.6
80.8
81.6
Data Augmentation. To further evaluate the generality of our proposed R-SAM, we also try to combine R-SAM with strong augmentation methods and the results are shown in Table 3. In this experiment, we select mixup [49] and RandAugment (RandAug) [10] as the augmentation methods find that augmentation methods can further improve the
to conduct the experiments. Firstly, accuracy of Adam W and vanilla SAM. For instance, we can observe that mixup can improve the accuracy of vanilla ViT-B-16 with Adam from 74.7% 7% to 75.7%. In addition, when SAM is selected, the accuracy will further increase from 79.8% to 80.4% with mixup for ViT-B-16.
In addition, the experimental results illustrate that our proposed R-SAM can further improve the accuracy of vanilla SAM even with the strong augmentations. For example, R-SAM can improve the accuracy of SAM from 80.4% to 81.1% even using strong augmentations. These experiments further verify that our proposed R-SAM can combined with strong augmentations and illustrate a great generality. Finally, we find that mixup and RandAug can work together to benefit the performance of R-SAM. As shown in Table 3, when RandAug is selected, the accuracy is increased to 81.6% compared to simply using only the mixup.
Further Evaluations and Robustness. Firstly, to further validate the performance of R-SAM, we report the experiment results on ImageNet-Real and ImageNet-v2. As shown in Table 4, we can observe that R-SAM can still improve the accuracy of ViT compared with vanilla SAM. For example, R-SAM can improve the accuracy of ViT-B- 16 from 85.2% to 86.1% on ImageNet-Reall In addition, to evaluate the robustness of trained model with R-SAM, we also conduct the experiments in IeageNet-R and ImageNet-C datasets. The experimental results are also shown in Table 4. which illustrate that SAM can improve the robustness of trained model compared with AdamW. addition, R-SAM can maintain the advantage of SAM and further strengthen its robustness. For example, SAM
8

---

01.5
01.0
11.0
80.5
20.5
80.0
60.0
76.5
75.5
79.0
R-55AM
78.0
1.5
3.6
(a) Comparison with GSAM
(b) Sensitivity Analysis of p
(c) Sensitivity Analysis of À
Figure 3: (a) Comparison with GSAM. Vanilla R-SAM can achieve similar accuracy for ViT-B-16 training. R-GSAM can obtain a higher accuracy. (b) Sensitivity Analysis of R-SAM about p. We can find that R-SAM shows a stable pattern with different p value. (c) Sensitivity Analysis of R-SAM about A. R-SAM can obtain stable improvement under different value.
Table 4: Evaluation of R-SAM on ImageNet datasets for 300 epoch. The base optimizer for SAM and R-SAM A. We select batch size as 4096. We also use Inception-style preprocessing method for input image.
Model
Training
ImageNet
ImageNet-Rect
ImageNet-v2
ImageNet-R
ImageNet-C
AdamW
74.6
79.8
61.3
20.1
46.6
ViT-B-16
SAM
79.8
85.2
67.5
26.4
56.5
R-SAM
80.7
86.1
26.9
56.8
Adam W
74.4
80.4
20.0
46.5
VIT-S-16
SAM
77.9
84.1
65.6
24.7
53.0
R-SAM
78.7
84.9
66.7
25.1
53.2
can improve the accuracy of ViT-B-16 from 20.1% 26.4% ImageNet-R. R-SAM further improve the accuracy to 26.9%.
4.5 Compared with Algorithms in the Literature
In section 2.1, we introduced some related work about SAM. In this section, we try to evaluate the performance of R-SAM compared with the algorithms in the literature. We find that GSAM can achieve the best accuracy for VIT training. Therefore, we mainly focus on the comparison with GSAM. The main reults are from ViT-B-16 training on ImageNet-Ik dataset and are shown in Figure 3(a). We can find that R-SAM can obtain the similar accuracy compared with vanilla GSAM. Howvever, when combining R-SAM and GSAM and named it as R-GSAM, we can obtain better performance. For instance, the accuracy of R-SAM and GSAM are 80.7% 7% and 80.8% respectively. When using R-SAM for GSAM and obtain the results of R R-GSAM, we can observe that the accuracy increase from 80.8% to 81.2%. The main purpose of this paper is to illustrate that random smoothing can benefit the inner maximization and improve the performance of SAM.
4.6 Sensitivity Analysis
Reproducibility is critical for deep learning applications. We notice that deep learning is usually sensitive to hyper-parameters. Therefore, in this section, we to study the effects of hyperparameters the performance of ViT training on ImageNet-Ik lk dataset. More specifically, we mainly focus on the effects of p and A. The reason is that determines the radius of perturbation and À controls the ratio between Gaussian noise da and gradient-based perturbation []] The analysis results
g()))
shown in Figure 3. We can find that p and shows stable pattern for tuning. That further improves the reproducibility of R-SAM and make it more applicable in real scenarios.

---

Conclusion
In this paper, we observe that simply applying one-step gradient ascent in the inner maximization of SAM may suffer from poor approximation error due to unstable gradient. To resolve this problem, we propose novel algorithm R-SAM, which provably improves the gradient stability by initializing the inner maximization of SAM with a Gaussian noise. The experimental results on ResNet and ViT illustrate that R- SAM can improve the generalization performance of the model and yield better performance compared with SAM on CIFAR-10, 10, CIFAR- 100 and ImageNet datasets.
6 Acknowledgements and Disclosure of Funding
We thank Google TFRC for supporting us get access the Cloud TPUs. We thank CSCS (Swiss National Supercomputing Centre) for supporting us to get access to the Piz Daint supercomputer. We thank TACC (Texas Advanced Computing Center) for supporting us to get access the Longhorn supercomputer and the Frontera supercomputer. We thank LuxProvide (Luxembourg national super- computer HPC organization) for supporting us to get access the MeluXina supercomputer. CJH and XC are partially supported by NSF IIS-20081 173, NSF IIS-2048288 and research awards from Google, Sony, Samsung and Okawa Foundation. YY and YL. are also partially supported by grants from Alibaba, Bytedance, Huawei, and Singapore Maritime Institute.
References
[1] Maksym Andriushchenko and Nicolas Flammarion. Understanding and improving fast ad- versarial training. Advances in Neural Information Processing Systems, 33:16048-16059, 2020.
[2] Maksym Andriushchenko and Nicolas Flammarion. Towards understanding sharpness-aware minimization. In International Conference on Machine Learning, pp. 639-668. PMLR, 2022.
[3] Dara Bahri, Hossein Mobahi, and Yi Tay. Sharpness-aware minimization improves language model generalization. arXiv preprint arXiv:21 2021.
[4] Lucas Beyer, Olivier Hénaff, Alexander Kolesnikov, Xiaohua Zhai and Aäron van den Oord. Are we done with imagenet? arXiv preprint arXiv:2006.07/55 159, 2020.
[5] Devansh Bisla, Jing Wang, and Anna Choromanska. Low-pass filtering sgd for recovering flat optima in the deep learning optimization landscape. arXiv preprint arXiv:2201.0800 2022.
[6] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Minn and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL attp://githut com/ wwt.cooooo/o/////eooooo google/jax.
[7] Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi. Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-sgd. Biasing gradient descent into wide valleys. Journal of Statistical Mechanics: Theory and Experiment, 2019(12): 124018, 2019.
[8] Xiangning Chen, Cho-Jui Hsieh, and Boging Gong. When vision transformers outperform resnets without pre-training or strong data augmentations. In International Conference on Learning Representations, 2022.
[9] Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In International Conference on Machine Learning, pp. 1310-1320. PMLR, 2019.
[10] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops, pp. 702-703, 2020.
[11] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEFE conference on computer vision and pattern recognition, pp. 248-255. Icce, 2009.
10

---

[12] Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In International Conference on Machine Learning, pp. 1019-1028. PMLR, 2017.
[13] Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial attacks with momentum. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9185-9193, 2018.
[14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint WxXiv:0000.11929 2020.
[15] Jiawei Du, Hanshu Yan, Jiashi Feng, Joey Tianyi Zhou, Liangli Zhen, Rick Siow Mong Goh, and Vincent YF Tan. Efficient sharpness-aware minimization for improved training of neural networks. arXiv preprint 1rX0v:2110.03141, 2021.
[16] John C Duchi, Peter Bartlett, and Martin Wainwright. Randomized smoothing for stochastic optimization. SIAM Journal on Optimization, 22(2):674-701, 2012.
[17] Gintare Karolina Dziugaite and Daniel M Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv preprint 1.Xivv/7033. 1008, 2017.
[18] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware mini- mization for efficiently improving generalization. arXiv preprint arXiv:2010.01412, 2020.
[19] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv: 2018.
[20] Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015.
[21] Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In International conference on machine learning, pp. 1225-1234.
PMLR, 2016
[22] Kosuke Haruki. Taiji Suzuki, Yohei Hamakawa, Takeshi Toda, Ryuji Sakai, Masahiro Ozawa, and Mitsuhiro Kimura. Gradient noise convolution (gnc): Smoothing loss function for dis- tributed larch-batch sgd. arXiv preprint arXiv:/906. 10822, 2019.
[23] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pultern recognition, pp. 770-778, 2016.
[24] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: critical analysis of out-of distribtuii generalization. In Proceedings of the IEEE/CVF 2021.
International Conference Computer Vision pp.
[25] Zhongzhan Huang, Wenqi Shao, Xinjiang Wang, Liang Lin, and Ping Luo. Rethinking the pruning criteria for convolutional neural network. Advances in Neural Information Processing Systems, 34, 2021.
[26] Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. arXiv preprint 10Xiv:1803.05407, 2018.
[27] Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization measures and where to find them. arXiv preprint arXiv: 1912.2/1 2019.
[28] Chi Jin, Rong Ge, Praneeth Netrapalli, Sham M Kakade, and Michael I Jordan. How to escape saddle points efficiently. In International Conference on Machine Learning, pp. 1724-1732. PMLR, 2017.
11

---

[29] Chi Jin, Lydia T Liu, Rong Ge. and Michael I Jordan. On the local minima of the empirical risk. arxiv preprint arXiv: 1803.093339 2018
[30] Chi Jin, Praneeth Netrapalli, and Michael I Jordan, Accelerated gradient descent escapes saddle points faster than gradient descent. In Conference On Learning Theory, pp. 1042-1085. PMLR, 2018.
[31]
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprini 1rXiv (ee.1160900448 044366, 2016.
[32] Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large- batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representations, 2017.
[33] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[34] Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi. Asam: Adaptive sharpness- aware minimization for scale-invariant learning of deep neural networks. In International Conference on Machine Learning, pp. 5905-5914. PMLR, 2021.
[35] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in neural information processing systems, 31, 2018.
[36] Yong Liu, Siqi Mai, Xiangning Chen, Cho-Jui Hsieh, and Yang You. Towards efficient and scalable sharpness-aware minimization. arXiv preprint arXiv:2203.02714. 2022.
[37] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
(38] Iyya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 2017.
[39] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:/706. 1706.06083, 2017.
[40] Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow: Transferability in machine learn- ing: from phenomena to black-box attacks using adversarial samples. arXiv preprint drXiv:1605.07277.00 2016.
[41] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pp. 5389-5400. PMLR, 2019.
[42] Samuel Smith and Quoe Le. A bayesian perspective on generalization and stochastic gradient descent. arXiv preprint arXiv:1710.06451, 2017.
[43] Hugo Touvron, Matthicu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference Machine Learning, pp. 10347-10357, PMLR, 2021.
[44] Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick Mc- Daniel. Ensemble adversarial training: Attacks and defenses. arXiv preprint (xXiv:/705.07200 2017.
[45] Yisen Wang, Xingjun Ma. James Bailey, Jinfeng Yi, Bowen Zhou, and Quanquan Gu. On the convergence and robustness of adversarial training. In ICML, volume 1, pp. 2, 2019.
[46] Wei Wen, Yandan Wang, Feng Yan, Cong Xu, Chunpeng Wu, Yiran Chen, and Hai Li. Smoothout: Smoothing out sharp minima to improve generalization in deep learning. arXiv preprint arXiv:1805.07800 2018.
12

---

[47] Eric Wong, Leslie Rice, and J Zico Kolter. Fast is better than free: Revisiting adversarial training. arXiv preprint arXiv:2001.03994 2020.
[48] Sergey Zagoruyko and Nikos Komodakis.
Wide residual networks.
arXiv preprint
arXiv:1605.07146, 2016.
[49] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint 17Xiv:1710.09412, 2017
[50] Yihua Zhang, Guanhuan Zhang, Prashant Khanduri, Mingyi Hong, Shiyu Chang, and Sijia
Liu. Revisiting and advancing fast adversarial training through the lens of bi-level optimization. 2021.
arXiv preprint
[51] Yang Zhao, Hao Zhang, and Xiuyuan Hu. Ss-sam: Stochastic scheduled sharpness-aware minimization for efficiently training deep neural networks. arXiv preprint arXiv:220.009662, 2022.
[52] Wenxuan Zhou and Muhao Chen. 6-sam: Sharpness-awarr minimization minimmiiiooniiiiiniiiiiiiiiooo with dynamic reweight- ing. arXiv preprint arXiv:2112.08772,200 2021.
[53] Juntang Zhuang, Boqing Gong, Liangzhe Yuan, Yin Cui, Hartwig Adam, Nicha Dvorekk, Sekhar Tatikonda, James Duncan, and Ting Liu. Surrogate gap minimization improves sharpness-aware training. arXiv preprint arXiv:2203.08066 2022.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'le roe're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "dIGDYo8l07Z", "reviewer": "Reviewer_ciRx", "summary": "The paper proposes to modify the Sharpness-Aware Minimization (SAM) algorithm to include a random perturbation before taking the gradient ascept step. The paper presents some justifications to why it could be beneficial, based on the smoothing argument. The experimental results show the effectiveness of the proposed method.\n", "questions": "- Was a grid search done for SAM (e.g., Table 1 and 2)? And how are the hyperparameters tuned for each dataset for the proposed method, R-SAM?\n", "limitations": "The proposed method has three hyperparameters: the perturbation radius $\\rho$, $\\gamma$ for the noise standard deviation of the noise step, and $\\lambda$ for the step size of the gradient. In comparison, SAM has only one hyperparameter. The hyperparameter sensitivity presented in **4.6 Sensitivity Analysis** is not very convincing as it shows only one-dimensional grid search (while keeping the other two dimensions fixed to the optimal values) and, for example, the range of $\\rho$ presented in Fig. 3 is in fact quite narrow (from 1.2 to 1.6).\n\n---\n\n**Update after the rebuttal**\n*Overall, I'd say that the improvements coming from R-SAM are consistent across multiple settings which makes the proposed method practically useful. However, the core idea of simply adding random noise (although with specific covariance) before the ascent step of SAM is extremely simple and lacks novelty. Moreover, I think the paper could've been implemented more carefully in various aspects (mentioned above) such as careful ablation studies and better motivation of the proposed method. I strongly feel that since the proposed method is so simple, the paper has to compensate it with very careful and rigorous empirical validations. I increase my score to 4 in light of the improvements of the paper presented during the rebuttal but still feel that the paper is below the acceptance bar.*", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- **Originality.** Low. The idea of adding noise before taking the gradient ascent step of SAM is very simple.\n- **Quality.** Medium. The experimental evaluation of R-SAM is quite convincing. The findings that SAM/R-SAM can quite noticeably boost the OOD performance (ImageNet-R, ImageNet-C, etc) is also nice. On the other hand, the motivation of the proposed method requires significant improvements (see below).\n- **Clarity.** Medium. Most of the content is clear.\n- **Significance.** Overall, I appreciate the empirical effectiveness of the proposed method which could be of interest to practitioners. However, the paper requires multiple improvements listed below.\n\nThings that need improvement:\n- A crucial baseline, Bisla et al. (AISTATS’22), is cited but isn’t compared to or discussed in sufficient depth. A comparison to their method (which is **not** using worst-case perturbations) would be beneficial, especially since they leverage random perturbations with the same covariance as suggested in this paper. I wonder if it’s key to the performance of this method.\n- *“That means the gradient of smoothed weight $w + \\delta_0$ can better maximize the loss value in a large region.”* – But do we really need such large weight perturbations? For $\\rho=1.0$, the training loss is increased to $\\approx 6$ (Fig. 2) which is even higher than the loss of a random classifier ($\\approx ln(100) \\approx 4.6$).\n- Fig. 1 and 2: is it done for test-time BatchNorm? If yes, then it should be remade for training-time BatchNorm or for a network trained with a different normalization method. In practice, the difference between sharpness estimates of test-time and training-time BatchNorm is very significant (see, e.g., https://arxiv.org/abs/2206.06232).\n- The loss values in Fig. 2 are checked for the algorithm described in Eq. (6) while the proposed algorithm of R-SAM presented in Eq. (7) differs from Eq. (6) by the presence of the projection and a special covariance matrix of the noise. In order to properly motivate R-SAM (and not some other possible variation), one has to perform the experiment in Fig. 2 specifically for R-SAM. Intuitively, the projection step can make a lot of difference and I’m not sure if the algorithm in Eq. (7) will lead to a more accurate maximizer compared to the vanilla SAM.\n- In addition, the role of the noise covariance is not sufficiently discussed. It’s unclear whether it’s crucial or not. For such a simple proposed idea, I’d expect much more rigorous ablation study for the key elements of the proposed method.\n- The original SAM paper already pointed out that multiple steps of projected gradient ascent aren’t helpful (see Table 11 in https://arxiv.org/abs/2010.01412). That would be a more direct way to more accurately solve the inner maximization.\n- Error bars are necessary to claim that the observed improvement is indeed significant.\n\n\nIn multiple places throughout the paper, the writing is not precise and some claims don’t seem to be justified:\n- “However, the model usually locates in the sharp minima [31] where the unstable gradient, to a large extent, makes one-step gradient ascent performs poorly.” – This statement is surprising given that SAM aims to minimize sharpness and converge to a flat region. Moreover, this sentence leads to the impression that [31] demonstrated that this is somehow the case, although [31] has nothing to do with this.\n- “Therefore, how to improve the approximation of the inner maximization procedure and obtain a more aggressive weight is an important problem” – What is meant by the “aggressive weight”?\n- “However, the loss landscape of neural network is usually sharp and non-linear [30, 34].” – Surprising to see statements like this as the loss landscape of a **linear** model is also non-linear and may be considered sharp depending on the data covariance.\n- “we observe that models can still locate in a sharp region, leading to poor performance on one-step gradient ascent.” – This statement isn’t complemented by sharpness measurements.\n- “we are able to apply the one-step gradient ascent on the smoothed weights for a much more accurate measurement of the inner maximization” – There is no evidence to this in the paper. Fig. 2 is done for a different algorithm (see my concern about Eq. (6) vs. Eq. (7)).\n- Eq. (2): there should be a division by the gradient norm, not multiplication.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "SMdKl8qeio", "reviewer": "Reviewer_uTRr", "summary": "Sharpness-Aware Minimization (SAM) is an emerging deep learning technology in recent years, which aims to find parameters lie in a \"flat\" region of the energy landscape. Its purpose can be formulated as a minimax problem as shown in (4): A parameter vector w_adv is \"optimal\" in the sense of SAM if the loss of worst solution in the neighborhood of w_adv is minimized. The vanilla SAM solved this problem by approximating the landscape around a solution w as a linear function. This paper investigates this approximation thoroughly by experimental and theoretical analysis, and shows that this approximation is possibly poor in applications. Inspired by recent works in the field of adversarial training, authors improve the vanilla by introducing noise wo weight parameters.\n", "questions": "1) In Fig. 1, the \"flatness\" of w is measured by the cosine value between g(w) and g(w+rho*g(w)/||g(w)||). But is it a suitable measure for all w? If w is close to a local minimum and g(w) is close to zero, then directions of both g(w) and g(w+rho*g(w)/||g(w)||) are \"random\", and the cosine value could be small even in a very flat region. The detailed settings of experiments in Fig. 1 are not provided. If the w is obtained after training, we need to consider this case. \n\n2) There are some typos in figures. In Fig. 1, \\hat g(w) should be g(\\hat w). In Fig. 2, L(w+rho*g(w)/g(w)) should be g(w+rho*g(w)/||g(w)||).\n\n3) Do authors miss the mathematical expectation notation somewhere? For example, in Theorem 1, L_s(w)=L(w+delta_0) or E[L(w+delta_0)]? (If it is the former one, we cannot say that L_s(w) must have a smaller Lipschitz constant.)\n\n4) Baseline methods include SGD (and modified SGD) without SAM and existing SAM methods. Certainly it is impossible to compare the proposed method with all regularization techniques, but it would be better to consider some advanced non-SAM strategies here.\n\n5) As stated by authors, \"deep learning is usually sensitive to hyper-parameters\". So the small improvement of the accuracy cannot strongly prove that RSAM significantly outperforms the other SAM methods. I suggest to add some numerical results to show that RSAM can find \"flatter areas\", which is more meaningful than 0.1% improvement in accuracy.", "limitations": "Authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nBoth theoretical and numerical analysis is thorough. The idea is simple and easy-to-use. The paper is clearly written and the contribution is significant.\n\nWeakness:\n\nSee Questions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "RACcRH0dqV7", "reviewer": "Reviewer_Dv1G", "summary": "The authors argue that the approximation of SAM's perturbation step, which requires solving an inner maximization objective, is prone to unstable gradients. To improve this approximation step without adding significant additional computational steps, the authors propose adding Gaussian noise to the perturbation initialization. ", "questions": "Let me rephrase my previously mentioned weakness as a question here. \n\nIt is not entirely clear why performing the inner maximization gradient ascent step on a smoothed loss function necessarily leads to a more \"accurate\" solution or exactly what that means. Following the same argument, we would expect that we want to apply SGD (solving a minimization objective) on a smoothed loss function too. Yet, this is not standard practice. Works like [1] recently proposed to explore such smoothening; however, with the motivation of finding flat minima (akin to SAM), not \"gradient stability\" or more \"accurate\" solutions. It would be great if the authors could clarify why \"gradient stability\" is important for inner maximization. \n\nI'm happy to raise my score if this gets sufficiently addressed. \n\n[1] Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscape, Bisla et al, AISTATS 2022. ", "limitations": "As far as I can see, the authors do not discuss any limitations of their method. \n", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths \n\n* Significance: Efficient stochastic optimization is fundamental for deep learning, and sharpness-aware methods often (but not always [3]) lead to significantly better generalization performance than previous optimizers. Hence, improving methods like SAM can be very relevant to many optimizer users.\n* Clarity: the paper is well-written and easy to follow. \n* Simplicity: The proposed method is simple, easy to implement, and effective for the shown experiments shown.  \n\n# Weaknesses\n\n\nWhile I'm open to being convinced otherwise and raising my score, I'm not fully convinced yet that the main premise of the paper -- randomly smoothening (RS) the loss function **significantly** **improves** over a non-smoothed loss function for approximating the gradient ascent step -- is true. RS seems very heuristical to me; hence, I'd appreciate either more clarification on why it always improves (if so) or more discussion on when it may not hold and what its trade-offs are. \n\nSAM's inner objective's gradient ascent step involves the same gradient computation as standard SGD. Yet, when using standard SGD, we typically do not additionally smooth the loss function. For example, [4] recently proposed doing so, however, with the motivation of finding flatter minima (akin to SAM) and by sampling multiple perturbations instead of just one. If RS leads to more \"stable\" or \"accurate\" gradients, why not apply it to the descent step too? For example, in Algorithm 1, \"Compute SAM gradient\", why do we not additionally perturb $\\mathbf w_{\\text{adv}}$  as well if such smoothening is always beneficial? \n\nFigure 1 shows Vanilla's cosine similarity between the standard and perturbed gradient decays faster than when RS gets applied. However, it is not clear to me why this matters and can't be simply resolved by re-scaling rho.\n\nFigure 2 shows that the RS loss function increases faster in loss when rho is increased. Again, it is unclear why reaching higher losses by perturbing the weights further and further with increasing standard deviations is strictly better or important for ultimately reaching flatter (and therefore, better generalizing) minima.\n\nThe authors claim that Vanilla can not reach loss values as high as RS reaches (although this is not fully proven as rho's range ends at $\\rho=1.0$). However, first perturbing the weights before approximating the sharpest loss in their neighborhood effectively moves the neighborhood ball in a worse-performing region -- it is not centered around $\\mathbf w$ but at $\\mathbf w + \\mathbf \\delta_0$. Since $\\mathbf \\delta_0 \\neq \\nabla L(\\mathbf w)$ , almost surely $L(\\mathbf w) < L(\\mathbf w + \\mathbf \\delta_0)$ . Therefore, it is not surprising that an ascent step starting from $L(\\mathbf w + \\mathbf \\delta_0)$ will reach even higher losses. \n\nFurther, we may note that the authors generated Figure 2 using the checkpoint of the 200th epoch on CIFAR-100. Using a decent learning rate/schedule, 200 epochs are enough for training a Resnet-18/WRN to (almost) convergence on CIFAR100. Hence, it is not clear why we want to reach higher-loss regions at this point during training. \n\nLastly, the loss function/y-axis of Figure 2 is not precisely specified. Assuming that it is sth like the cross-entropy loss and based on my experience, my guess would be that parameterizations with losses above 3 (which is what vanilla still reaches) resemble the performance of randomly initialized networks or within the first few optimization steps epochs during training. That is when the parameters are far from fruitful valleys and hence, not where we want the gradient ascent step jumping towards.  \n\nDespite these issues, I'd be easily convinced of RSAM's utility if the empirical results showed its superiority against other SAM variants and versatility across different tasks. Unfortunately, such results are largely missing.\n\n* While the empirical performance gains over SAM are consistent across the shown experiments, they are less than 1% on average, which resembles the performance gains of other SAM variants, such as GSAM (cited in the paper), ASAM [1], SAM+SWA[3].  However, only for one task (model/dataset combination), the authors compare against GSAM, in which their performance differs by 0.1% and is identical (as the authors say themselves). \n* I appreciate the additional R-GSAM experiment, but again, I find the number of tasks (which is only one here) too limited. It would be great to see how R-SAM compares against GSAM, ASAM, SWA+SAM and on more tasks, and even if they reach similar performances; it would then be interesting e.g. to combine these and see if they are complementary.\n* This method introduces another hyper-parameter $\\gamma$, which may incur additional tuning overhead for practictioners. While Figure 3c) shows that it is rather stable for three different values on one particular dataset, it would be nice to see a more thorough sensitivity analysis with e.g. more values or a default value used on a few more datasets/architectures (like SAM's rho=0.05).\n* The paper includes thorough image classification results; however, it lacks empirical analyses on other tasks such as NLP or graph learning [2,3]. \n* All empirical results seem to be the result of one training run. It would great to see averaged results across at least three random seeds $\\pm$ std. \n\nMinor: \n* Figures 1 and 2 show the effect of random smoothing (RS) on the loss surface. Since RS is a stochastic operation, it would be great to see some empirical confidence intervals (e.g., $\\pm 1$ std across 3 seeds).   \n\n\n\n[1] ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks, Kwon et al, ICML 2021.\n[2] Sharpness-Aware Minimization Improves Language Model Generalization, Bahri et al, ACL 2022.\n[3] A Fair Comparison of Two Popular Flat-Minima Optimizers: Stochastic Weight Averaging vs. Sharpness-Aware Minimization, Kaddour et al, arXiv:2202.00661.\n[4] Low-Pass Filtering SGD for Recovering Flat Optima in the Deep Learning Optimization Landscape, Bisla et al, 2022. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "BJxQxxDvN9j", "reviewer": "Reviewer_tXqv", "summary": "This work analyzes the approximation quality of the inner maximization term in Sharpness-Aware Minimization (SAM). It is shown that the unstable gradients can hurt the inner maximization. A novel method called Random Sharpness-Aware Minimization (R-SAM) is proposed that smooths the landscape, enabling a better approximation of the inner maximization. This is done by the addition of Gaussian Noise before the computation of the inner maximization term. The results are shown on CIFAR-10, CIFAR-100, and ImageNet datasets. R-SAM improves upon the performance of SAM in most settings.", "questions": "1. In Figures 1 and 2, it would be clearer if the authors also mentioned the model's accuracy in a Table. For instance, $\\rho$ = 1.0 leads to the max loss value in Figure 2. Is that observed in the performance as well? \n(Also, why is the scale of the rho value in both the figures very different?)\n\n2. The experimental details of the CIFAR-100 experiments, including the $\\rho$ value, seem to be missing. A note comparing the $\\rho$ value with SAM and R-SAM can be added. It is to be observed whether training with high $\\rho$ is also stable for ResNets. With ViTs (Table 6), the optimal $\\rho$ value is higher for R-SAM.\n\n3. The details regarding the value of $\\gamma$ have not been mentioned. A sensitivity analysis of the value of $\\gamma$ can also be added.\n\n4. Question 3c of the checklist is answered as, \"It would be too computationally expensive for us. We conduct all the experiments at least three times and report the average value.\" This is a bit confusing. I just wanted to clarify whether the authors have run all the experiments three times. This is significant because the increase with R-SAM is small in many experiments and within the error bound (as mentioned in SAM).\n\n4. In the results section, it is unclear why the authors are discussing the improvement of the original SAM over SGD+M and AdamW (L233-238, L246-250, L261-264). These results have been discussed in the SAM work.  \n\n5. The method proposed by Chaudhari et al. is called Entropy-SGD and not \"Energy-SGD,\" as mentioned by the authors. (L53)", "limitations": "The limitations of the work is not mentioned. This work has no potential negative societal impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The analysis of the effect of Gaussian noise on the perturbed loss-value (Figure 2) is interesting. The analysis of approximation of gradient in SAM has not been studied and is a good direction.\n2. The proposed method, R-SAM, is simple, has no additional cost compared to SAM, and allows for stable training with large $\\rho$.\n3. The results on combining G-SAM with R-SAM, resulting in R-GSAM, show that the R-SAM can be integrated with other variants of SAM.\n\nWeaknesses:\n1. Relationship between stable gradient and approximation of inner maximization term is not very clear. Also, the term stable gradient is not well-defined. If a stable gradient were the goal, then $\\rho$ = 0 (SGD) or a low $\\rho$ would lead to the most stable gradient according to the cosine definition. However, it is observed that a very low $\\rho$ value does not lead to a significant gain in accuracy over SGD.\n3. LPF-SGD [1] is a recently proposed method that aims to reach flat optima. The algorithm of LPF-SGD also adds Gaussian Noise to the parameters with the variance equal to the norm of filter weights, similar to R-SAM.  \n4. The increase in accuracy of R-SAM in comparison to SAM is minimal in most settings (Table 1, Table 2, Figure 3a).\n\n[1] Devansh Bisla, Jing Wang, and Anna Choromanska. Low-pass filtering sgd for recovering flat optima in the deep learning optimization landscape. arXiv preprint arXiv:2201.08025, 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept