Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approach
Authors: Peng Mi
Li Shen
Tianhe Ren
Yiyi Zhou
Xiaoshuai Sun
Rongrong Ji
Dacheng Tao
Venue: NeurIPS 2022 Accept (2022)
Keywords: Sharpness-Aware Minimization
Fisher information
Dynamic sparse training

Abstract: Deep neural networks often suffer from poor generalization caused by complex and non-convex loss landscapes. One of the popular solutions is Sharpness-Aware Minimization (SAM), which smooths the loss landscape via minimizing the maximized change of training loss when adding a perturbation to the weight. However, we find the indiscriminate perturbation of SAM on all parameters is suboptimal, which also results in excessive computation,~\emph{i.e.}, double the overhead of common optimizers like Stochastic Gradient Descent~(SGD). In this paper, we propose an efficient and effective training scheme coined as Sparse SAM (SSAM), which achieves sparse perturbation by a binary mask. To obtain the sparse mask, we provide two solutions which are based on Fisher information and dynamic sparse training, respectively. In addition, we theoretically prove that SSAM can converge at the same rate as SAM,~\emph{i.e.}, $O(\log T/\sqrt{T})$. Sparse SAM not only has the potential for training acceleration but also smooths the loss landscape effectively. Extensive experimental results on CIFAR10, CIFAR100, and ImageNet-1K confirm the superior efficiency of our method to SAM, and the performance is preserved or even better with a perturbation of merely 50\% sparsity. Code is available at \url{https://github.com/Mi-Peng/Sparse-Sharpness-Aware-Minimization}.

Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approach
Peng Mi Li Shen Tianhe Ren'
Yiyi Zhou
Xiaoshuai Sun' Rongrong Ji Dacheng Tao
Media Analytics and Computing Laboratory, Department of Artificial Intelligence, School of Informatics, Xiamen University, China 2JD) Explore Academy, Beijing, China Thhe University of Sydney, Australia mi mipeng@stu. xmu, edu. en, mathshenli@gmail com, rentianhe@stu, xmu edu. cn zhouyiyi@mma edu. en, xssun@xmu edu. rrjiuxmu. edu. en, dacheng www.gtttttmmmccocmmmmm
Abstract
Deep neural networks often suffer from poor generalization caused by complex and non-convex loss landscapes. One of the popular solutions Sharpness- Aware Min- imization (SAM), which smooths the loss landscape via minimizing the maximized change of training loss when adding a perturbation to the weight. However, we find the indiscriminate perturbation of SAM on all parameters is suboptimal, which also results excessive computation, i.e., double the overhead of common optimizers like Stochastic Gradient Descent (SGD). In this paper, propose efficient and effective training scheme coined as Sparse SAM (SSAM), which achieves sparse perturbation by binary mask. To obtain the sparse mask, provide two solutions which are based onFisher information and dynamic sparse training, respectively. In addition,
theoretically prove that SSAM can converge at the same rate as SAM, i.e., Ollog T/ TT) Sparse SAM not only has the potential for training acceleration but also smooths the loss landscape effectively. Exten- sive experimental results on CIFARIO. CIFARI00 and ImageNet- 1K confirm the superior efficiency of our method to SAM, and the performance is preserved or even better with perturbation of merely 50% sparsity. Code available https: //gi giithub.... com/ Mi Peng/ Sttr//Sparreeeeeeeeeeeeeeeeeeeee Saarrse--haapnsssseee Aware Min Minimization.
Introduction
Over the past decade or the great success of deep learning has been due in great part to ever-larger model parameter sizes [13, 56, 52, 10, 40, 5J. However, the excessive parameters also make the model inclined to poor generalization. To overcome this problem, numerous efforts have been devoted to training algorithm [24, 42, 50], data augmentation [11, 57, 55], and network design [26, 28].
One important finding in recent research is the connection between the geometry of loss landscape and model generalization [31, 17, 27, 44, 53]. In general, the loss landscape of the model is complex and non non-convex, which makes model tend to converge to sharp minima. Recent endeavors [31, 27, 44] show that the flatter the minima of convergence, the better the model generalization. This discovery reveals the nature of previous approaches [24, 26, 11, 55, 57, 28] to improve generalization, i.e., smoothing the loss landscape.
"This work was done during an internship at JD Explore Academy. 'Li Shen the corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Based on this finding, Foret et al. [17] propose a novel approach to improve model generalization called sharpness-aware minimization (SAM). which simultaneously minimizes loss value and loss sharpness. SAM quantifies the landscape sharpness as the maximized difference of loss when perturbation is added to the weight. When the model reaches a sharp area, the perturbed gradients in SAM help the model jump out of the sharp minima. In practice, SAM requires two forward-backwo computations for each optimization step, where the first computation is to obtain the perturbation and the second one is for parameter update. Despite the remarkable performance [17, 34, 14, 7J, This property makes SAM double the computational cost of the conventional optimizer, e.g., SGD [3].
Since SAM calculates perturbations indiscriminately for all parameters, question is arisen:
Do we need to calculate perturbations for all parameters?
Above all, we notice that in most deep neural networks, only about 5% of parameters are sharp and rise steeply during optimization [31]. Then we explore the effect of SAM in different dimensions to answer the above question and find out (i) little difference between SGD and SAM gradients in most dimensions (see Fig. 1); (ii) more flatter without SAM in some dimensions (see Fig. 4 and Fig. 5).
Inspired by the above discoveries, we propose a novel scheme to improve the efficiency of SAM via sparse perturbation, termed Sparse SAM (SSAM). SSAM, which plays the role of regularization, has better generalization, and its sparse operation also ensures the efficiency of optimization. Specifically, the perturbation in SSAM is multiplied by a binary sparse mask determine which parameters should be perturbed. To obtain the sparse mask, we provide two implementations. The first solution is to use Fisher information [16] of the parameters to formulate the binary mask, dubbed SSAM-F. The other one is to employ dynamic sparse training to jointly optimize model parameters and the sparse mask, dubbed SSAM-D. The first solution relatively more stable but bit time-consuming. while the latter is more efficient.
In addition to these solutions, provide the theoretical convergence analysis of SAM and SSAM in non-convex stochastic setting, proving that our SSAM can converge at the same rate as SAM, i.e., Ollog T/VT). At last, we evaluate the performance and effectiveness of SSAM on CIFARIO [33], CIFAR 100 [33] and ImageNet [8] with various models. The experiments confirm that SSAM contributes to a flatter landscape than SAM, and its performance on par with or even better than SAM with only about 50% perturbation. These results coincide with our motivations and findings.
To sum up, the contribution of this paper is three-fold:
We rethink the role of perturbation SAM and find that the indiscriminate perturbations are suboptimal and computationally inefficient.
We propose a sparsified perturbation approach called Sparse SAM (SSAM) with two variants, i.e., Fisher SSAM (SSAM-F) and Dynamic SSAM (SSAM-D), both of which enjoy better efficiency and effectiveness than SAM. We also theoretically prove that SSAM can converge at the same rate as SAM, i.e., Olog T/ T/VT).
We evaluate SSAM with various models CIFAR and ImageNet, showing WideResNet with SSAM of a high sparsity outperforms SAM on CIFAR; SSAM can achieve competitive performance with high sparsity; SSAM has comparable convergence rate SAM.
2
Related Work
In this section, we briefly review the studies on sharpness-aware minimum optimization (SAM), Fisher information in deep learning, and dynamic sparse training.
SAM and flat minima. Hochreiter et al. [27] first reveal that there is strong correlation between the generalization of a model and the flat minima. After that, there is a growing amount of research based on this finding. Keskar et al. [31] conduct experiments with a larger batch size, and in consequence observe the degradation of model generalization capability. They [31] also confirm the essence of this phenomenon, which is that the model tends to converge to the sharp minima. Keskar al. [3 1] and Dinh et al. [12] state that the sharpness can be evaluated by the eigenvalues of the Hessian. However, they fail to find the flat minima due to the notorious computational cost of Hessian.
Inspired by this, Foret et al. [17] introduce a sharpness-awart optimization (SAM) to find flat minimum for improving generalization capability, which is achieved by solving a mini-max problem.

---

Zhang et al. [58] make point that SAM [17] is equivalent to adding the regularization of the gradient norm by approximating Hessian matrix. Kwon al. [34] propose a ccale-invariam, SAM scheme with adaptive radius to improve training stability. Zhang et al. [59] redefine the landscape sharpness from an intuitive and theoretical perspective based on SAM. To reduce the computational cost in SAM. Du et al. [14] proposed Efficient SAM (ESAM) to randomly calculate perturbation. However, ESAM randomly select the samples every steps, which may lead to optimization bias. Instead of the perturbations for all parameters, i.c., SAM, we compute sparse perturbation, i.e., SSAM, which learns important but sparse dimensions for perturbation.
Fisher information (FT). Fisher information [16] was proposed to measure the information that an observable random variable carries about an unknown parameter of a distribution. In machine learning, Fisher information is widely used to measure the importance of the model parameters [32] and decide which parameter to be pruned [49, 51J. For proposed SSAM-F, Fisher information is used to determine whether weight should be perturbed for flat minima.
Dynamic sparse training. Finding the sparse network via pruning unimportant weights a popular solution in network compression, which can be traced back to decades [35]. The widely used training scheme, i.e., petraaningg-runnng-fieee
is presented by Han et.al. [23]. Limited by the
requirement for the pre-trained model, some recent research [15, 2, 9, 30, 43, 37, 38] attempts to discover a sparse network directly from the training process. Dynamic Sparse Training (DST) finds the sparse structure by dynamic parameter reallocation. The criterion of pruning could be weight magnitude [18], gradient [15] and Hessian [35, 48], etc. We claim that different from the existing DST methods that prune neurons, our target is to obtain binary mask for sparse perturbation.
3 Rethinking the Perturbation in SAM
In this section, we first review how SAM converges at the flat minimum of a loss landscape. Then, we rethink the role of perturbation in SAM.
3.1 Preliminary
W2,..., ..., wa) C W Rd In this paper, we consider the weights of deep neural network as = (wi, W2,
and denote a binary mask as m {0, 1jd, which satisfies 1 (1 s) d restrict the computational cost. Given a training dataset as S A {( ))) 1=1 i.i.d. drawn from the distribution D, the per-data-point loss function is defined by s(w, L;, V.). For the classification task in this paper, we use cross-entropy as loss function. The population loss is defined by Sn (DD..) 2,; y:).
while the empirical training loss function is fs 1 E. f(w, Li, y:).
Sharpness- aware minimization (SAM) [17] aims to simultaneously minimize the loss value and smooth the loss landscape, which is achieved by solving the min-max problem:
min max fs(w + c). |ellaSp
(1)
SAM first obtains the perturbation in neighborhood ball area with a radius denoted as p. The optimization tries to minimize the loss of the perturbed weight w + e. Intuitively, the goal of SAM is that small perturbations to the weight will not significantly rise the empirical loss, which indicates that SAM tends to converge to a flat minimum. To solve the mini-ma max optimization, SAM approximately calculates the perturbations e using Taylor expansion around w:
= arg max fs( + +)) x arg max fsw + e fw.((w= = P. |ee|l-Sp
(2)
|eellsSp
In this way, the objective function can be rewritten as minw fs(w + / (.. which could be implemented by a two-step gradient descent framework in Pytorch or TensorFlow:
In the first step, the gradient at w is used to calculate the perturbation e by Eq. 2. Then the weight of model will be added to + e.
In the second step, the gradient at un + is used to solve min fs(w + E). iee, update the weight w by this gradient.
3

---

3.2 Rethinking the Perturbation Step of SAM
How does SAM work in flat subspace? SAM perturbs all parameters indiscriminately, but the fact that merely about 5% parameter space is sharp while the rest is flat [31]. We are curious whether perturbing the parameters in those already flat dimensions would lead to the instability of the optimization and impair the improvement generalization. To answer this question, we quantitatively and qualitatively analyze the loss landscapes with different training schemes in Section 5, as shown in Fig. and Fig. 5. The results confirm our conjecture that optimizing some dimensions without perturbation can help the model generalize better.
What is the difference between the gradients of SGD and SAM? We investigate various neural networks optimized with SAM and SGD on CIFAR 10/100 and ImageNet, whose statistics are given in Fig. We use the relative difference ratio r, defined as log (gS gSGD) (SSD))/sGDl. gsGD). to measure the difference between the gradients of SAM and SGD. As showin in Fig. 1, the parameters with less than 0 account for the vast majority of all parameters, indicating that most SAM gradients are not significantly different from SGD gradients. These results show that most parameters of the model require no perturbations for achieving the flat minima, which well confirms our the motivation.
(81.0%)
84.2%
leegesteelaageNet Delis Tlny Klmaggeee
8:28
0.:9
8.30
(99.8%
Dunslty
Dendly
1.25
Density
8:38-
0.15
8.15
6:18
6/4
0.99
8.40
Ratio
Ratio
Ratio
Figure 1: The distribution of relative difference ratio r among various models and datasets. There is little difference between SAM and SGD gradients for most parameters, i.e., the ratio is less than 0.
Inspired by the above observation and the promising hardware acceleration for sparse operation modern GPUs, we further propose Sparse SAM, novel sparse perturbation approach, an implicit regularization to improve the efficiency and effectiveness of SAM.
4 Methodology
In this section, we first define the proposed Sparse SAM (SSAM), which strengths SAM via sparse perturbation. Afterwards, we introduce the instantiations of the sparse mask used SSAM via Fisher information and dynamic sparse training, dubbed SSAM-F and SSAM-D, respectively.
4.1 Sparse SAM
Motivated by the finding discussed in the introduction, Sparse SAM (SSAM) employs sparse binary mask to decide which parameters should be perturbed, thereby improving the efficiency of sharpness-awee aware minimization. Specifically, the perturbation e will be multiplied by a sparse binary
mask m. and the objective function is then rewritten as min fs ( + p
f 3 m To
[..(()))
stable the optimization, the sparse binary mask m is updated at regular intervals during training. We provide two solutions to obtain the sparse mask m. namely Fisher information based Sparse SAM (SSAM-F) and dynamic sparse training based Sparse SAM (SSAM-D). The overall algorithms of SSAM and sparse mask generations are shown in Algorithm 1 and Algorithm 2, respectively.
According the previous work [54] and Ampere architecture equipped with sparse tensor cores [47, 46, 45], currently there exists technical support for matrix multiplication with 50% fine-grained sparsity [54]². Therefore. SSAM of 50% sparse perturbation has great potential to achieve true training acceleration via sparse back-propagation
For instance, 2:4 sparsity for A100 GPU.

---

Algorithm 1 Sparse SAM (SSAM)
Algorithm Sparse Mask Generation
Input:
sparse ratio s. dense model w, binary mask m,, update interval Tm, number of sam-
1: Option I:(Fisher Information Mask)
2: Sample Np data from S: BF
ples Ny, learning rate 7) training set S. Initialize and m randomly.
3: Compute Fisher Fw by Eq. (5)
1:
4: (m. Em ArgTopK /-wl)
2: for epoch 1 1.2... do for each training iteration
5:
mm,00mm Em] {m. m. $ m2)
6: m t- mo U m2
4: Sample a batch from S: B
7: Option I1:(Dynamic Sparse Mask)
5: Compute perturbation by Eq. (2)
8: Narop- = (decou o) (11)))1111111 (1-8) 8) [m]]
6: if mod T.. = 0 then
9: N growth Ndrop
Generate mask m via Option I
10: m. {m- = 1m, m } [m, 1/m, m)
8: end if
ArgTopK mm (-|V
V- (w) Narop)
+ E m 10: end for
9:
11: mı {m, mm, m) + Randomm.gmm (N (Ngrowtth)
12: mo = mm,=0mm mm + {m, $ mm1)
11: w
v/ fw
13: m +- mg m1
12:
end for
14: return Sparse mask 772
13:
return Final weight of model w
4.2 SSAM-F:: Fisher information based Sparse SAM
Inspired by the connection between Fisher information and Hessian [16], which can directly measure the flatness of the loss landscape, we apply Fisher information to achieve sparse perturbation, denoted as SSAM-F. The Fisher information defined by
Fw Ex)p(()) [Ey-pw(u m) V Po Pw (y) 2)) V log Pw (y)x)²]
(3)
where Pwo (yl2z) is the output distribution predicted by the model. In over-parameterized networks, the computation of Fisher information matrix is also intractable, i.e., /w| |wr/. Following [49, 19, 22], we approximate Fw diagonal matrix, which is included in vector in R Ri Note that there the two expectation in Eq. (3). The first one is that the original data distribution p(x) is often not available. Therefore, we approximate by sampling NF data æ ........ NN from æ ~ p(z):
F 1 Np (ulo)) V. log Per Por (y)
For the second expectation over Pno (1/ x)), it is not necessary to compute its explicit expectation, since the ground-truth y: for each training sample 2n is available in supervised learning. Therefore rewrite the Eq. (4) as "empirical Fisher":
F
(V log (x)lz:))?
(5)
NF
We emphasize that the empirical Fisher is a |w]- -imeennnnoo vector, which is the same the mask m. To obtain the mask m, we calculate the empirical Fisher by Eq. (5) over NF training data randomly sampled from training set S. Then we sort the elements of empirical Fisher in descending, and the parameters corresponding to the the top ie Fisher values will be perturbed:
(6) where Arg (oyKo,, .N returns the index of the top N largest values among v. my is the set of values that are in m. k is the number of perturbed parameters, which is equal (1 -s) | for sparsity s. After setting the rest values of the mask to 0, ie.. mo {m, = 0Jm, & mi), we get the final mask m mo U my. The algorithm of SSAM-F shown in Algorithm 2.
m1 = {m, 1 1/m, E m) < HrgTopK(Fw...
4.3 SSAM-D: Dynamic sparse training mask based sparse SAM
Considering the computation of empirical Fisher is still relatively high, we also resort to dynamic sparse training for efficient binary mask generation. The mask generation includes the perturbation dropping and the perturbation growth steps. At the perturbation dropping phase, the flattest dimensions of the perturbed parameters will be dropped, i.e., the gradients of lower absolute values, which means that they require no perturbations. The update of the sparse mask follows
mi {m, 1 1|m m) <- m2 ArgTopkwem. (-IVf(w)). Narop),
5

---

where Ndrop is the number of perturbations to be dropped. At the perturbation growth phase, for the purpose of exploring the perturbation combinations as many as possible, several unperturbed dimensions grow, which means these dimensions need to compute perturbations. The update of the sparse mask for perturbation growth follows
m
{m; = 1 m, m) + mı + Randomangm (N (Ngrowth),
(8)
where Randoms (N) randomly returns N indexes in S, and N Ngrowth is the number of perturbation growths. To keep the sparsity constant during training, the number of growths is equal to the number of dropping, i.e., N Nyrowth-
Nrrop- Afterwards, we set the rest values of the mask to i.e., mo = [m, = =0mm; & m11) J. and get the final mask m mg U m., The drop ratio represents the proportion of dropped perturbations in the total perturbations 8 |/rl, i.e., a Ndrop/ (s /w)). In particular, a larger drop rate means that more combinations of binary mask can be explored during optimization, which, however, may slightly interfere the optimization process. Following [15, 9], we apply a cosine decay scheduler to alleviate this problem:
deccyy (1; or)
OF (1 + cos (tx/T))
2
where 7' denotes number of training epochs. The algorithm of SSAM-D is depicted in Algorithm 2.
4.4 Theoretical analysis of Sparse SAM
In the following, we analyze the convergence SAM and SSAM in non-convex stochastic setting. Before introducing the main theorem, we first describe the following assumptions that are commonly used for characterizing the convergence of nonconvex stochastic optimization [39, 59, 1, 6, 4, 20).
Assumption 1. (Bounded Gradient.) exists G 2 s.t. [V < G.
Assumption 2. (Bounded Variance.) It exists a 2 0 s.t. [[]g(()) - V/(ww) 19] o².
Assumption 3. (L-smoothness.) It exists L > 0 s.t. [ Vf(v)]l - L|| 111. Vw, R... Theorem Consider function f(w) satisfying the Assumptions 1-3 optimized by SAM. Let t) vi and perturbation amplitude decay with square root of e.g. P = 2g
With Pa S GToo, we have Vt
Σ E E
T (w:)l
C1
1 VT
+ C2
log
(10)
T
VT
where C = %u
(f(w)) and C2 = 2(Lo² no + LGPo).
Theorem 2. Consider functiooon f(w) satisfying the Assumptions 1-3 optimized by SSAM. Let T: = 10 Po.
Vi
and perturbation amplitude p decay with square root of e.g. P Vi
With po I Gno/2. we have:
T E B f(w)))l1
log VT
< C3 VT
+ C4
(11)
T
where C3 = 2 - (w)r + p22 + no I.) and C4 = 2(Lo2 + I.Go)..
For non-convex stochastic optimization, Theorems 1&2 imply that our SSAM could converge the same rate SAM, i.e., Ollog T/VT). Detailed proofs of the two theorems given in Appendix.
5
Experiments
In this section, we evaluate the effectiveness of SSAM through extensive experiments on CIFAR10, CIFAR 100 [33] and ImageNet- 1K [8]. The base models include ResNet [25] and WideResNet [56]. We report the main results on CIFAR datasets in Tables &&2 and ImageNet- IK in Table 3. Then, we visualize the landscapes and Hessian spectra to verify that the proposed SSAM can help the model
generalize better. More experimental results are placed in Appendix due to page limit.

---

5.1 Implementation details
Datasets. We use CIFAR IO/CIFA 100 [33] and ImageNet-11 K [8] as the benchmarks of our method. Specifically, CIFARTO and CIFAR 100 have 50,000 images of 32x32 resolution for training, while 10,000 images for test. ImageNet-11 [8] is the most widely used benchmark for image classification, which has 1,281,167 images of 1000 classes and 50,000 images for validation.
Hyper-parameter setting. For small resolution datasets, i.e., CIFARIO and CIFARIO0, we replace the first convolution layer in ResNet and WideResNet with the one of 3 kernel size, 1 stride and 1 padding. The models on CIFAR OO/CIO/AIIIIIII 100 are trained with 128 batch size for 200 epochs. We apply the random crop, random horizontal flip, normalization and cutout [11] for data augmentation, and the initial learning rate is 0.05 with a cosine learning rate schedule. The momentum and weight decay of SGD are set to 0.9 and 5e-4, respectively. SAM and SSAM apply the same settings, except that weight decay is set to 0.001 [14]. We determine the perturbation magnitude from (0.01, 0.02, 0.05, 0,1, 0.2, 0.5) via grid search. In CIFARIO and CIFAR 100, we set p as 0.1 and 0.2, respectively. For ImageNet- IK, we randomly resize and crop all images to resolution of 224 x 224, and apply random horizontal flip, normalization during training. We train ResNet with a batch size of 256, and adopt the cosine learning rate schedule with initial learning rate 0.1. The momentum and weight decay of SGD set as 0.9 and le-4. SAM and SSAM use the same settings as above. The test images of both architectures are resized to 256x256 and then centerly cropped to 224x224. The perturbation magnitude p is set 0.07.
5.2 Experimental results
Results on CIFARIO/CIEAR100. We first evaluate our SSAM on CIFAR-10 and CIFARIO0. The models we used are ResNet-18 [25] and WideResNee-28-100 [56]. The perturbation magnitude p for SAM and SSAM are the same for fair comparison. As shown in Table 1, ResNet18 with SSAM of 50% sparsity outperforms SAM of full perturbations. From Table 2, we can observe that the advantages of SSAM-F and SSAM-D performance than SAM with up to 95% sparsity. Note that the parameter size of WideResNet28 is much larger than that of ResNet- 18, and CIFAR is often easy overfit. In addition, even with very large sparsity, both SSAM-F and SSAM-D can still obtain competitive performance against SAM.
WideResNet28
more significant, which achieve better
Table 1: Comparison between SGD, SAM and SSAM CIFAR using ResNet- 18
Model
Optimizer
Sparsity
CIFARIO
CIFAR100
FLOPs*
SGD
96.07%
77.80%
1x
SAM
0%
96.83%
81.03%
2x
50%
96.81% (-0.02)
81.24% (+0.21)
I.63 x
80%
96.64% (-0.19)
80.47% (-0.56)
1.44x
90%
96.75% (-0.08)
80.02% (-1.01)
1.36x
SSAM-F (Ours)
95%
96.66% (-0.17)
80.50% (-0.53)
1.33
98%
96.55% (-0.28)
80.09% (-0.94)
1.31
ResNet18
99%
96.52% (-0.31)
80.07% (-0.96)
1.30x
50%
96.87% (+0.04)
80.59 (-0.44)
1.63X
80%
96.76% (-0.07)
80.43% (-0.60)
1.44X
90%
96.67% (-0.16)
80.39% (-0.64)
L.6xX
SSAM-D (Ours)
95%
96.56% (-0.27)
79.79% (-1.24)
1.33
98%
96.61% % (-0.22)
79.79% (-1.24)
1.31x
99%
96.59% (-0.24)
79.61% (-1.42)
1.30x
Results on ImageNet. Table reports the result of SSAM-F and SSAM-D on the large-scale ImageNet-IK [8] dataset. The model we used is ResNet50 [25]. We can observe that SSAM-F and SSAM-D can maintain the performance with 50% sparsity. However, as sparsity ratio increases. SSAM-F will receive relatively obvious performance drops while SSAM-D is more robust.
Training curves. We visualize the training curves of SGD, SAM and SSAM in Fig. 2. The training curves of SGD are more jittery, while SAM is more stable. In SSAM, about half of gradient updates are the same as SGD, but its training curves are similar to those of SAM, suggesting the effectiveness.
'The FLOPs theoretically estimated by considering sparse multiplication compared to SGD in Tables 1-
7

---

Table 2: Comparsion between SGD, SAM and SSAM on CIFAR using WideResNee-28-10
Model
Optimizer
Sparsity
CIFARIO
CIFARI00
FLOPs
SGD SAM
97.11%
81.93%
1x
97.48%
84.20%
50%
97.71% (+0.23)
85.16% (40.96)
I.65x
80%
97.67% (+0.19)
84.57% (+0.37)
1.44x
WideResNer28- 10
90%
97.47% (-0.01)
84.76% (+0.56)
1.36x
SSAM-F (Ours)
95%
97.42% (-0.06)
84.17% (-0.03)
1.33x
98%
97.32%
(-0.16)
83.85% (-0.35)
1.31
99%
97.59%
(40.11)
84.00% (-0.20)
1.30x
50%
97.70%
(40.22)
84.99 (40.79)
1.65X
80%
97.72%
(+0.24)
14.3%%
1.44x
90%
97.53%
(+0.05)
(40.64))
1.36x
SSAM-D (Ours)
95%
97.52%
(+0.04)
83.66% (-0.54)
1.33x
98%
97.30%
(-0.18)
83.30% (-0.90)
1.31x
99%
97.27% (-0.25)
84.16% (-0.04)
1.30
Table Comparsion between SGD, SAM and SSAM on ImageNet- IK using ResNet50.
Model
Optimizer
Sparsity
ImageNet
FLOPs
SGD
76.67%
1x
SAM
0%
77.25%
2x
50%
77.31% (+0.06)
1.65x
SSAM-F (Ours)
80%
76.81 (--.44)))
1.44x
ResNet50
90%
76.74%((-055)
1.36x
50%
(-0.00))0
I.65X
SSAM-D (Ours)
80%
77.00% (-0.25)
1.44x
90%
77.00%((-0.25)
1.36x
160
Test Accuracy (%)
Test Accuracy (%)
Text Accuua 000
90
60
2
SAM
SGD
SGD
85
&
SSAM-D SSAM-R
50-
SSAM-F
SSAM-F
50
160 Epoch
150
260
50
100
150
200
30
60
90
Epoch
Epoch
ResNet18 CIFARIO
(b) ResNet18 CIFAR 100
(c) ResNer50 ImageNet-IK
Figure 2: The training curves of SGD, SAM and SSAM. The sparsity of SSAM is 50%.
Sparsity vs. Accuracy. We report the effect sparsity ratios in SSAM, as depicted in Fig. 3. We can observe that on CIFAR datasets, the sparsities of SSAM-F and SSAM-D pose little impact performance. In addition, they can obtain better accuracies than SAM with up to 99% sparsity. On the much larger dataset, i.e., ImageNet, higher sparsity will lead more obvious performance drop.
98.0
86
77.6
SSAM-F SSAM-D
SSAM-F
Accuracy (%)
97.8
85
SAMI
77.4
SSAM-D
97.6
77.2
SAM
97.2
83
SSAM-F SSAM-D
SGD
76.8
SGD
82
0.5
0.8
0.9
0.95
0.98
0.99
0.5
0.8
0.9
0.95
0.98
0.99
0.5
0.8
0.9
Sparsity
Sparsity
Sparsity
(a) WideResNet CIFARIO.
(b) WedeResNet CIFARI00.
ResNet50 ImageNet.
Figure 3: Accuarcy v.s. sparsity CIFAR10, CIFAR 100 and ImageNet datasets.

---

5.3 SSAM with better generalization
Visualization of landscape. For a more intuitive comparison between different optimization schemes, we visualize the training loss landscapes of ResNet18 optimized by SGD, SAM and SSAM as shown Fig. Following [36], we sample 50 50 points in the range of [-1, from random "filter normalized" [36] directions, Le., the and y axes. As shown in Fig. 4, the landscape of SSAM is flatter than both SGD and SAM, and most of its area is low loss (blue). This result indicates that SSAM can smooth the loss landscape notably with sparse perturbation, and it also suggests that the complete perturbation on all parameters will result suboptimal minima.
15
3.99
20 -3.58
3.0
20
SAM
SSAM
Figure 4: Training loss landscapes of ResNet18 CIFAR1O trained with SGD, SAM, SSAM.
Hessian spectra. In Fig. 5, we report the Hessian spectrum to demonstrate that SSAM can converge to flat minima. Here, we also report the ratio of dominant eigenvalue to fifth largest ones, i.e., A1 / used as the criteria [17, 29]. We approximate the Hessian spectrum using the Lanczos algorithm [21] and illustrate the Hessian spectra of ResNet18 using SGD, SAM and SSAM on CIFARIO. From this figure, we observe that the dominant eigenvalue A1 with SSAM is less than SGD and comparable to SAM. It confirms that SSAM with sparse perturbation can still converge to the flat minima as SAM does, or even better.
10.
Åι -2.66
À: = 26.2
14*
d = 19.4
Deasity
A1/A5 16.9
Denslty
18*
Àz/Aş = =.0
Density
t0"
Az/ds = 1.7
10"
10
31
3
30
15
0.
20
Eigenvlaue
Eigenvlaue
Eigenvlaue
SGD Hessian spectrum
(b) SAM Hessian spectrum
SSAM Hessian spectrum
Figure 5: Hessian spectra of ResNet18 using SGD, SAM and SSAM on CIFARIO.
6 Conclusion
In this paper, we reveal that the SAM gradients for most parameters are not significantly different from the SGD ones. Based on this finding, we propose an efficient training scheme called Sparse SAM, which is achieved by computing a sparse perturbation. We provide two solutions for sparse perturbation, which are based on Fisher information and dynamic sparse training, respectively. In addition, we also theoretically prove that SSAM has the same convergence rate as SAM. We validate SSAM extensive datasets with various models. The experimental results show that retaining much better efficiency, SSAM can achieve competitive and even better performance than SAM.
eknowweegemment
This work is supported by the Major Science and Technology Innovation 2030 "Brain Science and Brain-like Research" key project (No. 2021ZD0201403 the National Science Fund for Distinguished Young Scholars (No. 62025603), the National Natural Science Foundation China (No. U21B2037, No. 62176222, 62176223, No. 62176226, No. 62072386. No. 62072387, No. 62072389, and No. 62002305), Guangdong Basic and Applied Basic Research Foundation (No. 201981515120044 and the Natural Science Foundation of Fujian Province of China (No. 2021301002).

---

References
[1] Maksym Andriushchenko and Nicolas Flammarion. Understanding sharpness-aware minimization. 2021.
[2] Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein. Deep rewiring: Training very sparse deep networks. arXiv preprint arXi:: 2017.
[3] Léon Bottou. Large-scale machine learning with stochastic gradient descent. In Proceedings of COMP- STAT"2010, pages 177-186. Springer, 2010.
[4] Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. Siam Review, 60(2):23 2112,11122223 2018.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809. 1009, 2018.
[6] Congliang Chen, Li Shen, Fangyu Zou, and Wei Liu. Towards practical adam: Non-convexity, convergence theory, and mini-hatch acceleration. Journal of Machine Learning Research, 23:1-47, 2022.
[7] Xiangning Chen, Cho-Jui Hsieh, and Boqing Gong. When vision transformers outperform resnets without pre- pre-training or strong data augmentations. arXiv preprint arXiv:2106.01544 2021.
[8] Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: large-scale hicrarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-24 248-255. leee, 2009.
[9] Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprint arXiv: arXiv:1907.04840, 2019.
[10] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint 18X1::1810004805, 2018.
[11] Terrance Veeries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint 0rXiv:1708.04533 2017.
[12] Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep In International Conference on Machine Learning, pages 1019-1028. PMLR, 2017.
[13] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
[14] Jiawei Du, Hanshu Yan, Jiashi Feng, Joey Tianyi Zhou, Liangli Zhen, Rick Siow Mong Goh, and Vincent YF Tan. Efficient sharpness-awe aware minimization for improved training of neural networks. arXiv preprint (xXx:21100031111 2021.
[15] Utku Evei, Trevor Gale, Jacob Menick. Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Muchine Learning, pages 2943-2952. PMLR. 2020.
[16] Ronald A Fisher: On the mathematical foundations of theoretical statistics. Philosophical transactions of the Royal Society of London. Series A, containing papers of a mathematical or physical character, 222 (594-664), 1922.
[17] Pierre Foret, Ariel Kleiner, Hosscin Mobahi, and Behnam Neyshabur. Sharpness-awan minimization for efficiently improving generalization. International Conference on Learning Representations, 2020.
[18] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint N.X.::1883.0333. 2018
[19] Thomas George, César Laurent, Xavier Bouthillier, Nicolas Ballas, and Pascal Vincent. Fast approximate natural gradient descent a kronecker factored eigenbasis. Advances in Neural Information Processing Systems, 31, 2018.
[20] Saced Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization, 23(4):2341-233 2013.
[21] Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao, An investigation into neural net optimization via hessian eigenvalue density, In International Conference on Machine Learning, pages 2232 22211 PMLR, 2019.
10

---

[22] Roger Grosse and James Martens. A kronecker-factored approximate fisher matrix for convolution layers. In International Conference Machine Learning, pages 573-582. PMLR, 2016.
[23] Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv 2015.
[24] Lars Kai Hansen, Jan Larsen, and Torben Fog. Early stop criterion from the bootstrap ensemble. In 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 4, pages 3205-3208. TEEE, 1997.
[25] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-776 2016.
[26] Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Im- proving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.055 028000 2012.
[27] Sepp Hochreiter and Jürgen Schmidhuber. Simplifying neural nets by discovering flat minima. Advances in neural information processing systems, 7, 1994.
[28] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448-456. PMLR, 2015.
[29] Stanislaw Jastrzebski, Maciej Szymczak, Stanislav Fort, Devansh Arpil, Jacek Tabor, Kyunghyun Cho, and Krzysztof Geras. The break-even point on optimization trajectories of deep neural networks. arXiv preprint 2000020002. 2020.
[30] Siddhant Jayakumar, Razvan Pascanu, Jack Rae, Simon Osindero, and Erich Elsen. Top-kast: Top-k always sparse training. Advances Neural Information Processing Systems, 33:20744 20754, 2020.
[31] Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. 2016.
[32] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):521-3356,, 2017.
[33] Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
[34] Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi, Asam: Adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks. In International Conference on Machine Learning, pages 19144 PMLR, 2021.
[35] Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 1989.
[36] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in neural information processing systems, 31, 2018.
[37] Shiwei Liu, Tianlong Chen, Xiaohan Chen, Zahra Atashgahi, Lu Yin, Huanyu Kou, Li Shen, Mykola Pechenizkiy, Zhangyang Wang, and Decebal Constantin Mocanu. Sparse training via boosting pruning plasticity with neuroregeneration. Advances in Neural Information Processing Systems. 34:9908 9922, 2021.
[38] Shiwei Liu, Tianlong Chen, Xiaohan Chen, Li Shen, Decebal Constantin Mocanu, Zhangyang Wang, and Mykola Pechenizkiy. The unreasonable elfectiveness of random pruning: Retur of the most naive baseline for sparse training. In International Conference on Learning Representations, 2021.
[39] Yong Liu, Xiangning Chen, Minhao Cheng, Cho-Jui Hsich, and Yang You. Concurrent adversarial learning for large-batch training. arXiv preprint arXiv:2106.0022 2021.
[40] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
[41] James Martens and Roger Grosse. Optimizing neural networks with kronecker cronecker-factored approximate curvature. In International conference machine learning, pages 2408 2417. PMLR, 2015.
11

---

[42] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley. David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pages 1928-1937. PMLR, 2016.
[43] Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In International Conference on Machine Learning, pages 4646-4655. PMLR, 2019.
[44] Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring generalization deep learning. Advances in neural information processing systems, 30, 2017.
[45] nvidia ampere architecture. Nvidia ampere tensor core.
com/ ennus/
data- hatpeeeeeetrraeeeeeeeeeeeeeee pantttrr/mmerr-rrchteeee. architecturr//
[46] nvidia auto sparsity.
Accelerating sparsity the nvidia ampere architecture. https:
/ //developer.oo dwwnload. avidia.iiiiiiiiiii wtt.a/awo//ooevvooceeooeeoooeo otw.tt/owooooooooooooooooooooooo http///cccoongg//////////rrtttte
s22085 acc.eeeatccciriiiinggg. www.eeeerrriinngpsssttt---ee in the-tteeee-eeeee mniiii- ampere httae-aarrcctccccrreee/////33330 pdd.
[47] nvidia whitepaper. Nvidia a100 tensor core gpu architecture. https://imagee: http://maaaeeiiiiiiiaaaaa wwwiia.iii...m.... com//aa.ccoo/aaoooooooooa/o/oaaa com/ aem- dam/ en www.///tooss//eeeeeeeeeee /dta- center conterr/vidia--a hww.riddiaaaprre--eeeee ar architecture- whitepaper -ddf.
[48] Sidak Pal Singh and Dan Alistarh. Woodfisher: Efficient second-order approximation for neural network compression. Advances in Neural Information Processing Systems. 33:18098-1818 2020.
[49] Yi-Lin Sung, Varun Nair, and Colin A Raffel. Training neural networks with fixed sparse masks, Advances in Neural Information Processing Systems, 34, 2021.
[50] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedingg of the IEEE conference on computer vision and pattern recognition, pages 2818 2826, 2016.
[51] Lucas Theis, Iryna Korshunova, Alykhan Tejani, and Ferenc Huszár. Faster gaze prediction with dense networks and fisher pruning. arXiv preprint arXiv:1801.0578) 2018
[52] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
[53] Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust generalization. Advances Neural Information Processing Systems, 33:2958-2969, 2020.
[54] Weixiang Xu, Xiangyu He, Ke Cheng, Peisong Wang, and Jian Cheng. Towards fully sparse training: Information restoration with spatial similarity. 2022.
[55] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023 6032, 2019.
[56] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arX1v:1605.07146. 2016.
[57] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: veXiv:1710.09412. 2017.
[58] Yang Zhao, Hao Zhang, Xiuyuan Hu. Penalizing gradient norm efficiently improving generalization in deep learning. arXiv preprint arXiv:2202.08550 2022.
[59] Juntang Zhuang, Boqing Gong, Liangzhe Yuan, Cui, Hartwig Adam, Nicha Dvomek, Sekhar Tatikonda, James Duncan, and Ting Liu. Surrogate gap minimization improves sharpness-aware training. arXiv preprint 220: 08065, 2022.
12

---

Checklist
1. For all authors...
(a) Do the main claims made the abstract and introduction accurately reflect the paper's contribu- tions and scope? [Yes]
(b) Did you describe the limitations your work? [No]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [Yes] See Section 4.4.
(b) Did you include complete proofs of all theoretical results? [Yes] The detailed proof is in Appendix.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as URL)? [Yes]
(h) Did you specify all the training details (e.g., data splits, hyperparamcters, how they were chosen)? [Yes] Section 5.1.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [No]
(d) Did you include total amount of compute and the type resources used (e.g.. type GPUs, internal cluster, or cloud provider)? [No]
4. If you using existing assets (e.g.. code, data, models) curating/releasing assets...
(a) your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you'l you're us-
ing/curating? [N/A]
(c) Did you discuss whether the data you using/curating contains personally identifiable informa- tion or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you deseribe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(C) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "pH0s9GyVkPx", "reviewer": "Reviewer_KeX3", "summary": "The problem addressed in this study is that the indiscriminate perturbation manipulated by SAM on all parameters is suboptimal. The authors give two interesting observations that: there is little difference between the updating gradients of SGD and SAM in most dimensions; and there exist certain parameters that will still be flatter in some dimensions even if SAM is not used. \nBased on the observations, they propose an efficient and effective training scheme to obtain sparse perturbations by crafting a binary mask. This paper employs the Fisher Information and dynamic sparse training to select top-K most significant and relative parameters to craft a sparse binary mask for perturbation. \nThe authors verify the effectiveness of their proposed training scheme on CIFAR10/CIFAR100 and ImageNet-1K datasets. The experimental results demonstrate that even with very large sparsity, both SSAM-F and SSAM-D can still obtain competitive performance against SAM. The authors also conduct visualization of landscape and Hessian spectra to show better generalization of SSAM.\n", "questions": "1. The results shown only from sparsity 50% and only for ResNet model. Could the authors have more experimental results on different sparsity and models with different architectures? \n\n2. Why does it appear that employing fewer perturbation in Table 1 is unable to enhance the performance, if according to your purpose, that you can find certain parameters that can make loss flatter without perturbation? \n\n3. As stated in Section 3.2, “merely about 5%parameter space is sharp while the rest is flat”. Do you believe you have correctly identified the 5% parameters that do not require perturbation application through your methods and that experimental results demonstrate this? \n\nI am willing to increase my rating if my questions could be my addressed properly. \n", "limitations": "The authors haven’t addressed the limitations and potential negative societal impact of this work. It would be better to have discussion on SSAM’s effectiveness of more different datasets and model structures, as well as the potential future work of SSAM. ", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\n1. The motivation is transparent and derived from the experimental observations. The authors accurately identify the indiscriminate perturbation in previous sharpness-related works. The motivational analysis and experiments are persuasive to me. \n2. The written is straightforward and easy to understand. In addition, the landscape visualization is also informative from a motivational standpoint. \n\nWeakness: \n1. The experimental settings are not sufficient as compared with SAM paper. It would be better to have further experiments and a few ablation studies to confirm the model-agnostic characteristic of this technique. \n2. There is a misunderstanding between the definitions of sparsity in line 176. The sparsity s should be the percentage of parameters to be perturbed, i.e., s=1 means all the parameters should be perturbated. However, s=1 in the table 1 means no parameters will be perturbated. Could the authors check for the definition? \n3. It would be more convincing if the authors could have more analysis of the experimental outcomes. In the section titled \"Questions,\" several inquiries regarding experiments are presented. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "oHs-ctrB0m", "reviewer": "Reviewer_ym8S", "summary": "The paper proposes to make the recent Sharpness-Aware Minimization (SAM), which tries to find flatter minima (which tend to generalize better) more efficient. Vanilla SAM incurs a 2x compute as 2 gradient computations (one for the perturbation and one at the perturbation) are needed at each training step. The paper proposes sparsifying SAM (SSAM) by only applying SAM on some fraction $s$ of the model weights. Which $s$ fraction is determined either in a cheap greedy fashion (SSAM-D) or via Fisher Info (SSAM-F). Non-convex analysis is done showing that SSAM converges at the same rate as SAM. SSAM is shown to perform nearly as well as SAM on benchmark tasks like CIFAR10, CIFAR100, and ImageNet using a ResNet model of modest size.\n", "questions": "* Ablations would greatly strengthen the paper.\n* More experiments / datasets would also help.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The paper is well-motivated -- SAM is a powerful method but its 2x performance hit may hinder its widespread adoption and this paper shows how it can be sped up with little loss in performance.\n* Paper is clear, easy-to-follow, and mostly free of typos.\n* Theoretical analysis supports the proposed method.\n\nWeaknesses:\n* Although SSAM does in theory reduce SAM's FLOPS, it's unclear whether it would actually help in practice. Due to the nature of how gradients are computed in practice, computing gradients for a sparse subset of weights is often not much faster than computing the full set, especially when the subset is 50% and the weights could be from any of the layers. While the paper shows theoretical FLOPS, it does not have any numbers on actual wall-clock speed, etc, though I do understand that reporting wall-clock speed has some issues of its own.\n* Key ablations missing. What happens if you invert the mask? For SSAM-D, instead of the dropping the flattest weights from the mask, drop the sharpest weights. Furthermore, what if a random subset is dropped?\n* What's the influence of m-sharpness (doing m perturbations and averaging the gradients at the perturbations -- see original paper) on SSAM? \n* Another strategy to reduce compute is using a 20% subset of the minibatch for the ascent gradient calculation (e.g. to find the perturbation). This strategy reduces the FLOPs significantly. How does SSAM compare to SAM when this approximation is used? Reference: https://arxiv.org/abs/2110.08529.\n* The claim that SSAM can outperform SAM is weak: looking at Table 1, cifar100 95% and 98% have 0.40% gap for SSAM-F, which is larger than SSAM-F 50%'s improvement over SAM, suggesting the claimed improvement may just be noise.\n* Tables start sparsity at 50%. Interested in seeing 1%, 5%, 10%,.. etc. At low sparsities it should be nearly identical to SAM.\n\n\n\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ed-lcCrSo_4", "reviewer": "Reviewer_kk3C", "summary": "This paper presents a sparse version of sharpness-aware minimization (SAM). The proposed SSAM does not apply perturbation to all the model parameters, where the criteria is based on the Fisherman Information or dynamically generating a mask through training. The authors claim improved generalization and training speed over SAM.", "questions": "Please see the weaknesses for questions.", "limitations": "The authors haven't covered the limitations. Potential limitations are limited results on other architectures besides ConvNets like Transformers.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Strengths:\n    - The idea and observation that not all the model weights need to be perturbed in SAM is generally novel. The authors also provide empirical justification that the difference between $g_{SAM}$ and $g_{SGD}$ is not very significant for lots of weights.\n    - I like the provided training curves as they show that SSAM behaves very alike to SAM in terms of the training dynamics.\n\n\n- Weaknesses:\n    - I don't quite understand how the FLOPs reduction are calculated in Table 1, 2, 3. From the algorithm, SSAM still requires twice full back-propagation. The only difference is multiplying the mask with the perturbations. Can the authors explain the FLOPs and it would be better to show the clock time of the training process.\n    - I don't quite understand why option 2 is faster than option 1 when generating the mask (the first sentence in Section 4.3). From my understanding, they both rely on the gradient scale and require twice gradient computations.\n    - Can the authors further demonstrate that the weight they select to perturb coincide with those appear to the right half in Figure 1? Just want to confirm that the masking strategy is align with the gradient information.\n    - A random masking baseline should be included to show the effectiveness of SSAM.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "HSyTEc5YM76", "reviewer": "Reviewer_BQW5", "summary": "This paper rethinks the indiscriminate perturbations of SAM on all parameters and claims that not all parameters are needed to be perturbed. The conclusion is supported by two observations --- (i) little difference between SGD and SAM gradients in most dimensions and (ii) more flatter without SAM in some dimensions. Inspired by these observations, this paper proposes to improve the effectiveness and efficiency of SAM via a sparse perturbation scheme (SSAM). Two realizations of SSAM are provided, namely SSAM-Fisher and SSAM-Dynamic. The convergence of SSAM is also discussed. Extensive experiments, including visual and numerical results, demonstrate the effectiveness and efficiency of SSAM.", "questions": "Please refer to the weakness part", "limitations": "N.A.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n- The paper is well-written and organized.  \n- Interesting observations on the flatness of SGD and SAM. It is surprising that the difference between SGD and SAM gradients is little and flat dimensions will be flatter without SAM\n- It is quite natural to apply fisher information and dynamic strategy to encourage sparsity\n- The effectiveness of SSAM is comparable to SAM; the computational cost is clearly reduced.\n\nWeakness:\n- I am very interested in why SAM makes the originally flat gradients worse?\n- The ArgTopK operation selects $k$ parameters to be perturbed. How sensitive is the performance of SSAM with respect to $k$ in a certain task? How sensitive is the value of $k$ in different tasks (if training SOTA classifiers)?\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept