A Single-Step, Sharpness-Aware Minimization is All You Need to Achieve Efficient and Accurate Sparse Training
Authors: Jie Ji
Gen Li
Jingjing Fu
Fatemeh Afghah
Linke Guo
Xiaoyong Yuan
Xiaolong Ma
Venue: NeurIPS 2024 poster (2024)
Keywords: sparse training
efficient learning algorithm
Primary area: optimization_for_deep_networks

Abstract: Sparse training stands as a landmark approach in addressing the considerable training resource demands imposed by the continuously expanding size of Deep Neural Networks (DNNs). However, the training of a sparse DNN encounters great challenges in achieving optimal generalization ability despite the efforts from the state-of-the-art sparse training methodologies. To unravel the mysterious reason behind the difficulty of sparse training, we connect the network sparsity with neural loss functions structure, and identify the cause of such difficulty lies in chaotic loss surface. In light of such revelation, we propose $S^{2} - SAM$, characterized by a **S**ingle-step **S**harpness_**A**ware **M**inimization that is tailored for **S**parse training. For the first time, $S^{2} - SAM$ innovates the traditional SAM-style optimization by approximating sharpness perturbation through prior gradient information, incurring *zero extra cost*. Therefore,  $S^{2} - SAM$ not only exhibits the capacity to improve generalization but also aligns with the efficiency goal of sparse training. Additionally, we study the generalization result of $S^{2} - SAM$ and provide theoretical proof for convergence. Through extensive experiments, $S^{2} - SAM$ demonstrates its universally applicable plug-and-play functionality, enhancing accuracy across various sparse training methods. Code available at https://github.com/jjsrf/SSAM-NEURIPS2024.

A Single-Step, Sharpness-Aware Minimization is All You Need to Achieve Efficient and Accurate Sparse Training
Jie Ji, Gen Li, Jingjing Fu, Fatemeh Afghah, Linke Guo, Xiaoyong Yuan, Xiaolong Ma
Clemson University jji0g. clemson.edu
Abstract
Sparse training stands as a landmark approach in addressing the considerable training resource demands imposed by the continuously expanding size of Deep Neural Networks (DNNs). However, the training of sparse DNN encounters great challenges in achieving optimal generalization ability despite the efforts from the state-of-the-art sparse training methodologies. To unravel the mysteri- ous reason behind the difficulty of sparse training, we connect network sparsity with the structure of neural loss functions and identify that the cause of such difficulty lies in a chaotic loss surface. light of such revelation, we propose SH-SAM. characterized by Single-step Sharpness-Aware Minimization that is tailored for Sparse training. For the first time, S innovates the traditional SAM-style optimization by approximating sharpness perturbation through prior gradient information, incurring zero extra cost. Therefore, S²-SAM not only ex- hibits the capacity improve generalization but also aligns with the efficiency goal of sparse training. Additionally, we study the generalization result of s²- SAM and provide thcoretical proof for convergence. Through extensive experi- ments, S2-SAM demonstrates its universally applicable plug-and-play functionality, enhancing accuracy across various sparse training methods. Code available https: //tg:://tttttt com hwommmjssr//SAaa.. NEURIPS2024
Introduction
The arrival the Artificial General Intelligence (AGI) era has urged an ever-expanding realm of artificial intelligence, bringing significant growth in deep neural networks (DNNs) depth and intricacy. The efficient training of DNN has thus emerged as an uttermost imperative, demanding immediate and concerted efforts.
To train an overparameterized, large-scale DNN efficiently while preserving its high accuracy, state- of-the-art literature has introduced sparse training (2+6 as a straightforward solution that reduces both parameter footprint and computation cost. With the presence of large portion of zeros in the network. both forward and backward computation for training can be saved by skipping those zeros, as well as reducing the memory consumption since the zeros are necessary to stored. However, training sparse neural network is difficult since the optimization under sparse regime readily converges to stationary points with sub-optimal generalization accuracy (i.e., saddle points) [7]. Due to the difficulty of directly (i.e., without the time-consuming linear interpolation) assessing if the sparse solution is in the saddle point, finding workaround approaches for training a better sparse neural network is critical, especially for the practical usage in efficient learning [8-17].
To address the above difficulty, many efforts have been made. For instance, the Static Sparse Training
(SST) such as the Lottery Ticket Hypothesis (LTH) 200, SNIP (21]. GraSP [8] and SynFlow [2
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

GraSP
Done Ra- 0.311
Sparsis 68 Rs- 1/734
Spansty .8 Av 1934
Spariit tos Ru 2:15
Spardit
MEST-EM
Done Ra. -L71
Spardly 6J Aa 1J15
Sparity 88 AV 1.561
Sparsty 0.9 Nu 1.39
Spervity 8.08 AS 2
Figure 1: The loss surface visualization for training sparse neural network using ResNet-32 on CIFAR-10. We select two representative sparse training methods [8 levels of sparsity. We also quantify the loss surface behavior using coefficient Ra [19] to evaluate sharpness. With increased sparsity, Ra becomes larger, indicating sharper and steeper surface.
3 and incorporate different
determine static sparse pattern training initialization. On the other hand, Dynamic Sparse Training (DST) such as SET [2]. RigL. [3 and MEST (3 iteratively updates the sparse topology during training find a better sparse model. However, those methods either suffer from suboptimal generalization ability due to heuristic nature of their methodology or metric settings, or experience difficulties in parameter setting and dynamic sparsity scheduling. With the increase of the sparsity, these phenomenon become more severe.
We identify that the cause of the sparse network learning difficulties lies in the variable learning dynamics, which is closely related to the network topology. When incorporating sparsity into a neural network, the effective structure of the original network becomes narrower. According the study of neural loss function structure (24). a wide network resulted in flat minima and wide regions of apparent convexity, which helps prevent the chaotic behavior that occurs during training. Evidently, higher sparsity indicates narrower structure, which suggests more chaotic behavior is expected during training. thus degrading the accuracy. We perform different types of sparse training with different levels of sparsity and plot the loss surface in Figure [ to demonstrate the convexity of the training behavior. According to the sharpness of the loss basin, we observe a transition from smooth and lose-to-conver surface to a steep one as sparsity is introduced and increased. Since higher sparsity levels lead more chaotic behavior, in landscapes that are narrow and sharp with a large Ra value, we might expect to encounter more chaotic training behavior that degrades generalization ability.
It seems that achieving coexistence of sparsity and good generalization ability during training is challenging. Therefore, we raise the following question: is there simple, effective method that can improve generalization ability of training a sparse neural network, without sacrificing efficiency (sparsity) and incurring zero extra cost? We believe the answer the above question lies in the sharpness of the loss surface. Inspired by the Sharpness- Aware Minimization (SAM) [25] technique that finds flatter minima that have uniformly low loss nearby regions, we argue that such technique is especially suitable for sparse training since the steep loss surface induced by sparsity can be directly mitigated. However, to leverage sharpness for better generalization, SAM uses an additional full training step (i.e., forward and backward propagation) to quantify and evaluate loss on the summation of current weights and a constrained perturbation, which roughly doubles the computation of training. Such extra costs contradict the goal of sparse training. Although some literature (26) (7 have been
proposed to reduce the computation cost for SAM, the computation is still significant, posing a roadblock to the wide application of SAM to the realm of sparse training.
In this paper, we propose a novel approach to achieve sharpness-awart minimization with zero extra cost, tailored for the sparse training regime to maintain efficiency while improving generalization ability. is the first time that a Single-step Sharpness-A Aware Minimization is proposed for Sparse training 2S²- -AMM. Different from the traditional two-step computation regime of SAM, S²-SAM uses unique single-step approach that leverages sharpness and trains weights with one training step. Specifically, S S2-SAM uses the weight gradients from the prior step to approximate the perturbation to the weights, thus solving the sharpness evaluation without performing an extra full training step. Therefore, SAM incurs zero extra cost to achieve sharpness-awee aware training, which aligns with the

---

efficiency goal of sparse training. We also study the generalization result of SA-M and provide theoretical proof for convergence. We demonstrate that S S'MSAM provides a straightforward plug-and- play functionality on variety of sparse training methods, significantly boosting their accuracy. Our contributions are summarized as follows:
We identify that the difficulty of training a sparse neural network lies in the increasingly chaotic and steep loss surface when sparsity is introduced and increased.
We develop a novel Single-step Sharpness- ing (S²-SAM) and is the first time that SAM-style optimization with zero extra computation cost has been proposed.
Minimization technique tailored for Sparse train-
We study the generalization result of S²-SAM and provide theoretical proof to demonstrate that the S²-SAM is guaranteed for convergence.
Through systematic evaluations, we show that -2MSAM provides a plug-and-ple and-play functionality applied to variety of sparse training methods, and consistently improves accuracy on different networks and datasets.
2 Proposed Method
2.1 Preliminary of Sharpness-Aware Minimization
W, (Original update)
w.
W, (S (S-M
SAM is optimization technique designed to enhance neural network generalization and mitigate overfitting. It minimizes the maximum loss in neighborhood around the current parameters, as opposed to solely focusing on the loss the current point. This approach identifies flatter minima that have uniformly low loss in nearby regions, ultimately contributing improved generalization perfor- mance.
p9. 10.
Wiz
u,
1. AW
Figure Illustration of the optimization mechanism of S²-SAM. The perturba- tion on the current weights approxi- mated by the weight gradients from prior step. Please see Section 2.2 for detailed discussion.
Specifically, consider family of models parameterized by w E W C Rdd; the loss function, and S denotes the training dataset. SAM aims to minimize the following upper bound of the PAC-Bayesian generalization error: for any p 0,
L(w) S max Ls(w + +e)
2 w ?
Hell.
To solve the above minimax problem, at each iteration f. SAM updates the following steps:
6 = p-sign Eessgn(VLLie (LL((::-)))
W. W Ne LV (Wi-1 1 + +&) Aw:-1),
where 1/p + 1/g 1, is a hyperparameter, X 0 is the parameter for weight decay, and T > is the learning rate. By setting p = q = 2 and introducing an intermediate variable u, we have:
LLL (w--1)
u, Wi-1
LV (w,-1)ll
W. W ni LV (wx) + Aw.-1).
(4)
2.2
The Proposed S2-SAM Method
As shown in Equation 2 SAM needs to compute the gradient twice at each iteration, involving additional computation costs. Further, the two-step gradient computation is not parallelizable, which presents challenges for deployment in large-scale training scenarios. Thus, we propose new algorithm that only needs to compute the gradient once each iteration. Different from prior efficient SAM works 28 aiming to reduce the computation by introducing periodically SAM steps or data

---

selection, our proposed framework S'M achieves zero extra computation cost while maintaining the improved generalization ability of sparse network training.
In Figure 2 we demonstrate that the perturbation on the current weights is approximated by the weight gradients -t-1 from the prior step. The rational behind such design is that the gradient direction (i.e., Wr-2 to w--1) from prior step is optimizing the prior loss to a considerable extent, thus it can be used to represent sharp direction among all perturbation directions at 1. Since the loss surface of a sparse network is too chaotic, a perturbation with relatively high degree of sharpness (i.e., the prior gradient) can still find a neighborhood with mear-maximum loss. The following equations specifying 9t-1 information are used to substitute the V Ls (We-1) step in Equation (2). Thus, we only need to compute the gradient once each iteration:
Pg:-1
Ur Wt-1
(5)
gg.-1]
g: V Ls (ur).
(6)
W W,-1 "/t (g: + AW;-1)
(7)
Remark
The parameter p can vary in terms of iteration t: Pt /c/t, where O constant.
2.3 Generalization Analysis
In this section, we study the generalization result of S²-SAM. First, we give some notations, where
most of them are followed by [29 30.. Let ws A(S) be solution that generated by random
algorithm A based on dataset Recall that problem (
min Fs(w) = Σ (w.x.) wEW
(8)
is called empirical risk minimization in the literature, and the true risk minimization is given by
min F(w) := Exx,y) ['yy, f(w;
wEW
We define its optimal solution: w. E arg min F(w). Then the excess risk bound (ERB) is defined as
EASS (ws)) F (w.).
(10)
It has been shown that the ERB be upper bounded by optimization error and generalization
error (2 30) We notice that there are several works [29] 30 studying the generalization result of SGD for non-convex setting under different conditions, such bounded stochastic gradient || ( ((w:x)) < G and decaying learning rate < f with a constant e >> O, where t is the optimization iteration. In this paper, we are not interested in establishing fast rate ERB under different conditions, but we want to explore the generalization ability of S²-SAM with the fewest possible modifications when building bridge between theory and practice. For example, weight decay is widely used trick when training deep neural networks. With the use of weight decay, the empirical risk minimization in practice becomes minwew (Fs(w) = Fs(w) + 11 ll]] Then, define some notations as follows. Specifically, let
Fs(w) = Fs(w) + 1 ||
(y:, (w; x:)) + I|
[w;x))
(1I)
F(w) F(w) + |
Followed by 33). use the following decomposition of testing error:
EA.S [F (ws)] Es [F's (w3)] < Es EA [E's (ws) Fss (w3))l + EA,S [F (ws) Fs (ws)]
(12)
where the upper bound is the optimization enror plus the generalization error.
Next, we present some notations and assumptions that will be used in the convergence analysis. Throughout this paper, also make the following assumptions for solving the problem (

---

Assumption 1. Assume the following conditions hold: (i) The stochastic gradient of Fs(w) is unbi- ased, ie.. E(x,y) ((x:) (Fs(w), and the variance of stochastic gradient bounded, ie., there exists a constant o² 0, such that
E (xx)
Vily, f((xx((;;x)) - VFs(w))l o².
(ii) Fs(w) is smooth with an L-Lipchitz continuous gradient, i..,, it differentiable and there exists a constant such that [|VFs(w) - VFs(u)) I L|(w u/l kw, u W.
Assumption 2. There exists constant > 0 such that 2μ (Fs(w) - Fs (w'))
[VFs(w)]l Vww W, where w minwew Fs (sww) is optimal solution (PL condition B1
Now consider that A S2-SAM. We define the gradient update rule Gē,, follows
u w PV J(w, x))
(13)
1 Wy J(W [(w))
gi,n (w) w 77 (Vully, fu, x)) + Aw).
(14)
[(mx))
Then we have the following lemma, which similar to Lemma 2.5 and Lemma 4.2 in (29 that use recursive definition through variable G' and w/-
Lemma 1. Assume that εly, x)) is L-smooth and B-Lipschitz. Let Wt+1 (W₂) and another sequence W(+1 = G' (w'). then
(1 + + 17) l[,
w/ll 2mp. 9 G'
||W141 - W'41ll
(1 m/A) ||w, will 2mB, 9
Please the proof of Lemma in Appendix B
Theorem 1. Under Assumption assume that E(y, f(w, L-smooth and B-Lipschitz, suppose
is (w) satisfies Assumption and minwew Fs(w)= S Fs (w's + z /|w.ll with = 2I, where we
the intermediate solution A, then
Fs (wo) pTT
ry(I 2)o²
6B +1
ER,AS (F(w)) Es (w.)l
24
where A SGD.
The w minwew Fs(w) ) is optimal solution, and show that the generalization error is bounded. Based on Lemma the proof of Theorem is derived in Appendix
3 Experimental Results
In this section, we carry out comprehensive experiments to demonstrate how S2-SAM improves sparse training performance. We test S²-SAM on CIFAR-10/100 (34] with ResNet-32 (3 and VGG-19 (35), and we also perform experiments on ImageNet-IK 1K [36] and ImageNet-C 37 based
on ResNet-50 (38)
Following the recent developments in sparse training techniques, we apply S2-SAM on static sparse training such as LTH 200, SNIP 21. GraSP [8] as well as dynamic sparse training methods such
as SET (2] DSR 33 RigL 23 MEST B. CHEX [10] and Chase (39] We apply S²-SAM to the official codes or published implementations to show performance gains. All of our experiments are performed on NVIDIA 4x A6000 GPUs. We repeat training experiments for 3 times and report the mean and standard deviation of the accuracy. For training throughput evaluation, we adopt the original settings of each bascline method and record the throughput on 4x A6000 GPUs.
5

---

Table 1: Test accuracy (%) of pruned ResNet-32 on CIFAR-10/100.
Dabasts
CIPAR-10
CIFAR-100
Prusing uaio
90%6
05%
985
90%
95%
955
ResNet-32
94.55 (Dense)
74.39 (Dense)
LT 20
92.31
91.06
88.78
68,99
65,02
$7.37
LT+ S'AM (nun)
92.58:0000 (0,271)
91.47:0010 (0.41))
19.36-0.111
(0.571)
69.34:0009 (O 50,5
65.46-0.11 (0.45")
57.76+-0113 (0.
SNIP 21
92.59980010
91.0110.11
87.51
08, 2.45
15.02..0.09
57.37
SNIP 5 -SA (ours)
93.17:00116 (0.581)
91.59:00222 (0.58))
88.00-0.29
(0.571)
69.33
65.66=9.49 (0.64")
58.55-0777 (0.85))
CinSP 12
92, 38--611
91 30-0.25
86.11-0114
69 24
56 50+-011 11
58.43-0-33
GndPP-3 3
SAM (ours)
92.87:10.14 (0.49%)
91.9810.23 (0.99)
89.66
(0.8511
69 10.98.0..3
(o 7419
67.12
(0.62")
59.45 (L02
SBT R
92.30
90,76
36
09,06
67.41
62.25
SET St SAM (surs)
92.92:0.23 (0.621)
91.50:0.199 (0.741)
08.78
(0.497)
70.23 2 (0.57))
68.28=0.15 (0.77)
6356-0 (3.31f)
DSR 33
92.97
91.61
38.45
09,63
68.20
61.24
DSR+ S -SAM (CMD))
93.40:00.21 (0.521)
02.08-00.22 (0.47))
89.11=8.117 (1,657)
70.11 (0.45))
66.7 (0.67))
62008:0077 (0.761)
Righ 83
93.07
91.83
89.00
70.34
68.22
64.07
Higt- SSM -SM tours)
93.58 9.14 (0.481)
92.11:021
0.21 (0228))
99.40.16.17 (1.40)
72.38100111
(2,0
70.29 (2.07)
64.91 0L66 (1911)
Rigt. (BRK) [X]]
91.55
92.39
90.22
70,62
68.47
64.14
RigL (BRK) SAM
93.75:00119 (0.201)
42.110008 (0-421)
91.16:1..11 (0,9411
72.56:0007
(1.941)
70.33=0.30 (1.60))
651151011
MEST
00.56:00077
91.15
89 59.22:50111
70.4
68,4360.32
5.220.27
SAM
(0.871)
91.58-1077 (0.43))
91,22+6.14 (2.00))
71.8-00.3 S1ef)
70.040010
(5.69-0134
(EMAS) 11
28.27:00114
92.44-0.13
90:51=0.11
30:000011
70, 70,36-+0.05
16.0.5
(EM&S) SAM (ours)
(0.127)
92.97..177 (USM)
91,32.1.888 (0.811)
72.741008 (1.44))
71335 (1.499)
69.13.0.00 (1.9p)
3.1 Accuracy Enhancement for Sfate-of-the-art Sparse Training Methods
CIFAR-10 and CIFAR-100. We show S²-SAM is universally applicable on various sparse training methods. The results
-2 is shown in Table D. Due to limited space, demonstrate results of VGG-19 19 on CIFAR- 10 in Table A. in the appendix. For each baseline method, we perform training with S -AAM 90%, 95% and 98% sparsity, and compare the accuracy with the original accuracy. We set hyper-parametee parameter p = 0.05 for all experiments. From the comparison results, we can notice that all sparse training methods trained with S2-SAM obtain consistent and significant improvements, not only for static sparse training but also for dynamic sparse training. We stress that for all experiments, we only use one setting for S2-SAM, which indicates an easy implementation that has the potential for wide applicability. We also find that S²-SAM works better on high sparsity, For all sparse training with a 98% sparsity ratio, S²-SAM improves the original baseline accuracy by an average of 0.77% on CIFAR-10 and 1.12% on CIFAR-100, which shows S2-SAM is especially suitable for challenging tasks.
ImageNet- We evaluate S on ImageNet- 1K dataset. The results are shown in Table 2 We report the t accuracy of the sparse training results, as well the overall training cost with FLOPs. We set hyper-parameter p = 1.0 for all experiments. Compared to the baseline methods, our result shows higher test accuracy at the same computational cost. Similarly, S2-SAM also shows better gain on a higher sparsity ratio for most of the sparse training baselines, which indicates the consistency of using our methods on different scales of datasets. To make fair comparison with different training recipes, we slightly scale up our training epochs to have the same or less overall training FLOPs to compare with longer training baselines, and we find out that S²-SAM shows stable improvement on those methods.
3.2 Sparse Training with Structured Sparsity
In this section, we evaluate how S²-SAM performs on structured sparse training methods. Note that all previous baselines use unstructured sparsity where the zeros are scattered in the network. Such unstructured sparse training shows the potential of training sparse model, but cannot fully represent training acceleration ability. Therefore, we apply S2-SAM to two representative structured sparse trainings, CHEX [10] and Chase [39 which dynamically train sparse networks with channel-level sparsity. The results are shown in Table 3 From the results, we can see that the average accuracy improvement is around 0.5%. Considering struetured (channel-wise) sparsity is one challenging sparsity type to achieve good accuracy, these results indicate S2-SM works very well on structured sparse training methods.
3.3 Improvement on the Roughness of the Loss Surface
In this part, we plot the loss surface of the sparse training with and without S2 The comparisons are demonstrated in Figure 3. where we perform our experiments with three representative sparse
training methods SNIP 21. GraSP [8 and MEST ( at 90%, 95% and 98% sparsity ratios on

---

Table 2: Results of ResNet-50 ImageNet- IK.
Method
Spunitt Distribution
Top-1 Accuracy (8)
FLOPs
Top-1 Accuury
Training
FLOPs
RosNet-50
dense
76.9
(xe18)
(xee)
(xel1)
(xet))
Spanity
50%
90%
LI
20
72.6
mia
277
70.1
Wa
1.7
LI
SAM (ouns)
73.11 0.40 (0.511)
mia
2.7
70.78.00055 (0.68")
n/a
1.7
SNIP
21
non-uni furm
60.7
1.67
2.8
62.0
0.91
1.9
SNIP
-AM
70.55-0005 (0.851)
62.62-0.07
GmSPP
SAM (ours)
72.66 (0.561)
1.67
2.8
68.78 16.78.00.12 (0.681)
0.91
SET 2
72.9
0.74
1.7
69.6
0.10
0.1
SIST SSMM (ours)
non uniform
7..66=0.11 (0.7611)
0.74
1.7
70.41 (0.817)
0.30
0.1
DSR 53
73.3
1.28
3.3
71.6
0.36
2.5
DSR -SAM
:4.1 (0.781)
72.52=0013 (0.727)
0.96
RigL.
Rigl + SASSAM (purs)
75.39:0.12 (0.791)
0.74
1.7
-0.06 (0.447)
0.39
MEST (EM)
75.7
1.10
1.7
73.6
0.48
0.9
MEST (EM) + 5 SAM (ours)
unifons
76.35.0002 (0.851)
1.10
1.7
14.58 (0.987)
0.48
0.9
MEST (EM&S) B
75.7
1,27
1.7
75.0
0.65
0.9
MEST (BMAS) S' -AM
76.44=0.06 (0.741)
75.36-0.044
0.65
Top-KAST
73.0
0.65
Top-KAST
-AM (ours)
(0.827)
MEST
3
76.7
1.84
1.7
75.9
0.80
MEST SAM (ourS)
uniEom
0.11 (1.031)
1.84
1.7
76.82:00:12 (0:927)
0.80
0.9
Righ-s
23
76.6
3.71
1.7
75.7
1.95
0.9
Righov + SAM
77.72:0.05 (1.121)
3.71
76.88:0.33
1,95
Orgiial
Origial
S"SAM
S'SAM
SHIP
1332
No FATH
Ba 2
Sparsity
Sparsity
Origral
S-AM
Origial
S-AM
Original
SAA
GraSP
Ro 1.22
Bs 1575
2213
By 2.0
Sparstty
Sparatty
Sporaity
MEST
L6B
Sparatty 5.
Speraity
Figure Loss surface sharpness comparison of different sparse training methods with original training and with SAMS We also quantitatively evaluate the coefficient Ra. Using S²-SAM compared to the original method results a smaller Ra, indicating a wider and smoother loss surface, which suggests improved generalization ability.
CIFAR-10 10 dataset with ResNet-32. We observe that as sparsity increases, suggesting more chaotic training behavior, the coefficient Ra also increases, indicating steeper loss surface and narrower basin. While with S²-SAM, we can see that under the same sparsity, the basin of the loss surface widens and enlarges as Ra values decrease, suggesting smoother loss trajectory during training of sparse network.
3.4 Training Speed Comparison on GPU
S2-SAM a highly efficient approach to optimizing the sharpness of loss surface, which incurs zero extra cost for achieving its functionality. Compared to the traditional SAM, S2-SAM uses fewer computations and has better training performance on GPUs. Table we obtain the computation cost well as the training speed in terms of throughput (i.c., imgs/sec) on 4x NVIDIA A6000 GPUs
7

---

using ResNet-50 on the ImageNet- IK dataset. We record the throughput for different sparse training methods with their original training, with SAM (25 and with SZ From the results, we can see that S achieves the same training throughput (with only negligible < 20 imgs/s decrease due to processing) as the baseline methods, where no sharpness optimization involved. Compared to original SAM, we can see that training throughput with SAM is less than half of the ones that train with S SSMM, which aligns with the observation that SAM uses twice the computation cost of S*-SAM or original training. Although SAM yields slightly better accuracy, the associated training cost outweighs the benefits, rendering it impractical.
Table 3: Accuracy of S-SAM structured
Table 4: Training speed of SAM 25 and S²-SAM
sparse training CHEX 10 and Chase [39
for different sparse training 90% sparsity.
Methods
Networks
Training
FLOPs
Accuracy (98
Methods
Training
Accuracy (%)
Throughput ())
Re
Original
2.0G
73.50
Original
68.10
2148 imgs/s
SF-SAM
2.0G
73.94 (0.44))
GraSP
SAM
68,95
1021 imgs/s
CHEX
S*-SAM
68.78
2132 imgs/s
Original
1.0G
76.00
ResNet-50
S SAM
1,0G
76.51 (0.511)
Original
72.00
3133 imgs/s
RigL
SAM
72.75
1508 imgs/s
ResNet-34
Original
2.1G
72.34
S2-SAM
72.44
3098 imgs/s
S*-SAM
2.1G
72.77 (0.431)
Chase
Original
73.60
2981 imgs/s
ResNet-50
Original
1.3G
75.62
MEST (EM)
SAM
74.88
1398 imgs/s
SPASAM
1.30
76.17 (0.551)
S'-SAM
74.58
2977 imgs/s
RRoNees original
ingienectaiio
3.5
Robustness Improvement by S2-SAM
Since sparse training approaches have potentially high usage in practical scenarios, the system's 's robustness against perturbation (e.g., inclement weather conditions for image/video tasks) is usually critical. We perform experiments to evaluate the sparse model (80% sparsity) robustness against perturbations in T Our intuition is that the model trained with its sharpness optimized has a wider loss basin, which indicates higher endurance on perturbations since the loss won't 't change much when it located in wide and flat region. We adopt ImageNet-C (37) which contains a test set with the same images as ImageNet-IK IK but with nineteen types of corruptions applied with five different levels of severity. We train the sparse networks using different methods on ImageNet- 1K, and then test their accuracy on ImageNet-C. We report test accuracy for both datasets. We can see that when the sparse model is trained using the original method, the accuracy of ImageNet-CC is significantly lower than the accuracy of the original ImageNet-1 1K test set (around 35% lower). With S"-SAM, the test accuracy on ImageNet-C shows promising improvement. The robust accuracy improves by an average of 3.23%.
Table 5: Testing accuracy on ImageNet-C test set. We compare the results with and without S²-SAM using 80% sparsity.
Table 6: Testing accuracy on dense model training. We compare original training with S2-SAM in same settings.
Methods
ImageNet-18 1K
ImageNet-CC C Accuracy (%)
Networks
Params. Count
Original Accuracy (%)
8 .AAM Accuracy (%)
Accuracy (%)
SNIP SNIP S SAM
69.70
31.12
CIFAR-I 10
70.55 (0.85t)
34,87 (3.75t)
ResNet-32
1.86M
94.58
94.99 (0.411)
GraSP
72.10
32.24
MobileNet-V2
2.30M
94.55 (0.421)
GraSP S SAM
72.66 (0.561)
35.17 (2.93()
VGG-19
20.03M
94.21
94.48 (0.271)
MEST (EM)
75.70
33.87
Image Net-IK
MEST (EM) SAM
76.35 (0.651)
36.98 (3.1
EfficientNet- BO
5.30M
76.54
77.10 (0.56t)
Rigl.
74.60
33.68
ResNet-34
21.80
74.09
74.58 (0.491)
RigL. + S
75.39 (0.791)
36.80 (3. (3.121)
ResNet-50
25.50M
76.90
77.32 (0.421)
3.6 Applying SSMM to Dense Model Training
We also apply S2-SAM to the dense model training. The results are shown in Table 6. We test on two datasets CIFAR-10 and ImageNet-18 IK with two additional networks MobileNet-V3 V2 [41 and EfficiietNee--0 42 For ResNet-32 and VGG-19 on CIFAR- 10, we follow the settings in 3 and train for 160 epochs, and for MobileNet-VS V2, we use the 1.0 width version and train for 350 epochs
8

---

for better convergence [3 For ResNet family on ImageNet-IK, IK. we follow the setting in 3 and train for 150 epochs, and we train EfficientNet-BO for 350 epochs for better convergence 44 From
the results, we can see that S2-SAM still improves the accuracy of the dense networks. We find out that the less the parameter count of the network, the better effectiveness of S2-SAM achieves. This phenomenon proves our finding that narrower network is harder to train, and S*-SAM is an effective solution for better generalization. We must also stress that S is focusing on sparse neural network training, and we will leave the study of SSMMAM on dense model training for our future research.
4 Related Works
Static Sparse Training Static sparse training determines the structure of the sparse network through the application of a pruning algorithm in the early stages of training. The lottery ticket hypothesis
(LTH) [20 45 46 uses iterative magnitude-based pruning (IMP) to find subnetwork that can be trained from scratch without losing accuracy. SNIP [2T. GraSP [8], SynFlow [22] determine a
static sparse pattern at training initialization by obtaining gradient information with a few iterations of dense training. FISH [47 acquires fixed subnetwork by pre-computing a sparse mask using Fisher
information.
Dynamic Sparse Training Dynamic sparse training starts with randomly selected sparse network structure and adapts it throughout the training process in an effort to find a better sparse structure. Sparse Evolutionary Training (SET) [ prunes small magnitude weights and grows back randomly at the end of each training epoch. Deep R [48] uses a combination of stochastic parameter updates and dynamic sparse parameterization for training. Dynamic Sparse Reparameterization (DSR) (33 proposes to redistribute parameters between layers during training. Sparse Networks from Scratch (SNFS) [49 creates the sparse momentum algorithm, which finds the layers and weights that effectively reduce the eтror by using exponentially smoothed gradients (momentum). In RigL (23. the gradients of all the weights are computed when the model needs to be updated to grow new connections. Top-KAST 40 proposes a scalable and performant sparse-to-spars DST framework for maximum efficacy. Powerpropagation (50) suggests a novel neural network weight parameterization that largely preserves oow-mannttude parameters from learning. ITOP (51) investigates the underlying DST mechanism and finds that the advantages of DST result from a time-based search for all potential factors. MEST 3 designs memory-cconomic sparse training framework targeting accurate and fast execution on edge devices. By co-training dense and sparse models, AD/AC [52) suggests a technique that, at the conclusion of training, produces precise sparse dense model pairings. Chase [39] dynamically translates the unstructured sparsity into channel-level sparsity to achieve direct speedup on GPU.
Sharpness-Awar Minimization Sharpness-Awan Minimization was first introduced in 25 This optimization technique is designed to identify flatter minima characterized by consistently low loss neighboring regions, aiming to enhance the model generalization capability during training. Nevertheless, the computational cost of Serrpness-Awart Minimization is doubled due to its two-step gradient computation regime, presenting challenges for deployment in large-scale training scenarios. To reconcile such, ESAM (26] introduces two novel and efficient training strategies: stochastic weight perturbation and sharpness-sensitive sensitive data selection, enhancing the efficiency of the SAM process without compromising its generalization performance. LookSAM (27) only periodically calculates the inner gradient ascent to significantly reduce the additional training cost of SAM. SAF (28) introduces novel trajectory loss based on KL.-divergence measure the rate of change in training loss along the model update trajectory and replace the SAM sharpness measure. CrAM 53 optimizes over the compression projection applied to the intermediate model every training step, which
results in compressible models that can be pruned in an on-shot manner after training. However, all those methods need extra computation cost, and not target on sparse training method.
5 Conclusion
In this paper, we propose a novel Single-step Sharpness-Awar Minimization that tailored for Sparse training (S*-SAM) which revolutionizes the originally computation-intensii sharpness-aware optimization into highly efficient tool with zero extra cost. In light of the improved optimization trajectory in the loss surface, S²-SAM successfully enhances the accuracy of the sparse network

---

training. as well as the robustness of the sparse model in practical scenarios. S2-SAM offers seamless plug-and-play functionalityy showcasing its potential for widespread upplicability in the evolving landscape of efficient training. The research is inherently scientific, and we anticipate no adverse societal impact stemming from its findings.
Acknowledgments
This work is partly supported by the National Science Foundation CCF-2427875, CCF-231261 16, CNS-2232048, CNS-2008044 CNS-2431440, National Aeronautics and Space Administration (NASA) 80NSSC23K1393, and the Army Research Office W911NF-24-1-0044 The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of NSF, NASA, the Army Research Office or the U.S. Government. The U.S. Government authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
References
[1] Ben Goertzel and Cassio Pennachin. Artificial general intelligence, volume 2. Springer, 2007.
[2] Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature Communications, 9(1):1-12, 2018.
[3] Geng Yuan, Xiaolong Ma, Wei Niu, Zhengang Li, Zhenglun Kong, Ning Liu. Yifan Gong, Zheng Zhan, Chaoyang He, Qing Jin, Siyue Wang, Minghai Qin, Bin Ren, Yanzhi Wang, Sijia Liu, and Xue Lin. Mest: Accurate and fast memory-economic sparse training framework on the edge. In Advances Neural Information Processing Systems (NeurIPS). 2021.
[4] Lu Yin, Gen Li, Meng Fang, Li Shen, Tianjin Huang, Zhangyang Wang, Vlado Menkovski, Xiaolong Ma, Mykola Pechenizkiy, Shiwei Liu, et al. Dynamic sparsity is channel-level sparsity learner. Advances Neural Information Processing Systems, 36, 2024.
[5] Gen Li, Lu Yin, Jie Ji, Wei Niu, Minghai Qin, Bin Ren, Linke Guo, Shiwei Liu, and Xiaolong Ma. Neurrev: Train better sparse neural network practically via neuron revitalization. In The Twelfth International Conference on Learning Representations, 2024.
[6] Jie Ji, Gen Li, Lu Yin, Minghai Qin, Geng Yuan, Linke Guo, Shiwei Liu, and Xiaolong Ma. Advancing dynamic sparse training by exploring optimization opportunities. In Forty-first International Conference on Machine Learning, 2024.
[7] Utku Evei, Fabian Pedregosa, Aidan Gomez, and Erich Elsen. The difficulty of training sparse neural networks. ICML. Workshop Deep Phenomena, 2019.
[8] Alex Bronstein, Pablo Sprechmann, and Guillermo Sapiro. Learning efficient structured sparse models. In International Conference on Machine Learning, 2012.
[9] Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Leurning Research, 22(241):1-124 2021.
[10] Zejiang Hou, Minghai Qin, Fei Sun, Xiaolong Ma, Kun Yuan, Yi Xu, Yen- Yen-Kuang Chen, Rong Jin, Yuan Xie, and Sun-Yuan Kung. Chex: Channel exploration for cnn model compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12287-12298, 2022.
(11) Xiaolong Ma, Minghai Qin, Fei Sun, Zejiang Hou, Kun Yuan, Yi Xu, Yanzhi Wang, Yen-Kuang Chen, Rong Jin, and Yuan Xie. Effective model sparsification by scheduled grow-and anu-prune methods. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[12] Bo Hui, Da Yan, Xiaolong Ma, and Wei-Shinn Ku. Rethinking graph lottery tickets: Graph sparsity matters. In The Twelfth International Conference on Learning Representations, 2023.
10

---

[13] Haoyu Ma, Chengming Zhang. Xiaolong Ma, Geng Yuan, Wenkai Zhang, Shiwei Liu, Tianlong Chen, Dingwen Tao, Yanzhi Wang. Zhangyang Wang, al. Hrbp: Hardware friend-r regrouping towards block-based pruning for sparse cnn training. In Conference Parsimony and Learning. pages 282-301. PMIR, 2024.
[14] Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, PRANAY SHARMA, Sijia Liu, et al. Model sparsity can simplify machine unlearning. Advances in Neural Information Processing Systems, 36, 2024.
[15] Lu Yin, You Wu, Zhenyu Zhang, Cheng- Yu Hsieh, Yaqing Wang. Yiling Jia, Gen Li, Ajay Kumar Jaiswal, Mykola Pechenizkiy, Yi Liang, Michael Bendersky, Zhangyang Wang, and Shiwei Liu. Outlier weighed layerwise sparsity (OWL): A missing secret sauce for pruning LLMs to high sparsity. In Proceedings of the 4 International Conference on Machine Learning, volume 235, pages 57101-571 15. PMIR, 2024.
[16] Shuyao Wang, Yongduo Sui, Jiancan Wu, Zhi Zheng, and Hui Xiong. Dynamic sparse learning: A novel paradigm for efficient recommendation. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, pages 740-749, 2024.
[17] Haizhong Zheng, Xiaoyan Bai, Beidi Chen, Fan Lai, and Atul Prakash. Learn to be efficient: Build structured sparsity in large language models. arXiv preprint arXiv:2402.061166 2024.
[18] Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations (ICLR), 2020.
[19] NO Myers. Characterization of surface roughness. Wear, 5(3):182-189, 1962.
[20] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. ICLR, 2018.
[21] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. Snip: Single-shot network pruning based on connection sensitivity. In International Conference on Learning Representations (ICLR), 2019.
[22] Hidenori Tanaka, Daniel Kunin, Daniel L Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), volume 33, 2020.
[23] Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning (ICML), pages 2943-2952. PMLR, 2020.
[24] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in neural information processing systems, 31, 2018.
[25] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness- aware min- imization for efficiently improving generalization. In International Conference on Learning Representations, 2021.
aware
[26] Jiawei Du, Hanshu Yan, Jiashi Feng, Joey Tianyi Zhou, Liangli Zhen, Rick Siow Mong Goh, and Vincent YF Tan. Efficient sharpness-aware minimization for improved training of neural networks. arXiv preprint arXiv:2110.03141] 2021.
[27] Yong Liu, Siqi Mai, Xiangning Chen, Cho-Jui Hsich, and Yang You. Towards efficient and scalable sharpness-aware minimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12360-12370, 2022.
[28] Jiawei Du, Daquan Zhou, Jiashi Feng, Vincent Tan, and Joey Tianyi Zhou. Sharpness-aware training for free. Advances in Neural Information Processing Systems,
2022.
[29] Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. International conference on machine learning, pages 1225-1234. PMLR, 2016.
11

---

[30] Zhuoning Yuan, Yan Yan, Rong Jin, and Tianbao Yang. Stagewise training accelerates con- vergence testing error over sgd. Advances in Neural Information Processing Systems, 32. 2019.
[31] Hamed Karimi, Julie Nutini, and Mark Schmidt. Linear convergence of gradient and proximal- gradient methods under the folyak--ojasiewicz condition. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2016. Riva del Garda, Italy, September 19-23. 2016, Proceedings, Part 16, pages 795-811, Springer, 2016.
[32] Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations, 2020.
[33] Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In International Conference on Machine Learning (ICML), pages 4646-4655, PMLR, 2019.
[34] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
[35] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409, 1556, 2014.
[36] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International jounaa of computer vision, I 11:2211-222, 2015.
[37] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common conuptiios and perturbations. arXiv preprint arXiv: 12261,, 2019.
[38] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[39] Lu Yin, Gen Li, Meng Fang, Li Shen, Tianjin Huang, Zhangyang Wang, Vlado Menkovski, Xiaolong Ma, Mykola Pechenizkiy, and Shiwei Liu. Dynamic sparsity is channel-level sparsity learner. Advances in Neural Information Processing Systems (NeurIPS), 2023.
[40] Siddhant Jayakumar, Razvan Pascanu, Jack Rae, Simon Osindero, and Erich Elsen. Top-kast: Top-k always sparse training. Advances in Neural Information Processing Systems, 33:20744- 20754. 2020.
[41] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chier Chen. Mobilenetv2: Inverted residuals and linear boutlnecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510-4522 2018
[42] Mingxing Tan and Quoe Le. EfficientNet: Rethinking model scaling for convolutional neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97, pages 6105-6114. PMLR, 2019.
[43] Zhuangwei Zhuang, Mingkui Tan, Bohan Zhuang, Jing Liu, Yong Guo, Qingyao Wu, Junzhou Huang, and Jinhui Zhu. Discrimination-aware channel pruning for deep neural networks. Advances in neural information processing systems, 31, 2018
[44] Yanyu Li, Geng Yuan, Yang Wen, Ju Hu, Georgios Evangelidis, Sergey Tulyakov, Yanzhi Wang, and Jian Ren. Efficientformer: Vision transformers at mobilenet speed. Advances in Neural Information Processing Systems, 35: 15:12934-1299 12949, 2022.
[45] Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. In International Conference on Learning Representations, 2020. arXiv:2003.02389.
[46] Xiaohan Chen, Yu Cheng, Shuohang Wang, Zhe Gan, Zhangyang Wang, and Jingjing Liu. Earlybert: Efficient bert training via early-bird lottery tickets. ACL-IJCNLP, 2021.
[47] Yi-Lin Sung, Varun Nair, and Colin A Raffel. Training neural networks with fixed sparse masks. Advances in Neural Information Processing Systems, 34:24193-24205, 2021.
12

---

[48] Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein. Deep rewiring: Training very sparse deep nor-works. In International Conference on Learning Representations (ICLR), 2018.
[49] Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprint arXiv:1907.04840 2019.
[50] Jonathan Schwarz, Siddhant Jayakumar, Razvan Pascanu, Peter Latham, and Yee Teh. Power- propagation: A spansity inducing weight reparameterisation. Advances in neural information processing systems, 34:28889-28903, 2021.
[51] Shiwei Liu, Lu Yin, Decebal Constantin Mocanu, and Mykola Pechenizkiy, Do we actually need dense over-paramettriizaionn? in-time peeepparamettrizztion sparse training. In International Conference on Machine Learning, pages (9999-0000 PMLR, 2021.
[52] Alexandra Peste, Eugenia lofnova, Adrian Vladu, and Dan Alistarh. Ac/de: Alternating compressed/decompressel training deep neural networks. Advances neural information processing systems, 34:8557-8570, 2021.
[53] Alexandra Peste, Adrian Vladu, Eldar Kurtic, Christoph H Lampert, and Dan Alistarh. Cram: compression-awent minimizer. arXiv preprint arXiv:2207. 14200, 2022.
13

---

Appendix
A Results of 19 on CIFAR-10/100
We test S -SM on VGG-19 19 using CIFAR-I0 and CIFAR- 100. The results are shown in Table A. We demonstrate results on both uniform and ERK distributions, and achieve SOTA results on CIFAR- 10/100/
Table A.I: Test accuracy (%) of pruned VGG-19 on CIFAR-10/100.
Dutaeess
CIFAR-10
CIFAR-100
Pruning calio
90%
95%
96%
90%
95R
988
VGG-19
94.20
74.17
LT 27
93,51
92.92
92.34
72.78
71.44
68.55
LT S' SAM (ouEs)
93.82=0.12 (0.31
93.41-0.11 (0:491)
92.92-0.14
(0.581)
73,0088US
(0.30f)
7169:88.0
0.251)
69.8=0.17 (0.83))
SNIP 2J
93. 33.67600066
93.47+0:20
92.05+0225
72.84+0222
21,8360335
18.46+1:110
SNIP SAM
93.91.0.1 (0.281)
93.91.0.211
92.86.19.25 (0.811)
13.33
(0:491)
72.4110.35
(0.58))
60.1219.43 (1.61)
GraSP
93. 00.1010114
03.42110188
92,1910.12
71.9510.18
71,2310112
68.010041
GruSP
92.95-00223
(0.41))
(1.01)
91.73
69.11
65.94
SAM
(0.511)
(0,521)
(1.01)
1244 (0:29))
(0:00))
70.70
(ouzs)
04.3 (2.49))
94.27:0.15 (0.41))
93.91 (0:68))
72.78:0.20 (0447))
(0.57))
(0.00))
Righ
92:43
Rigl.+ SAM (ouns)
93611:
3361110116 (0:491)
92.87.0.211 (0:44))
01.88 (1,231)
719881611 (0.841)
70.16:1..18 (1,14)
66.1110 (LIM
Rigl. (ERR)
93.
91:75
9087
71.34
Rigl. (FRK) S-MA (sung)
94,07-0.21 (0.901)
93.33-0.12 (0.581)
91.79-0.12 (0:921)
72.12+0.20 (0.781)
69.96-0.188 (0.74f)
66.3-0.199 (1.1I))
MEST (EM) B
93,07::0.36
92.99981441
00.55
71.2320.37
69:05:0.411
64.92200.34
MFST (EMD + 37-SAM (sum))
9..76-0.13
98.18-0.12
(0. 0.591)
91.86-0.23
311)
71:93-0.11 (0.001) (01)
69.98-0.10 (0:00))
66.08-0.16 (1.16)
MEST (LM&S) [
93.61.10.36
93, 46100.1
92, 02.3010:46
72.5210377
71.2110.41
09.02.0.34
MEST (EMASS) S SAM (ouEs)
0.111 (1.907)
93.98:0.11 (0.52))
91.55:0.14 (0,751)
72.90:0.17 (0.381)
72.42::0.09 (1.211)
71.01:0.13 (1.991)
B Proof of Lemma 1
Proof. We consider two cases.
(1) When G G' and E(:, f(w:)) is L-smooth, then
We+1 w/+1
- (1 - n/a [w: w(ll + || (u:x))) - V (v.,, (ol
< (1 - na)l
- [w:-w;]
+ 1y Ilu u/ll
$ (1 + n/ - -)) |/, - w/ll + 2mp.
(2) When 9 + G' and [(: ((::)) is B-L-ipschitz then
| - w
< (1 1/ ||, w + 7 Vv (y,, (ux:x)) Vul (u,, (w]xx)))
< (1 17 |w, w/ll + Bn
C Proof of Theorem
Proof. By Theorem 3.2 of 29 we have
A+t1 = E W W
7/1)+++ + (1-
1
+ пp
2mp
- ((1 m))) +nBB
71
n
1
nn mn A, 2nB
1 1 1
71
11
(6)
21BBB + 2(n - 1)pp
(1 - nL) +
(0) BnB + 2(n
1)mp Σ (1 (1- <
(d) 2B + 2(n 1)p
(15)
nL
0=0
14

---

where (a) uses Lemma] 1 (b) uses À = 2L; (c) uses A.o = 0; (d) uses nL < 1. Then, by Lemma 3.11 of [29] and is (L + smsothh, we have
to 2B + (n 1))) + 2) 6(B (n 1)p) +
E i (we:z)= - 7 (w/: -
(16)
n
nn
where the last inequality holds by selecting to 1 and X 2L. Then, by Theorem 2.2 of (29)
EA.S | (w.) Fs (w.) 6(B + (n 1)p) +1
(17)
n
which is the generalization error. Next, we will bound the optimization error. Since Fs is (I + X)- smooth and satisfies p-PPI. condition, then follow the standard analysis, we have
Since Fs is (L A)- smooth,,, we have
Es (we) Fs (wa-1)
L+
X
1 V Fs (wc-1), We - -W1-1
w -1-1l?
2
17 (vfs (wi-1). Vul(y, fu,,,))
18 ( 2)
Va (y, x1)/
2
I 17 (vF' (w. V (y. (f - 17 ( (VFs (wi-1) Vuē(y, (u, f (f (
(w1-1,
(18)
+ 2
-- n (v's (w, (w. (y (. + " vFs (w.-1) )] + np² 2 + 1 + A)B²
2
(19)
where the last inequality is due to f(y, f(w, (w) is B B-Lischhitz Therefore, we get
T-1
Fs (w:) 12]
2Fs (wo)
T
+ + n( + +AB2
(20)
"
t=
Since F's satisfies 4-PL condition, then
1 T-1
T-1
Es
EA
Fs (we)
Fs (W3)
$
1
Σ
E
(wr) [1
2μT
Fs (wo)
p + m(L p + A)B²
+
(21)
pnn/T
24
Then by
12 11 and (1.. we get
T
is (wo) Mn!
+ (T A)B2
6(B
(n 1)p) + 1
E
F (w:) - -Es Fs (Ws)
+
+
T
24
n
(22)
By using the conditions that minwew Fs(w) S F's (w'))
+ > || 12
definitions
we get
2
F (w.) Fs (w') < /. (w:) F. (W5)
(23)
Therefore, have the following inequality by 22 and 23
Fs (Wo)
P + TTLL
+ A)
6(B + (n - 1)p) + 1
EA. [F (w:)] - Es [Fs (w3)) <
+
T
2/1
(24)
By Lemma 5.1 of 29 24 implies the ERB is
Fs (Wo) unL
P + 1T(( 2)B²
6(B + (n - 1)p) + (25)
ER.A.S [F (wa)
Es [ (w.)
<
+
24
By setting 17 = 011vV)) and T = O(n) then ER.AS (F (WR)l - F (w.) < 0(1/ VT), where A is
SA-M
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We propose S2-SAM which is tailored for sparse training. The proposed method is efficient with zero extra cost and achieves better accuracy on sparse training methods. We also demonstrate theoretical and experimental results in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The research is inherently scientific, and we anticipate no adverse societal impact stemming from its findings. We discuss the limitations in the experiments that our method is focusing sparse training. But we also include dense training results to show potentials of our method, We will leave the dense model training of S2-SAM our future research.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specificattin, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
16

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:
[Yes]
Justification:
We provide theoretical results in Section 2.3 and appendix.
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented
by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All of our results can be reproduced and will release our code after the paper is accepted.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers have some path reproducing verifying the results.
17

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: All datasets used in the paper publicly accessible. We will also release code after the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (kttpss:/niip.ccm cc/
public/griiee//odeStttiii wtw.amiisssiooiiiiioooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: All important settings can be found in our paper and the papers we referenced.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We perform experiments for times as shown in experimental settings. We report mean and standard deviation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
18

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report the training resources we use in the experimental settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: /ttes://eeuueiee htt.uuuurpp.... cc/puulli/Bthicccuiieeee
Answer: [Yes]
Justification:
Our research conforms with the NeurlPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the broader impacts in the conclusion. The research is inherently scientific, and anticipate adverse socictal impact stemming from its findings.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
19

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Our method doesn't have high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We credit every referenced assets with citations.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
20

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, datasets.paperssttt www.ppppptsthhhccc thcode www.coo//aaaao datasets
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: This paper introduces new code for sharpness- aware minimization, and will release the code after the paper is accepted.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper doesn't perform crowdsourcing and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper doesn't 't perform research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
21

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "sL4EbnX3oh", "reviewer": "Reviewer_ZjXc", "summary": "This paper studies the challenges of training sparse neural networks directly and identifies one of the contributing factors, i.e., the chaotic loss surface. Consequently, it proposes a new method, i.e., Single-step Sharpness-Aware Minimization (S2-SAM), tailored specially to train sparse networks. S2-SAM is based on SAM for dense neural networks training with the main difference that it uses just one gradient computation (thus, more efficient), while SAM uses two gradient computations. Experimental results show unanimously that S2-SAM improves the performance of all sparse training methods studied.", "questions": "Q1) It seems that S2-SAM works with sparse networks, but not too well with dense networks. As sparser the network is as more impactful S2-SAM is on the overall performance (lines 252-253 and the majority of the results). Can you prepare a systematic study (experiment) to quantify and illustrate better this behavior (e.g., by varying the sparsity level from 0 to 100% in small steps, or alternatives…)?\n\nQ2) Could you present in an Appendix how the loss surface visualisations from Figures 1 and 3 have been computed? I see that you cite [6], but overall this seems to be an approximation method to visualize the loss surface of a very high dimensional space. If so, this shall be properly acknowledged to avoid bringing in readers’ mind inaccurate ideas. \n\nQ3) It seems that on ImageNet, which is a much more challenging dataset, your proposed method together with MEST or RigL outperforms the dense baseline (without S2-SAM). Can you present the results on this latter case also? Do you have any idea why this behaviour is different on CIFAR 10/100? Probably, this deserves a longer qualitative discussion.\n\nQ4) The whole empirical validation has been performed on convolutional neural networks (not on all neural network architectures), and the “chaotic” loss surface is probably just one (not the only one) of the reasons which can help understand the training behaviors of sparse neural networks. Can you please identify better in the paper the boundaries of your work to avoid inaccurate conclusions in the readers’ mind?\n\nQ5) (minor) To expand the broadness of the experimental results. In Table 1, would it be possible to add also the 80% sparsity level?In Table 2, would it be possible to add also a SET 5x?", "limitations": "n/a", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "* Original paper idea with well-designed execution\n* The paper is easy to read and follow\n* Novel proposed method with theoretical flavour\n* Well designed empirical validation\n* Impressive boost in performance for all sparse training methods studied when the proposed method is applied to them\n* The paper is significant for the sparse training community and has the potential of changing how sparse training methods are designed nowadays", "weaknesses": "* I don’t see major weak points, except that it is not clear when the source code will be made available for easy reproducibility", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer ZjXc,\n\nWe sincerely appreciate your thoughtful comments. All source code will be released after the paper is accepted. \n\n**Q1: Systematic study to quantify and illustrate the proposed method?**\n\nThank you for the question. We want to stress that our paper is focusing on providing a universal solution for training a sparse neural network with different algorithms. Compared to dense training, our contribution has more ***practical*** significance. Meanwhile, our method works fine on dense training (negligible accuracy loss against SAM) with a significant speed improvement compared to SAM. The table below shows the comparison between MEST with and without our proposed method, S$^2$-SAM, in terms of accuracy on ResNet-32 under different sparsity levels from 0% to 98%.\n\n| Sparsity | Dense | 20\\% | 40\\% | 60\\% | 80\\% | 90\\% | 95\\% | 98\\% |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| MEST(EM) | 94.58 | 94.35 | 93.92 | 93.58 | 93.23 | 92.56 | 91.15 | 89.22 |\n|MEST(EM) + S$^2$-SAM| 94.99 | 94.67 | 94.38 | 94.11 | 93.88 | 93.43 | 91.58 | 91.22 |\n| $\\Delta$ Accuracy | 0.41 | 0.32 | 0.46 | 0.53 | 0.65 | 0.87 | 0.43 | 2.00 |\n\nAs shown in the table, as sparsity increases, the accuracy difference between MEST with and without S$^2$-SAM becomes more significant. We will include this table in the revised version of our paper.\n\n**Q2: Loss surface visualizations method and citation [6].**\n\nThank you for your question and sorry for the confusion we made. In our paper, we used the method cited as [11] to generate the loss surface visualizations in Figures 1 and 3, and we use the metric in [6] to compute the sharpness of the surface. We will make it clear in our revised paper.\n\nThe method in citation [11] is a ***widely used*** technique for showing loss surface visualizations [R1-R4], employing a random direction approach to approximate the 2D projected space of the loss surface (i.e., 2D contour with loss values which can be converted to 3D image). We will further explain the method of loss surface visualization in our revised paper, and we will cite [11] again in the captions of Figures 1 and 3 for clarity.\n\nFrom citation [6] in our paper, we identify the $Ra$ value in the captions of Figures 1 and 3 as the mean absolute deviation of the z-axis value (loss value) to evaluate the sharpness of the surface. A smaller $Ra$ indicates a smoother loss surface, suggesting improved generalization ability. \n\nWe will explain with more details of [6] and [11] in the Appendix in the revised version of our paper.\n\n[R1] Chen, Xiangning, et al.\"When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations\", ICLR 2022\n\n[R2] Zhang, Xingxuan, et al. \"Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalization\", CVPR 2023\n\n[R3] Du, Jiawei, et al. \"Efficient Sharpness-aware Minimization for Improved Training of Neural Networks\", ICLR 2022\n\n[R4] Mi, Peng, et al. \"Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approach\", NeurIPS 2022\n\n**Q3: Why some sparse training accuracy higher than dense baseline on ImageNet but not on CIFAR?**\n\nThe reason for this behavior is that the CIFAR datasets are relatively small, which can make DNNs prone to overfitting and makes it harder to improve accuracy significantly. In contrast, the ImageNet dataset is significantly larger, providing a more challenging and diverse set of images that help in training more generalized models that conquer overfitting. Under such circumstances, extending training time (***MEST 1.7$\\times$*** or ***RigL 5$\\times$*** in Table 2) will be very helpful to improve accuracy as the original accuracy (MEST or RigL without S$^2$-SAM) is already very close to the dense training baseline. We will provide this discussion in the revised paper.\n\n**Q4: Identify in the paper the boundaries of your work regarding tasks.**\n\nThank you for the good suggestion. Our paper mainly explores the CNN structures and their loss surfaces to understand the training behaviors of sparse neural networks. We will emphasize this focus in the Introduction part in the revised version of the paper to ensure clarity and avoid any inaccurate conclusions.\n\nWe also conduct additional experiments on the NLP task due to reviewer's comment. We evaluate our methods on a translation task using the Transformer model [R5] on WMT-14 En-De dataset, reporting the best SacreBLEU scores on the validation dataset in the table below. We applied our method with uniform sparsity levels of 80\\% and 90\\% across all layers. Compared to MEST(EM), our method demonstrates improved performance in both dense and high-sparsity scenarios. We will integrate more NLP results in our revised paper.\n\n| Method | SacreBLEU |\n| :--- | :---: |\n| Dense | 27.6 |\n| Dense + S$^2$-SAM | 27.9 |\n\n| Method | SacreBLEU | SacreBLEU |\n| :--- | :---: | :---: |\n|  | 80\\% sparsity | 90\\% sparsity |\n| MEST(EM) | 27.1 | 26.4 |\n| MEST(EM) + S$^2$-SAM | 27.5 | 27.2 |\n\n[R5] Vaswani A, et al. \"Attention is all you need\", NeurIPS 2017\n\n**Q5: Add 80% sparsity in Table 1 and add SET 5x in Table 2.**\n\nThank you for your thoughtful suggestions. We report 90\\%, 95\\%, and 98\\% sparsity in our paper at first place because most current literature primarily utilizes them for comparison. Due to time and resource limits, we report the accuracy at 80\\% sparsity on CIFAR-10 in the response of Q1 for Table 1, and we will include that and all the other accuracy result in 80\\% sparsity in the revised version of our paper to guarantee a more comprehensive comparison. \n\nFor Table 2, we plan to include the results from SET 5$\\times$ in a future revision. And below is the table for the comparison of SET 5$\\times$ with and without our method on ImageNet dataset at 80% and 90% sparsity respectively.\n\n| Method | 80\\% sparsity | 90\\% sparsity |\n| :--- | :---: | :---: |\n| SET $_{5 \\times}$ | 74.60 | 72.43 |\n| SET $_{5 \\times}$ + S$^2$-SAM | 75.43 | 73.16 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g51tawrFZS", "reviewer": "Reviewer_uQD3", "summary": "This article introduces S2-SAM (Single-step Sharpness-Aware Minimization), an innovative sharpness-aware optimization method tailored specifically for sparse training with zero extra computational cost.", "questions": "1. What is the topological initialization of Table 2? Why some of them are non-uniform and some of them are uniform?\n\n2. The datasets in the article are all computer vision domains. Have you done some experiments on NLP domain?", "limitations": "This paper doesn't include a limitation section", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The method improves the generalization ability of sparse neural networks, which is a significant challenge in sparse training.\n\n2. Figures of loss surface significantly demonstrate the effectiveness of the proposed methods.\n\n3. S2-SAM provides a general improvement in all the sparse training methods.", "weaknesses": "1. S2-SAM seems to be a zero-extra-cost variety of SAM that is implemented in sparse cases. However, when the sparsity is high, the extra cost of SAM can be ignored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer uQD3, \n\nWe appreciate your review of our paper. The issues you have raised are very important and deserve further discussion. Below are our responses to your comments.\n\n**W1: S2-SAM seems to be a zero-extra-cost variety of SAM that is implemented in sparse cases. However, when the sparsity is high, the extra cost of SAM can be ignored.**\n\nThank you for your comment. It is true that as sparsity increases, the computation cost decreases for both training and SAM computation. However, we must also consider this scenario in a more ***practical*** perspective, which is the implementation of DNN training on the ***resource-limited devices*** (i.e., the most common case for sparse training). Hardware design usually needs to consider compact footprint to save resources. No matter how much computation saved due to high sparsity, traditional SAM will always double the remaining, leading to at least 100\\% increase of the hardware footprint. There remains a significant difference in computational cost between traditional SAM and S$^2$-SAM. Ignoring the additional computation introduced by traditional SAM undermines the purpose of sparse training, which is the ***applicability*** and ***efficiency***.\n\nOur method maintains the benefits of SAM while minimizing computational overhead, offering a practical solution for implementing SAM in resource-limited environments. As shown in Table 4, S$^2$-SAM sacrifices only marginal accuracy compared to SAM, yet SAM requires about ***twice*** the computational cost of S$^2$-SAM or original training. In summary, S$^2$-SAM provides significant advantages in efficiency and applicability across various computational settings, even with high sparsity.\n\n**Q1: What is the topological initialization of Table 2? Why some of them are non-uniform and some of them are uniform?**\n\nThe topological initialization refers to the method of distributing non-zero weights across the network layers, which is presented in two ways in Table 2: uniform and non-uniform.\n\nUniform: The sparsity $s^l$ of each individual layer is equal to the total sparsity $S$ of the network.\n\nNon-uniform (ERK): The number of parameters in the sparse convolutional layers is scaled proportionally to the width and height of the $l^{th}$ convolutional kernel.\n\nThese two sparsity distributions are widely used in current research [R1][R2][R3]. The reason we use these two sparsity distributions in Table 2 is to provide a fair and comprehensive comparison with other methods.\n\n[R1] Evci U, Gale T, Menick J, et al. \"Rigging the lottery: Making all tickets winners\", ICML 2020\n\n[R2] Liu, Shiwei, et al. \"Do we actually need dense over-parameterization?\", ICML 2021\n\n[R3] Yuan, Geng, et al. \"Mest: Accurate and fast memory-economic sparse training framework on the edge\", NeurIPS 2021\n\n**Q2: The datasets in the article are all computer vision domains. Have you done some experiments on NLP domain?**\n\nYes, we evaluate our methods on a translation task using the Transformer model [R4] on WMT-14 En-De dataset, reporting the best SacreBLEU scores on the validation dataset in the table below. We applied our method with uniform sparsity levels of 80\\% and 90\\% across all layers. Compared to MEST(EM), our method demonstrates improved performance in both dense and high-sparsity scenarios. We will also integrate more NLP results in our revised paper.\n\n| Method | SacreBLEU |\n| :--- | :---: |\n| Dense | 27.6 |\n| Dense + S$^2$-SAM | 27.9 |\n\n\n| Method | SacreBLEU | SacreBLEU |\n| :--- | :---: | :---: |\n|  | 80\\% sparsity | 90\\% sparsity |\n| MEST(EM) | 27.1 | 26.4 |\n| MEST(EM) + S$^2$-SAM | 27.5 | 27.2 |\n\n[R4] Vaswani A, et al. \"Attention is all you need\", NeurIPS 2017", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vGF9vKwdta", "reviewer": "Reviewer_Z2wG", "summary": "The authors of this paper posit that sparse training is difficult due to a chaotic loss landscape as opposed to standard training of a dense network. In order to address this problem, they propose to perform sparse training with a Sharpness Aware Minimization approach. In order to do so efficiently, they leverage the gradient of the previous iteration to identify the SAM perturbation (S^2-SAM) instead of performing a second forward-backward step like SAM. They show that this approximation of SAM also converges to an optimal solution theoretically. \nExtensive empirical evaluations show that the proposed method is effective for sparse training and can also improve robustness of the sparse networks.", "questions": "Does the proposed method largely behaves as a regularizer and hence improves generalization of sparse networks? And if so, are there other such regularization methods that could potentially offer similar benefits other than the proposed S^2-SAM. For example, it was suggested by Paul et. al. [3] that as long as the loss landscape is linearly connected for LTs, sparse networks generalize well. Hence, would S2-SAM allow for faster training in this case or potentially pruning a larger fraction of parameters in one iteration in comparison to Iterative Magnitude Pruning?\n\n[1] Jin, Tian, et al. \"Pruning’s effect on generalization through the lens of training and regularization.\" Advances in Neural Information Processing Systems 35 (2022).\n\n[2] Renda, Alex, Jonathan Frankle, and Michael Carbin. \"Comparing Rewinding and Fine-tuning in Neural Network Pruning.\" International Conference on Learning Representations.\n\n[3] Paul, Mansheej, et al. \"Unmasking the Lottery Ticket Hypothesis: What's Encoded in a Winning Ticket's Mask?.\" The Eleventh International Conference on Learning Representations.", "limitations": "See above", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors propose a simple method which can be plugged into training any sparse network to improve training via sharpness minimization, without the additional cost of an extra forward pass.\n2. Theoretical as well as extensive empirical evaluations are provided to showcase the effectiveness of  S^2-SAM.", "weaknesses": "1. The paper starts from the premise that sparse training is difficult in comparison to standard dense training. However, sparse training does not necessarily generalize poorly, in fact some methods are able to train sparse networks that outperform their dense counterparts as shown by Jin et. al. [1] In fact, Jin et. al. claim that pruning can behave as a regularizer, enabling better generalization. \nSimilarly, Renda et. al. [2] have shown that Learning Rate Rewinding (LRR), a sparsifying algorithm similar to LTs, can find sparse networks that outperform their dense counterparts.   \nDo these findings suggest that sparse networks can also be found without performing S^2-SAM. Does LRR also find better loss landscapes for sparse networks and if so, then is S^2-SAM necessary? It would also be beneficial to compare with LRR.\n2. Does performing SAM instead of the proposed S^2-SAM have a stronger effect on sparse networks i.e. do sparse networks generalize better when trained with SAM instead of S^2-SAM, no matter the training cost. It would be nice to have this comparison to shed light on the robustness of the proposed training method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Z2wG,\n\n**W1: Some methods are able to train sparse networks that outperform their dense counterparts, do these findings suggest that sparse networks can also be found without performing S$^2$SAM? Is S$^2$SAM necessary? It would be beneficial to compare with LRR.**\n\nThank you for your constructive comments. We totally agree with you that there are works which can improve the generalization of sparse training. In fact, we will also ***cite those papers and acknowledge their contributions*** to training sparse networks. What we want to stress here and in our paper is that, no matter what sparse training algorithms are used (e.g., LT-based, static sparse training, dynamic sparse training, etc.), the proposed S$^2$-SAM is ***an efficient way to boost*** the performance of such algorithms by improving accuracy with ***no*** extra cost. S$^2$-SAM is not a standalone algorithm, but a ***universal component*** that serves to help any sparse training algorithm achieve better performance. Therefore, no matter how good a sparse training algorithm is, our method will have its ***necessity***. According to Table 1 and Table 2, we already prove that S$^2$-SAM can work well with a variety of sparse training methods. We also conduct LRR and LRR+S$^2$-SAM to further demonstrate its universality. We use ResNet-32 on CIFAR-10 and original training settings, and we report sparsity and accuracy of the IMP process of LRR. From the table below, we can see that LRR+S$^2$-SAM achieves better results compared to LRR, which further proves S$^2$-SAM is a universal applicable method to different kinds of sparse training methods.\n\n| Method | 80\\% sparsity | 90\\% sparsity | 95\\% sparsity |\n| :--- | :---: | :---: | :---: |\n| LRR | 94.68 | 94.05 | 93.82 |\n| LRR + S$^2$-SAM | 94.87 | 94.39 | 94.22 |\n\n**W2: Does performing SAM instead of the proposed S$^2$-SAM have a stronger effect on sparse networks, no matter the training cost?**\n\nThank you for your question. In fact, we have evaluated different sparse training methods with original SAM and the proposed S$^2$-SAM in Table 4 in our paper. It is true that original SAM achieves slightly better accuracy compared to S$^2$-SAM, but such small accuracy loss is negligible. More importantly, our paper is built upon a very ***practical*** research domain, which is the training efficiency, and our method offers a practical solution for implementing sparse training in ***resource-limited environments***. According to Table 4, traditional SAM doubles the computational costs (i.e., 100\\% more computations, half training speed), while our method S$^2$-SAM maintains the benefits of accuracy enhancement while achieving same training speed as original training. Therefore, S$^2$-SAM extends the Pareto boundary by achieving superior outcomes without necessitating compromises. \n\n**Q1: Does the proposed method largely behaves as a regularizer and hence improves generalization of sparse networks? And if so, are there other such regularization methods that could potentially offer similar benefits other than the proposed S$^2$-SAM. Would S$^2$-SAM allow for faster training in this case or potentially pruning a larger fraction of parameters in one iteration in comparison to IMP?**\n\nThank you for your insightful comments. Regularization generally refers to techniques that improves generalization ability of neural networks by preventing overfitting, such as L1 or L2 regularization that add penalty terms to the loss function to constrain the model's complexity. Therefore, we think SAM-like approaches are also behaving as a regularizer because they are improving the generalization of deep neural networks for various settings. The key idea of SAM-like approaches is to make the model parameters robust to small perturbations in the parameter space, effectively seeking flatter minima. This is achieved by adjusting the parameters to minimize the worst-case loss within a neighborhood around the current parameter values, thus it can be characterized as regularization method. For similar approaches that improve generalization using sharpness of the loss surface, we have conducted a thorough survey and the related literature are cited in our related work section (line 284 - 297). For example, ESAM [13], LookSAM [14] and SAF [15] are all using sharpness information to improve generalization, as well as using specific algorithms to reduce cost. Different from our proposed S$^2$-SAM that has ***zero extra computation cost***, those methods all need extra computation to find a valid perturbation, and they do not target on sparse training. \n\nApplying S$^2$-SAM in Iterative Magnitude Pruning (IMP) with larger pruning fraction of parameter is an intriguing idea. Based on our experiments, S$^2$-SAM enhances the generalization ability of a variety of sparse training methods, including LT (please see Table 1 and Table 2 in our paper). Therefore, S$^2$-SAM may also be able to use fewer LT iterations to achieve good accuracy on similar sparsity. The original LT prunes 20\\% remaining parameters at each iteration. We perform additional experiments, which prune 30\\%, 40\\% parameters at each iteration. The results show when a larger fraction of parameters is pruned, the LT winning ticket accuracy slightly drops. With the proposed S$^2$-SAM, the winning ticket accuracy ***improves*** as expected. We also notice that when the pruning fraction is larger, the effect of S$^2$-SAM is ***more significant***, which proves that S$^2$-SAM successfully solves the difficulty in sparse training. We will conduct more rigorous experiments for future exploration.\n\n| Method | Prune Ratio / Iteration | Iteration | 90\\% Sparsity |\n| :--- | :---: | :---: | :---: |\n| LT | 20\\% | 11 | 92.31 |\n| LT + S$^2$-SAM | 20\\% | 11 | 92.58 |\n| LT | 30\\% | 6 | 91.71 |\n| LT + S$^2$-SAM | 30\\% | 6 | 92.32 |\n| LT | 40\\% | 4 | 90.73 |\n| LT + S$^2$-SAM | 40\\% | 4 | 91.44 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MN0niXRBNL", "reviewer": "Reviewer_UgUE", "summary": "This paper presents a sparse training method, $S^2$-SAM, which applies sharpness-aware minimization to sparse training. The authors demonstrate that sparsity during training leads to a sharper (to use the authors' words, \"more chaotic\") loss surface, something that can be mitigated by a variant of sharpness-aware training. Additionally, the authors avoid the extra gradient computation of SAM by reusing the gradient from the previous step as a proxy. The authors demonstrate theoretically that the error from $S^2$-SAM is bounded, and empirically by testing their algorithm with a number of sparse training methods, as well as with dense training.", "questions": "How much test accuracy is lost by using the previous-gradient estimate of SAM as compared to the normal SAM?", "limitations": "The limitations were adequately addressed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The method is practical and efficient, and achieves very good experimental results. I appreciated the clock-time comparison in addition to the usual FLOPs. Overall, the paper is quite convincing, and I personally plan to give the method a try in my own work.", "weaknesses": "The main experimental weakness of the paper is restricting the method to sparsities no higher than 90 on ImageNet and not doing a full ablation of the components of  $S^2$-SAM. It would have been interesting to see the high-sparsity results, as it seems that $S^2$-SAM would have been quite effective there (and if not, why not)? Likewise, an ablation would have been helpful to understand how much we lose by using the previous-gradient version of SAM (the authors do try $S^2$-SAM on dense models, but compare to regular SGD, not regular SAM). Comparing with the original SAM paper, the numbers actually seem pretty promising on dense models, but this is not included in this paper.\n\nI found the sharpness figures confusing. The overall intuition is clear, but the actual process by which the figures were obtained is not.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer UgUE, \n\nThank you for your review and thoughtful suggestions on our paper. Regarding the questions you raised, we believe they are important points that merit further attention.\n\n**W1: No result on sparsities higher than 90 on ImageNet; how much we lose by using S$^2$SAM compared to regular SAM; no result compared with original SAM on dense model.**\n\nThank you for your great suggestions. We select our experiment settings of sparsity levels based on existing sparse training papers for easy comparison. To demonstrate S$^2$-SAM is more effective on higher sparsity, we add 95\\% sparsity experiments with ResNet-50 on ImageNet. Due to limited time, we adopt RigL and MEST algorithm with S$^2$-SAM. The results are shown in the table below. From the results, we can see that S$^2$-SAM consistently improves the accuracy of sparse training algorithms, and the improvements are similar or higher than 80\\% and 90\\% sparsity results. Therefore, we can say that S$^2$-SAM is quite effective with high sparsity on ImageNet. We will conduct more experiments with 95\\% sparsity on ImageNet and add the results in the revised paper.\n\n| Method | Acurracy at 95\\% sparsity |\n| :--- | :---: |\n| RigL | 69.02 |\n| RigL + S$^2$-SAM | **69.83** |\n| MEST(EM) | 69.95 |\n| MEST(EM) + S$^2$-SAM | **70.81** |\n\nTo compare with original SAM, we have conducted experiments with sparse training algorithms in Table 4. The reason is that the focus of our paper is on the generalization ability and efficiency of sparse training, instead of dense training. As shown in Table 4, S$^2$-SAM achieves negligible accuracy drop compared to original SAM, while original SAM doubles the computational costs (i.e., 100\\% more computations, half training speed). In response to your comment about dense training, we conduct ***additional experiments with SAM on dense training***, and the results are shown in the table below. We can see that dense training with S$^2$-SAM consistently demonstrates accuracy improvements. Compared to original SAM, S$^2$-SAM only experiences negligible accuracy drop, which are completely normal since our method is based on the approximation of sharpness perturbation. We want to stress again that our paper's focus is on a ***practical scenario*** (i.e., sparse training), which means we must consider both generalization ability and the efficiency of the training. Therefore, such small accuracy degradation is totally acceptable since our method has ***zero extra cost*** compared to original SAM which doubles the computation cost. We will integrate those results in our Table 6 and discuss more on this matter in the final version of our paper.\n\n| Method | Original | S$^2$-SAM | SAM |\n| :--- | :---: | :---: | :---: |\n| **CIFAR-10** |  |  |  |\n| ResNet-32 | 94.58 | 94.99 | 95.32 |\n| MobileNet-V2 | 94.13 | 94.55 | 94.77 |\n| VGG-19 | 94.21 | 94.48 | 94.71 |\n|**ImageNet-1K** |  |  |  |\n| EfficientNet-B0 | 76.54 | 77.10 | 77.38 |\n| ResNet-34 | 74.09 | 74.58 | 74.77 |\n| ResNet-50 | 76.90 | 77.32 | 77.58 |\n\n**W2: I found the sharpness figures confusing. The overall intuition is clear, but the actual process by which the figures were obtained is not.**\n\nThank you for your comments and we are sorry for the confusion. We use the method in citation [11] in our paper to obtain the loss surface visualizations in Figures 1 and 3. The method in citation [11] is a ***widely used*** technique to show the loss surface visualization [R1][R2][R3][R4], employing a random direction method to approximate the 2D projected space of the loss surface. We mention citation [11] in line 44 of our paper to illustrate that higher sparsity indicates a narrower structure, which suggests more chaotic behavior during training, thus degrading accuracy. We will ***further*** explain the method of loss surface visualization in our revised paper, and we will ***cite [11] again in the captions*** of Figures 1 and 3 for clarity.\n\n[R1] Chen, Xiangning, et al. \"When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations\", ICLR 2022\n\n[R2] Zhang, Xingxuan, et al. \"Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalization\", CVPR 2023\n\n[R3] Du, Jiawei, et al. \"Efficient Sharpness-aware Minimization for Improved Training of Neural Networks\", ICLR 2022\n\n[R4] Mi, Peng, et al. \"Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approach\", NeurIPS 2022\n\n**Q1: How much test accuracy is lost by using the previous-gradient estimate of SAM as compared to the normal SAM?**\n\nThanks for your question. We compared S$^2$-SAM with SAM in Table 4 for sparse training. We can see from Table 4 that S$^2$-SAM sacrifices only marginal accuracy compared to SAM, but SAM requires about twice the computational cost of S$^2$-SAM or original training. And we also add a more detailed table of comparing the accuracy of SAM and S$^2$-SAM on dense model in the response of W1 (please see the table above) and we will add that to our paper later.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes the application of Sharpness-Aware Minimization (SAM) to sparse training to enhance its generalization performance. To maintain the low cost of sparse training, the paper introduces a variant of SAM that requires only a single gradient computation by approximating sharpness perturbation using prior gradient information. Extensive empirical results are presented across various sparse training approaches and evaluation metrics. All reviewers agree that this paper makes a significant contribution to the field of sparse training.
Decision: Accept (poster)