Randomized Sharpness-Aware Training for Boosting Computational Efficiency in Deep Learning
Authors: Yang Zhao
Venue: Submitted to ICLR 2023 (2023)
Keywords: Optimization
Sharpness-aware Training
Computation Efficiency.

Abstract: By driving optimizers to converge to flat minima, sharpness-aware learning algorithms (such as SAM) have shown the power to achieve state-of-art performances. However, these algorithms will generally incur one extra forward-backward propagation at each training iteration, which largely burdens the computation especially for scalable models. To this end, we propose an efficient training scheme, called Randomized Sharpness-Aware Training (RST). Optimizers in RST would perform a Bernoulli trial at each iteration to choose randomly from base algorithms (SGD) and sharpness-aware algorithms (SAM) with a probability arranged by a predefined scheduling function. Due to the mixture of base algorithms, the overall count of propagation pairs could be largely reduced. Also, we give theoretical analysis on the convergence of RST. Then, we empirically study the computation cost and effect of various types of scheduling functions, and give directions on setting appropriate scheduling functions. Further, we extend the RST to a general framework (G-RST), where we can adjust regularization degree on sharpness freely for any scheduling function. We show that G-RST can outperform SAM in most cases while saving 50\% extra computation cost.


Under review as a conference paper at ICLR 2023
RANDOMIZED SHARPNESS-AWARE TRAINING FOR BOOSTING COMPUTATIONAL EFFICIENCY IN DEEP LEARNING
Anonymous authors Paper under double-bling blind review
ABSTRACT
By driving optimizers to converge to flat minima, sharpness-aware learning algo- rithms (such as SAM) have shown the power to achieve state-of-art performances. prop-
However, these algorithms will generally incur one extra agation at each training iteration, which largely burdens the computation espe- cially for scalable models. To this end, we propose efficient training scheme, called Randomized Sharpness-AWe Aware Training (RST). Optimizers in RST would perform Bernoulli trial at each iteration to choose randomly from base algo- rithms (SGD) and sharpness-aware algorithms (SAM) with a probability arranged by a predefined scheduling function. Due to the mixture of base algorithms, the overall count of propagation pairs could be largely reduced. Also, we give theoret- ical analysis on the convergence of RST. Then, we empirically study the compu- tation cost and effect of various types of scheduling functions, and give directions on setting appropriate scheduling functions. Further, we extend the RST to gen- eral framework (G-RST), where we can adjust regularization degree on sharpness freely for any scheduling function. We show that G-RST can outperform SAM most cases while saving 50% extra computation cost.
INTRODUCTION
Deep neural networks (DNNs) have shown great capabilities in solving many real-world complex tasks (He al., 2016; Redmon et al., 2016; Devlin et al., 2018). However, it is quite challenging to efficiently train them to achieve good performance, especially for today's severely overparameter- ized networks (Dosovitskiy et al., 2021: I; Han et al., 2017). Although such numerous parameters can improve the expressiveness of DNNs, yet they may complicate the geometry of the loss surface and generate more global and local minima in this huge hypothesis weight space.
By leveraging the finding that flat minima could exhibit better generalization ability, Foret et al. (2021) propose a sharpness-aware learning method called SAM, where loss geometry will be con- nected to the optimization has shown the power to significantly improve model performance for various tasks (Foret et al., 2021: Chen et al., 2021). But on the other hand, the computation cost of SAM almost twice that of the vanilla stochastic gradient descent (SGD), since it will incur one additional propagation for each training iteration, which largely burdens the computation in practice.
guide optimizers converge flat minima. Training with the SAM
Recently, techniques are introduced to improve the computation efficiency in SAM. Specifically, instead of using the full batch samples, Bahri et al. (2021) and Du et al. (2021a) select only part of batch samples to make approximations for the two forward- hawwardbbakwward propagations. Although the computation cost can reduced to some extent, unfortunately, the forward-backwantt count in the SAM training scheme will not change essentially. Mi et al. (2022) randomly masking out part of weights during optimization in expectation to reduce the amount of gradient computations at each iteration. However, the efficiency improvement of such method is strongly limited by the chain rule of gradient computation (Du et al., 2021a). Besides, Liu et al. (2022) propose to repeatedly the past descent vertical gradients in SAM to reduce the incurred computational overhead.
propagation
Meanwhile, random selection strategy is a powerful technique for boosting optimization efficiency, particularly in the field of gradient boosting (Friedman, 2001), where small set of learners in

---

Under review as a conference paper at ICLR 2023
gradient boosting machines would be selected randomly to be optimized under certain rule (Lu & Mazumder, 2020; Konstantinov et al., 2021).
Inspired by such randomization scheme in gradient boosting, would present simple but efficient training scheme, called Randomized Sharpness-Aware Training (RST). In our RST, the learning pro- cess would be randomized, where optimizers would randomly select to perform from base learning algorithms and sharpness-aware learning algorithms at each training iteration with a given proba- bility. And this selecting probability is arranged by a custom scheduling function predefined before training. The scheduling function not only controls how much propagation count would be reduced, but also impacts the model performance.
Our contribution can be summarized as,
1. We propose a simple but efficient training scheme, called RST, which could reduce the propagation count via mixing base learning (SGD) algorithms and sharpness- learn- ing (SAM) algorithms randomly.
2. We give interpretation of our RST scheme from the perspective of gradient norm regular- ization (GNR) (Zhao et al., 2022), and theoretically prove the convergence of RST scheme.
3. We empirically study the effect when arranging different scheduling functions, including totally three typical types of function families with six function groups.
4. We extend the RST to general framework (G-RST,, where GRN algorithm is mixed such that regularization degree on gradient norm can be adjusted freely. By training both CNN models and ViT models on commonly datasets, we show that G-RST can outperform SAM mostly while saving at least 50% extra computation cost.
1.1 OTHER RELATED WORKS
We would like to discuss works associated with the research on flat minima. In Hochreiter & Schmidhuber (1997), the authors are the first to point out that the flatness of minima could be associ- ated with the model generalization, where models with better generalization should converge to flat minima. And such claim has been supported extensively by both empirical evidences and theoretical demonstrations (Keskar et al., 2017: Dinh et al., 2017). In the meantime, researchers are also fasci- nating by how to implement practical algorithms to force the models to converge such flat minima. By summarizing this problem to a specific minimax optimization, Foret et al. (2021) introduce the SAM training scheme, which successfully guides optimizers to converge to flat minima. Further, Zheng et al. (2021) perform gradient descent twice to solve the minimization and maximization re- spectively in this minimax optimization. In Kwon et al. (2021), Adaptive SAM training scheme for improving SAM to be able to remain steady when performing weight rescaling operations. Zhao al. (2022) seek flat minima by explicitly penalizing the gradient norm of the loss function. Un- like SAM-related training schemes, without a restriction on neighborhood region, Du et al. (2022) We propose to minimize the KL-divergence between the output distributions yielded by the current model and the moving average of past models, similar to the idea of knowledge distillation.
2 METHOD
2.1
RANDOMIZED SHARPNESS-AWARE TRAINING (RST)
The general idea of RST follows a randomization scheme, where the learning process will be ran- domized. Specifically, for each training iteration f, optimizers would perform a Bernoulli trial to choose from base learning algorithms and sharpness-aware learning algorithms. Here, we will con- sider first mixing the two most commonly-used algorithms, SGD and SAM. Thus, in each Bernoulli trial, the optimizer would perform the SAM algorithm with a probability p(t) or perform the SGD al- gorithm with probability 1-p(t). Here, p(t) could be predefined custom function of iteration f, and we would call it the scheduling function of RST. Apparently, the sample space for this Bernoulli trial corresponds to the set Ω = (SGD, SAM]. Correspondingly, random variable could be defined on this sample space, X(t) : Ω > 10.1). where X(t) = 0 denotes performing the SGD algorithm while X X() = 1 denotes performing the SAM algorithm. In summary, X(I Bernoulli (p))), and
0g x(1) 0 ... X(++1)
Ozt1, X(()= E (0,
(1)
2

---

Under review as a conference paper at ICLR 2023
Algorithm Randomized Sharpness-Aware Training (RST)
Input:
Training set S = { yi) )=so: loss function L()); batch size B; learning rate o; total iterations T; neighborhood radius of SAM p. scheduling function p(t).
Parameter: Model weights 0.
Output:
Optimized model weights Ó. Algorithm:
1: Initialize weight 0o: initialize optimizer with scheduling function p(!).
2: for iteration /
T do 3: Compute the gradient = VeL(O,).
4: Perform the Bernoulli trial with probability pt and record the result X
5: if X, 0 then
Implement SGD algorithm
6: 9: g-
7:
else
Implement SAM algorithm
8:
9: (O,( at 0, - 02 with E, - =
9:
end
10: Update weight 0.+1 0, ") 9t
11: end for
return final weight 0 0r
Additionally, Algorithm 1 shows the complete implementation when training with RST scheme.
Compared to the SAM training scheme, every time SGD algorithm is selected instead of SAM algorithm in the RST scheme, we would save one
propagation. Therefore, for training iteration I, the expectation of propagation count ft. in RST could
îr 2 Pt + 1- p.) 1
(2)
Here, Pr denotes the scheduling probability of p(t) at training iteration (. Equation indicates RST would incur extra more Pt propagation count expectation than the vanilla SGD training. Further, the average of the extra expected propagation count An over the total training iterations T is,
A Oi=((P)
T
where Ar E [0, 1]. bounded between Anj in the vanilla SGD scheme and the SAM scheme.
Obviously, the scheduling function p(() would straightforwardly control the number of propagations being saved. AT would be larger if performing the SAM optimization with higher probability. Also, an appropriate schedule could improve model performance further while bad one may largely harm the training. We would provide detailed study on the scheduling function in the later sections.
2.2 UNDERSTANDING RST FROM GRADIENT NORM REGULARIZATION
From previous demonstration, the gradient of RST training iteration could be expressed as, = (1-X;) Vo + X, LoL + c;)
(4)
where = /l And the expectation of this gradient over X is,
Ex[9]]
p.) Ve)()) + p. VeL(O, + Ex)
(5)
According Zhao et al. (2022), gradients in the form of Equation can be interpreted as regular- izations on the gradient norm (GRN) of loss function.
Specifically, when imposing penalty the gradient norm during training with a penalty coefficient 7. L()) + ((O)) the corresponding gradient could be approximated via the combination between (O.( and VoL(O, + 6z). which is (gar)
VoL( + Vo.L( + +
(6)
meaning that SAM is one special implementation of gradient norm regularization, where "sam
From Equation 5 and Equation 6, could reason that pt in Equation has equivalent effect with the term 7/p in GNR. means the equivalent penalty coefficient in RST would be
Trst Pt p Pt 'ssm
Compared to the SAM training scheme, the penalty degree is reduced by factor of pt in RST.
3

---

Under review as a conference paper at ICLR 2023
2.3 CONVERGENCE ANALYSIS OF RST
In this section, we would give analysis in regards to the convergence in RST.
Theorem Assume the gradient of the loss function L(-) is 3-smoothness, L( S 0| 02l1 for v0,, O2 E G. For iteration steps T 2 0, learning rate ot 1/B and vPD 1/3, we have
[VL(0))
|VL(0,112 2(L(0u) - EiE(O.,....TTI T-1) 0
min
1E(0,1,-- T-11
.T.. at
(... at
We would provide detailed proof in the Appendix. Basically, || I c generally used as one stopping criteria optimization. The theorem implies that the minimum of [ over the training steps would reach such condition at certain step within finite training steps.
Corollary 1. For constant learning rate a = C/3 or cosine learning rate schedules a = 2C/B- ( + cos( ())), and constant scheduling probability Pt. P. we have
|| 2,3(T(00) T.a)
min tE(0,1," T-11
+ pp? 82
CT
Corollary 2. For decayed learning ot C/t and constant scheduling probability p. have
2(L(0o) L.) C'log
min EE (0,1 T-
VL(0.)) <
+ +pp233
Corollary and show the convergence of common implementation in practice.
Theorem 2. Assume the gradient of the loss function T.(.) is B-smoothness. Assume Polyak-
Lojasiewicz condition, i.e. (()))( e(L(O,) - -..). For iteration steps T 2 0, learning
rate O₁ S 1/8 and /ppp S 1/3, have,
Ex [L)) L. L(On) - L.
II
(1 ar -::e((1(ppppp1 -pp
BE(0,1,"" (T-
Appendix shows the proof. Theorem indicates that RST experiences a linear convergence rate.
3 EMPIRICAL STUDY OF SCHEDULING FUNCTION p(t)
In this section, we would investigate the computation efficiency and the impact on model perfor- mance when training with the RST scheme under different types of scheduling functions p(t).
3.1 BASIC SETTING AND BASELINES
In our investigation of the effect of scheduling functions, we will train models with different schedul- ing functions from scratch to tackle the image classification tasks on Cifar- 100) datasets, and compare the corresponding convergence performance and the incurred extra computation overhead.
For models, we would choose ResNet18 (He et al., 2016) and WideResNett28-U1 (Zagoruyko & Komodakis, 2016) architectures our main target. For data augmentation, we would follow the ba- sic strategy. where each image would be randomly flipped horizontally, then padded with four extra pixels and finally cropped randomly to 32 x 32. Expect for the scheduling functions implemented the RST schemes, all the involved models are trained for 200 epochs with exactly the same hy- perparameters. For each training case, would run with five different seeds and report the average mean and standard deviation these five runs. All the training details could be found in Appendix. Meanwhile, have also reported additional results regarding other model architectures and other data augmentation strategy in Appendix.
Before investigations scheduling functions in RST, we would like to clarity the baseline first. where models are trained with the vanilla SGD scheme and SAM scheme. Table shows the corresponding results, including the testing error rate (Error column), the training time (Time
4

---

Under review as a conference paper at ICLR 2023
Table 1:
Testing error rate of ResNet18 and WideResNet28-10 10 models on Cifar10 and Cifar100 datasets when training with the SGD scheme and SAM scheme respectively.
Cifarl0
Cifar100
Model
Scheme
Anic
Time[m]
Error[%) [%]
Time[m]
SGD
15.8
4.48
15.4
20.7910.12
ResNet18
SAM
1.0
+16.040.5
3.81
+15.8
19.99
SGD
33.3
3.53
33.7
18.69
WRN28-10
SAM
1.0
+2..440.3
2.78+0.07
+28.0+0.4
16.53+0.13
column) and the extra expected propagation count (Ar) column). For the training time, would report the total wall time spent to train for 200 epochs on four A100 Nvidia GPUs. From the table, could find that compared to the SGD scheme, the SAM scheme could indeed improve the model performance, but in the meantime would incur more computations (102% for ResNet18 and 83% for WideResNet28-10).
3.2 IMPLEMENTATION OF SCHEDULING FUNCTION
Here, we will focus on studying three types of function families, which can cover most scheduling patterns. Table 2 shows the basic information regarding the three function scheduling families.
Table 2: Scheduling functions p(t) and extra propagation counts An of the three function families.
Constant
Piecewise
Linear
Scheduling Function p(t)
ac
ap.
8 Tp
att + by
- Up: 1> bpT
Propagation Count Aij
Og
1 + 2aphp - hp ap
pe( (((T)
Constant Function Family In constant scheduling function family, the scheduling probability is
Pt(t) = ae. where ag E [0. 1). Optimizers would select to perform the SAM algorithm with fixed probability a, and the SGD algorithm with ae during the whole training process. This implies that the extra computation overhead for constant scheduling function is proportional to the scheduling probability dg-
Here, we will experimentally investigate group of implementation with constant functions, where the scheduling probability a, will be set from 0.1 to 0.9 with an interval of 0.1. Figure 1A shows the scheduling functions of this group and Figure IB shows the relationship between the extra expected propagation count r (æ-axis) and the extra practical training wall time (y-axis) incurred by selecting SAM algorithm in RST. We could see that for both ResNet18 and WideResNet28- 10 models, all the points locate very close to the reference line (a = y). The actual extra training wall time can be almost fully decided by the theoretical extra A Therefore, could directly use Arj to indicate the extra computation cost for RST in the following demonstrations.
Then, Figure 1C shows the corresponding testing error rates of the two models with error bars (neighbor area) on Cifarl0 (left) and Cifar 100 (right). In the figure, æ-axis denotes the extra Anj and meanwhile the markers of the lines denote the testing error rates of training with the SGD scheme and the SAM scheme. Firstly, we could find that even with the lowest probability a 0.1, as long as SAM algorithm involved during training process. testing error rates could be generally reduced compared to those trained with only the SGD algorithm. But on the other side, model performance can not be im- proved continuously with the growth selecting probability towards the SAM algorithm. Secondly, compared to the SAM scheme, testing error rates would already reach comparable performance when a. = 0.6 in RST, which would save about 40% computation overhead. In particular, when around a = 0.8, models would achieve the best performance, slightly outperforming the SAM scheme (3.65 (3.65%//9.. 655%/1916199 99%.%% for ResNet18 and 2.7 191%//1..199 for WedeResNee28-100 10 in RST). Addition- ally, we could see from the error bars that despite the randomness introduced RST, training would still be fairly stable over the five runs.
scaled by the actual training wall time. And the endpoints on both sides
5

---

Under review as a conference paper at ICLR 2023
Cifart0
Clfar
10
Weiete Ne
estr 300m
0.6
A
00:
toinkigtteratoon
B
63 31
C
Figure 1: (A) Scheduling function plots for the constant scheduling function family. (B) Scatter plot between the extra expected propagation count r) (œ-axis) and the extra practical training wall time (y-axis) incurred in RST. (C) Testing error rates of ResNet18 and WideResNet-28- 10 models with error bars on Cifar10 (left) and Cifar (right) when training with these constant scheduling functions. The markers are scaled by the training wall time.
Piecewise Function Family Generally, the selecting probability in piecewise function would ex- perience stage conversion during training. In the first stage, optimizers would be arranged to perform SAM algorithm with a probability of ap in the beginning 7p training iterations, and then in the second stage, this probability would change to 1 1- ap for the rest training iterations. In our in- vestigation, we would consider totally three typical groups of piecewise scheduling functions, where Figure shows the corresponding scheduling function plots and their final results.
For the first group, we would set ap 0 and change the stage-related parameter by from 0.1 to 0.9 with an interval of 0.1. Now, the optimizer actually behaves in a deterministic manner, which performs SGD algorithm in the first b,T iterations and then switches to SAM algorithm for the rest. Therefore, the larger bp is, the longer SGD algorithm will be performed, and the less extra computation overhead will be incurred. From the results, we could find that for all the training cases in this group, as implementing more iterations with SAM algorithm, we could get better performance gradually, which could achieve better performance than those trained with the SAM scheme. And the best performance between this group and the constant group are very close (3.66% for ResNet18 and 2.69% % for WideResNNe22-100 10 in this group).
Next, in the second group, we would arrange training in an opposite way from piecewise group 1, where we will keep all the settings except deploying ap 1. Optimizers would perform SAM algorithm in the first T iterations and then switch to SGD for the rest steps. Actually, models could not get good performance under such arrangement. The results show that training needs to accumulate sufficient SAM iterations to completely outperform SGD scheme. Models could reach competitive performance only when performing SGD algorithm in the last few iterations. Intuitively, implementation pattern of piecewise group 2 would somewhat go against the core of sharpness- aware learning. Frequently implementing SGD algorithm near the end of training would be harmful to the convergence to flat minima.
Unlike previous patterns, in piecewise group 3, we fix by = 0.5 and change ap from 0.1 to 0.9 with an interval of 0.1. Now, optimizers will pick SAM algorithm with probability ap for the first half of training iterations and then switch the probability to 1-aa for the rest. For all the training instances in this group, we have A 0.5. And, the actual training wall time between these cases are rather close (Time[m]: +8.2(++0.4 for ResNet18 and +13.9(=0077 for WideResNNe28-100 Note that the results of this group are plotted against the evolution of ap. not the propagation count. We could see in the results that model performance would gradually get higher as the growth probability of implementation with SAM algorithm in the second stage. This somehow again confirms the previous demonstration of avoiding frequently implementing SGD algorithm near the end of training.
Linear Function Family For linear scheduling functions, the selecting probability p(t) is sched- uled linearly, changing monotonously with either an increasing or a decreasing pattern. Optimizers would select to perform SAM algorithm with decreasing probability when aj S O while with in- creasing probability when at 0. Notably, from the summary table 2, the computation overhead of such implementation is actually decided by the scheduling probability at T/2. We would focus on two typical groups of linear scheduling functions in our experiments. Figure show the scheduling functions and the results.
In the first group, we would schedule the functions to pass through two given points, where the first point is (T/2, m) and the second point is either (0, 0) or (1, 1) depending on the value of m. Here, the parameter m denotes the probability to be at the training iteration T/2, And we would

---

Under review as a conference paper at ICLR 2023
Piecewise Group 1
WoodhalNeaker Restled
Teat Eree man
Probabbtyy
0.5
0.0
84
8.3
1.0
84
training Poration!
exposie proparation
Piecewise Group 2
1.0
Texting Error Bate
Probabbyy
0.5
3.5
trahing Beration
www.seeddppggttoiicccom CounT
00733 espected wwwpagationnncmmmmmmmmoom COUTT
Piecewise Group 3
4.5
1.0
Tra
Reshee
Teathe croue Stace
0.5
0.0
08
8.4
0.5
trahting Reration
paramict 4g
Linear Group
WARRENNSE-- Reclied
1.0
Reshet
E
Testrrg Croer Taam
Probabbty
0.6
0.0
1.0
00
08
training
www.eepppppattttee.oooo COunT
expe wwwpagattpo.coooo
Linear Group
Teating Erooe Ratm [1)
Probablty
0.0
8.4 parameter
8.6
8-8
68
trathing
Figure 2: Testing eтrоr rates of ResNet18 and WideResNet-288 28- 10 models with error bars on Cifar10 (left) and C0 (right) when training separately with the three groups of piecewise scheduling functions and the two groups of linear scheduling functions in RST. set it from 0.1 to 0.9 with an interval of 0. Clearly, in this group, the probability of selecting SAM algorithm would increase over the iterations. Also, as m increases, SAM algorithm would experience overall higher probability of selection. We could find in the results that as performing more SAM algorithm, model performance would be more and more better. And the trend of model performance in this group would be quite similar that in piecewise group 1. Actually, these two groups share very close selection patterns in general, where the scheduling probability is changed instantaneously in piecewise group while becomes gradually this group.
As for the second group, the scheduling functions would pass through two points that are (T/2, 0.5) and (0, 61). This means that training will always incur 0.5 extra propagation count in expectation, An = 0.5. From the results, we can find that similar to those in piecewise group 3, model perfor-
7

---

Under review as a conference paper at ICLR 2023
Cifar10
Cifar100
Corolant Receolot
Placeetor
Phocetact
Poocooso
Testing Error Rote (4]
Phocentis
Linenn
Testing Emror Rate
Linser
SCTV
30
X
2.5
16
60
61
016
LB
1,0
(16
10
extra expected propagation count
propagarion counte
Figure 3: Summary plot of model performance in regards to the extra computation overhead for all the scheduling function cases.
mance will also progressively become higher, but more mildly. Likewise, the two groups also have close selection pattern, as in the same way of that between piecewise group I and linear group
3.3 SUMMARY
To give a summary view of these scheduling functions, Figure 3 gives the seatter plot of WideResNee28-10 10 between the model performance and the incurred extra propagation counts for all the scheduling function cases. From previous demonstrations and the figure, we can conclude,
Avoid to schedule the SGD algorithm with relatively high probability near the end of train- ing since it would largely harm the training.
Generally, scheduling SAM algorithm with higher probability in total would bring better model performance, where the best model performance in RST would outperform those in SAM scheme.
Compared to other schedules, simple constant scheduling functions could give decent model performance. So. we recommend using constant scheduling functions in practice for both their simplicity and effectiveness.
4 GENERAL FRAMEWORK FOR RST
Recall that from Equation 7, SAM training is actually regularizing the gradient norm with Ysam P. and RST to mix SGD algorithm and SAM algorithm would have a scaling effect on this penalty by a factor of Pt- However, when the scheduling probability Pt is low, RST may be unable to provide sufficient equivalent regularization effect on gradient norm. This motivates to expand RST a general form (G-RST) which mixes between SGD algorithm and GNR algorithm (Equation 6) such that G-RST could freely adjust the scaling effect of the penalty degree on gradient norm,
g: = (1 - X.) gi
(agd) +
Xx. 't (gnr)
In this way, G-RST would be given an extra freedom to control the scaled penalty degree via y in GNR, which would be Tist = Pt Ygur- allows training impose arbitrary regularization on gradient norm while enjoying a high probability of selecting SGD algorithm.
In our following experiments, we would use the constant scheduling functions because of their efficiency and simplicity as demonstrated previously. Here, we would consider p(t) 0.3, so we need to set 7gmr = 2 when mixing, to provide an equivalent regularization as that in SAM scheme.
We would first train models with G-RST on Cifar datasets, which involves both CNN models and VIT models (Dosovitskiy et al., 2021). For CNN models, we would keep the basic settings the same as those in the previous section. As for VIT models, would train each case for 1200 epochs and adopt some further data augmentation to get the best performance. Note that the base algorithm switch to Adam in ViT models. All the training details reported in the Appendix.
Table 3 shows the corresponding results these models on Cifar datasets. We could observe from the table that compared to the SAM scheme, G-RST could improve the model performance further to some extent while saving 50% of the extra computation overhead for all the training cases. This indicates that adjusting the penalty coefficient in RST give comparable effect as that in SAM.
8

---

Under review as a conference paper at ICLR 2023
Table 3: Testing error rate of CNN models and VIT models on Cifar10 and Cifar100 datasets when training with SGD, SAM and the G-RST where p(t) = 0.5.
Learning Methods
C-10&100
Cifarl0 Error[% %
Cifar100 Error[%]
Time[m]
SGD
9.940.2
2.7400.00
25.22 10.31
VGG16BN
SAM
20.3
5.24 20.08
24.23 10.29
G-RST[50%]
+4.4
5.21
24.37=0.29
SGD
15.610.3
1.100
20.79. 10.12
8
SAM
+15.9
3.81 20.07
10.13
G-RST[50%]
+7.920.3
3.65_0.10
19.95
SGD
33.5
3.53
18.99
WRN28-10
SAM
+27.7+0.4
2.78-0.07
16.53
G-RST[50%]
+14.3
2.68_0.06
16.19
SGD
119.7
3.42=0.09
17.82
Pyramid164
SAM
+83.210.9
2.61 .0.07
14.80
G-RST[50%]
+42.0
2.50
14.55
Adam
189.0+1.8
9.45=0.18
34.79=0.22
ViT-Ti16
SAM
+165.2 22.4
8.59 10
32.48 10.31
G-RST[50%]
+82.912.5
8.31 10.18
32.17 1.0.44
Adam
247.9+2.9
6.89
27 27.4840.33
ViT-S16
SAM
+263. 12.1
5.52.10.20
26.53 10.27
G-RST[50%%
+131.913.
5.39
26.24 10.28
Adam
407.8
6.56
27.95 10.28
VIT-B16
SAM
+400.2
2.45
26.51 +0.30
G-RST[50%)
+199.643.3
5.58
26.27 00.26
Table 4: Testing error rate CNN models on ImageNet datasets when training with SGD, SAM and the G-RST where p(() = 0.5.
Methods
Time[m]
Top-1[%]
Top-5[%]
SGD
750+9
23.6440.11
7.01
ResNe150
SAM
+51845
23.1640.11
6.72+0.06
G-RST[50%
+259+12
22.8240.19
6.63
SGD
1255
21.93
6.11
ResNet101
SAM
+904
21.0240.10
5.31
G-RST[50%]
+451+14
20.78+0.12
5.1640.10
Following the same setting of Pt as that on Cifar datasets, we would train ResNet-(50, 101) models on ImageNet for 100 epochs to further investigate the effectiveness of G-RST on large-scale dataset. Table shows the final results, where each case is trained over three random seeds. Likewise, we find that G-RST also give better model performance while being 50% less computational expensive than SAM scheme, which again confirms the effectiveness of G-RST.
5 CONCLUSION
We propose simple but efficient training scheme, called Randomized Sharpness-Awar Training (RST), for reducing the computation overhead in the sharpness- aware training. In RST, optimizers will be scheduled to randomly select from the base learning algorithm and sharpness- aware learning training scheme at each training iteration. Such a scheme can be interpreted as regularization on gradient norm with scaling effect. Then, we theoretically prove RST converges in finite training iterations. As for the scheduling functions, we empirically show that simple constant scheduling functions achieve comparable results with other scheduling functions. Finally, extend the RST to general framework (G-RST), where the regularization effect can be adjusted freely. We show that G-RST can outperform SAM some extent while reducing 50% extra computation cost.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Dara Bahri, Hossein Mobahi, and Yi Tay. Sharpness-aware minimization improves language model generalization. arXiv preprint arX1v:2110.08529, 2021.
Xiangning Chen, Cho-Jui Hsieh, and Boqing Gong. When vision transformers outperform
resnets without pretraining or strong data augmentations. arXiv preprint arXiv:2106.01548, abs/2106.01548, 2021.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.048005 2018.
Terrance Devries and Graham W. Taylor. Improved regularization of convolutional neural networks with cutout. arXivPreprint, abs/1 708.04552, 2017. URL http:://axxii..rrg 04552.
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In Proceedings of the 34th International Conference Machine Learning, ICML 2017, volume 70, pp. 1019-1028, 2017.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, 2021.
Jiawei Du, Hanshu Yan, Jiashi Feng, Joey Tianyi Zhou, Liangli Zhen, Rick Siow Mong Goh, and Vincent Y. F. Tan. Efficient sharpness-aware minimization for improved training of neural net- works. arXivPreprint, abs/2110.033 141, 2021a.
Jiawei Du, Hanshu Yan, Jiashi Feng, Joey Tianyi Zhou, Liangl Zhen, Rick Siow Mong Goh, and Vincent YF Tan. Efficient sharpness-aware minimization for improved training of neural net- works. arXiv preprint arXiv:21 2021b.
Jiawei Du, Daquan Zhou, Jiashi Feng. Vincent YF Tan, and Joey Tianyi Zhou. Sharpness-aware training for free arXiv preprint wrXiv:2205.14085 2022.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur, Sharpness-awarr minimiza- tion for efficiently improving generalization. In 9th International Conference on Learning Rep- resentations, ICL. 2021, 2021.
J. Friedman. Greedy function approximation gradient boosting machine. Annals of Statistics, 29, 2001.
Dongyoon Han, Jiwhan Kim, and Junmo Kim. Deep pyramidal residual networks, In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, pp. 6307-6315, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 20/6, pp- 770-778, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural Comput., 9(1):1-42, 1997.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In 5th International Conference on Learning Representations, ICLR 2017, 2017.
Andrei V. Konstantinov, Lev Utkin, and Vladimir Muliukha. Gradient boosting machine with partially randomized decision trees. In 28th Conference of Open Innovations Association, FRUCT 2021, Moscow, Russia, January 27-29, 202 1, pp. 167-173. IEEF, 2021.
Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi. ASAM: adaptive sharpness- aware minimization for scale-invariant learning of deep neural networks. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, volume 139 of Proceedings of Machine Learning Research, pp. 5905-5914, 2021.
10

---

Under review as a conference paper at ICLR 2023
Yong Liu, Siqi Mai, Xiangning Chen, Cho-Jui Hsieh, and Yang You. Towards efficient and scalable sharpness-aware minimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12360-123700 2022.
Haihao Lu and Rahul Mazumder. Randomized gradient boosting machine. SIAM J. Optim., 30(4): 2780-2808, 2020.
Peng Mi, Li Shen, Tianhe Ren, Yiyi Zhou, Xiaoshuai Sun, Rongrong Ji, and Dacheng Tao. Make sharpness-aware minimization stronger: A sparsified perturbation approach. arXiv preprint arXiv:2210.05177, 2022.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 779-788, 2016.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In 3rd International Conference on Learning Representations, ICLR 2015, 2015.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Proceedings of the British Machine Vision Conference 2016, BMVC 2016, 2016.
Yang Zhao, Hao Zhang, and Xiuyuan Hu. Penalizing gradient norm for efficiently improving gen- eralization deep learning. International Conference on Muchine Leurning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 26982-26992, 2022.
Yaowei Zheng, Richong Zhang, and Yongyi Mao. Regularizing neural networks via adversarial model perturbation. In IEEE Conference Computer Vision and Pattern Recognition, CVPR 2021, pp. 8156-8165, 2021.
Juntang Zhuang, Boqing Gong, Liangzhe Yuan, Yin Cui, Hartwig Adam, Nicha C Dvornek, sekhar tatikonaa, James Duncan, and Ting Liu. Surrogate gap minimization improves sharpness-awan training. In International Conference on Learning Representations, 2022.
A PROOF OF THEOREM & 2
A.1 PROOF OF THEOREM
In randomized sharpness-aware training (RST), weights 0, are updated stochastically with random variable X, ~ B(1,p:),
= - (x.V((0))))))
(9)
For 3-smoothness functions, we have
L( < L(02) VI(O.) - -02) + -2)++ +1,100 - -02112
(10)
Then, we set 0, = 0.+1 and 02 = O.,
9 01 0.11
L(O..)) < L(O,) + (V.(O.), 0.+1 0,) X.P.VI + 3 VI + X.
((O,))
0
[((O.) O. (V. VI X.P.V VL (.))
(11)
For at 5 1/3,
((0,1)) < L(O.) a, (VL(0,) (1 L(..)( 0: VL + +
(12)
11

---

Under review as a conference paper at ICLR 2023
Next, add 2 (V and subtract 7: [V
L(0(+1) - L(O)) + du [ (L(C
VL(0. Xipo
0 |V X,,.))((0))) -0 [V.(0.)]1
S I(00) + 04
2 VI. (.VVI(())) VI(0,)1 at
(13)
O4 S L(O.) + 2 11 - IVVL(.))11
Qt
S L(O.)
04
2 (1 x
So, P: 1/8 such that the loss would decrease continuously in training,
L(0,+1) - L(O.) s L(O.-1) L(0o)
(14)
Rearrange Equation 13,
[1 - / L(0))) - L(0,+1)
(15)
2
Taking expectation gives,
Qt (1 -1--X: (1-X S Ex [L(0.] Ex [L(0++))
Ex
2
O4 2
(1. + Pt 04 (1 - (. Ex [T(0.)] Ex [T((0+1)
(1 - pt)
(16)
2
0t (1 - -1 Ex [L(0.] Ex [1(++1)]
2
For p. = (( in SAM optimization, the Equation 16,
-. Ex [L().)] Ex [L(0,+1)] di (1
2
an
V((0))1
|VL < Ex [L(0)) Ex [L(0++1)]
2
2
a.p.p2 2 32
(17)
|L(00)1
VL Ex (40))) Ex [L(0+1))
0..)
0=pep2
| Ex [L(0,) - Ex [L(0++1)] +
2
Then, sum over the training steps,
Ospppp2 32
Σ
Or 2
([V < L(On) L,, +
Σ
(18)
IE (O. 1-
Here, L(O,) is the loss of the initialization model and L, denotes the optimal point, L, min L(0).
Since minke (0.1,-. 1- [[ < [|2(0) we have.
[[V < 2(L(Oo) - -Lo) +
min ee 10 T-1]
(19)
where,
(20)
ΣiE(o.....T-1) TT1) 0x
Generally, Equation 19 indicates that for e-suboptimal termination criteria [|L(0,)] < c, hybrid training would satisfy such convergence condition in finite training steps.
12

---

Under review as a conference paper at ICLR 2023
Further, for constant learning rate schedules a, C/B cosine learning rate schedules dg 2C/B ( + cos( m)), and constant scheduling functions p: = p. we have
[VL( ( - L.) + pp² 82
min EE (0,
(21)
CT
Here, we use Et= cos( ( O, which we would prove in the following lemma. In other words, the epsilon e associated with the OL1/T).
For decayed learning rate schedule C/t, and constant scheduling functions Pt = p, we have VL(0,)11 - 2(L(On) - L.) C'log T
min (0.1.
+ pp²23 8²
(22)
In other words, the epsilon is associated with the (11 log T). Lemma 1. For t E ...,...... ..2.,...TT))) we have
t cos(
Σ 1=U
x) 7-
(23)
Proof For trigonometric functions,
T
(24)
1=0
where g cos). We would use the Euler's identity,
f cos x + isn
(25)
Therefore, have R(c and sin 3fe where R(-)) and (.) denote the real part and imaginary part.
In this way, for cos, Equation 24 would be,
cos(
Rlett
T
R
c
[=0
T41
RY
1 e 7
(e i 1
e x
(26)
R{
e'27 (e
1 (e-
e e
sin( T+1 TT)
R(e 2r
27
sin 27" /T
sin (
T+1 27
cos(
2T
sin
0 (cos π
0)
2
A.2 PROOF OF THEOREM 2
From the Polyak-Lojasiewice condition,
e(((8,) VL(0))) 2(((1)) L.)
(27)
2
13

---

Under review as a conference paper at ICLR 2023
From the previous Equation 16, we would have,
at (1 (1-=p((5"))]L((00000 < Ex [L().)] Ex (L(0)-1)]
2
a p)pp [L.)] L.) s Ex [L().)] Ex (L(0t-1)]
a:e(1 - ) (Ex [L())) - L.) s (Ex [L(0+)] - L.) - (Ex [L(0++1)] -t.)
(28)
Ex [[(+11)]
Ex [L)):)] L, < agg(1 32)
Then, performing iterative multiplication over the training steps gives,
Ex [L(0,)] - L.
II
(1-are(L -DoP(
I(On) (0(0,1,",T-1)
(29)
End of the proof. C
B ADDITIONAL RESULTS
B. TRIGONOMETRIC SCHEDULING FUNCTION
We would like to use WideResNer28-10 to further investigate the scheduling functions which are trigonometric functions Ptr RST. Here, we would confine the trigonometric functions only sinusoidal functions and cosine functions. And more specifically, we focus on investigating four scheduling functions,
Poost (1)
+ 2 CO8 S.
Poos2 (t) 1 Poosl (t)
COS f.
(30)
Psin (l) sin न
(fin2(t) = Psin1 (t) sin f
Note that all these functions are in the range between and 1.
Pas (1)
Posa
Pan (L)
Puu
Pripaky
60
00
Figure 4: Scheduling function plots for the four trigonometric scheduling functions. The blue points stand for the instance of random variable X.
Table S: Testing error rate of WideRessNe18-110 10 models on Cifar10 and C0 datasets when training with the four trigonometrie scheduling functions.
Cifar-10&100 100
Cifar10
Cifar100
WideResNNe12-110
Time[m]
Ai
Error[]] []]
Error[%]
SGD
33.5+0.
0
3.53
18.99
SAM
+27.7+0.4
1
2.78+0.07
16.53+0.11
RST: Peosi (L)
+14.7+0.6
0.5
3.1640.00
17.08
RST: Peos2 (t)
+14.9+0.7
0.5
2,86+0.10
16.77
RST: Psint
+17.640.4
2/ RJ 0.63
2.8 2.81+0.00
16.69 +0.12
RST: Psin2(
+17.940.6
2/x
3.21 10.13
17.15
14

---

Under review as a conference paper at ICLR 2023
Figure 4 shows the training scheme plots of the four functions and Table 5 shows the final results. From the table, when training with these trigonometric scheduling functions, training will incur 50% extra expected average propagation count for cosine functions and æ x/2 64% for sinusoidal functions.
For cosine functions, could find that their pattern of scheduling probability could be quite close to linear functions. This could lead to that they may yield very similar performances. As for sinusoidal functions, implementations would present monotonously increasing or decreasing probability for the first half iterations and then switch to the opposite for the rest. Compared to that of cosine functions, as SAM would be implemented with more frequency total, the corresponding results would be better. Additionally. the results have also confirmed that the performance would be degenerate when SGD is frequently selected near the end of training. And in summary, training with such complex trigonometric scheduling functions could not present better results than that with simple constant scheduling functions. We would still recommend to use simple constant scheduling functions in practical implementation.
B.2 Trst IN G-RST
Based on the demonstrations the G-RST, we would know that G-RST could adjust the regular- ization effect on the gradient norm freely for a given selecting probability. Therefore, we would perform some more tuning on the "/nrr to be mixed in RST to present the relationship between the model performance and the equivalent regularization degree "/s- Here we would perform a grid searching the selecting probability from 0.1 to 0.9 with interval of 0.2, and then set the Yeur (Equation 9 in the main paper) in the RST to fix the equivalent regularization effect "/ra across 0.5 to 1.5.
Table shows the corresponding 2D plot. From the table, we could find that when the selecting probability P; is very low, even if we impose high regularization penalty, models could not be trained to achieve good performance. This is mainly because that based on "ist TgarPr, for these low Pt, we have to mix a very high "ggrr get fair equivalent effect Tist. When the 7gar very high in GNR, according to the paper Zhao et al. (2022). it would cause a lose of precision on the approximations on the Hessian multiplication. Secondly, we could also find from the figure that when the equivalent regularization degree "Yaa is around the range from 0.8 to 1, models could achieve the better performances than others. Imposing too much regularization on the gradient norm would instead harm the performance. For the fixed Yist, increasing the selecting probability Pe would somewhat improve the model performance, but not in a significant manner.
Cifaroo
Cifari00
1.-
equivalent regularization degree
equivalent regularization degree
14-
1.-
12-
09
0.-
08-
06
05
2000
03-
1614
02
04
0.6
088
0.2
0-1
066
08
selecting pr probability
selecting probabilityy P.
Figure 5: 2D image plot between the selecting probability Pt and the equivalent regularization degree for WideResNer28-110 when training with RST.
In summary, is recommended to set a moderate selecting probability and combine with a proper Ygar that could lead Yist around. In this way, training would enjoy gain the computation efficiency and give satisfactory performance at the same time. And the Table in the main paper actually follows
15

---

Under review as a conference paper at ICLR 2023
B.3 EXPERIMENT RESULTS WHEN USING CUTOUT REGULARIZATION
In addition the basic data augmentation strategy used in the previous section, we would also investigate the effect when using the Cutout Regularization Devries & Taylor (2017). Here, we would choose WideResNet12-111 as our main experiment target. Also, the training hyperparameters are the same as them used in the previous sections.
The tables below show the final results, where trainings are going be separately scheduled by constant scheduling functions (Table 6), the first group of piecewise scheduling functions (Table 7) and the first group linear scheduling functions (Table 8) and trigonometric scheduling functions (Table 9). From the results, we would come to the same conclusions as those in the summary sec- tions. In short, constant scheduling functions would be a good choice for practical implementation, which would be simple to implement and be able to yield at least comparable performance to other scheduling functions.
Table 6: Testing error rate of WideResNNe22-10 models Cifarl0 and Cifar100 datasets with Cutout regularization when training with constant scheduling functions.
Cifar- 10 & 100
Cifarl0 Error[%]
Cifar100 Error[%]
Training Scheme
Arie
SGD
2.81 10.07
16.91 10.10
SAM
1.0
2.43_0.13
14.87+0.16
2.67=0.05
16.1440.18
0.2
20.06
15.97
0.3
2.46_0.06
15.5640.14
0.4
2.40
15.1740.22
RST
0.5
2.32=0.06
15.1040.11
0.6
2.31
14.9640.00
0.7
2.25
14.94
0.8
2.23
14.81
0.9
2.31 10.06
14.71
Table 7: Testing error rate of WideResNee28-10 models on Cifarl0 and Cifar100 datasets with Cutout regularization when training with constant scheduling functions.
Cifar- 10 100
Cifar10 Error[%]
Cifar100
Training Scheme
Ario
Error[%]
SGD
2.81
16.91
SAM
1.0
2.43
14.87
0.1
2.69 10.04
15.70.1..1
0.2
2.51 2..07
15.37
0.3
2.47 2.47
15.38
0.4
2.44-0.01
15.2440.35
RST
0.5
2.46-0.02
15.2440.22
0.6
2.34=0.02
14.98+0.28
0.7
2.34=0.03
14.99+0.26
0.8
2.30
14.844.0.9
0.9
2.29
14.7540.21
16

---

Under review as a conference paper at ICLR 2023
Table 8: Testing error rate of WideResNee28-10 models on Cifar10 and Cifar100 datasets with Cutout regularization when training with constant scheduling functions.
Cifar- 10 & 100
Cifar10 Error[%]
Cifar100 Error[%]
Training Scheme
Aric
SGD
2.81 =0.07
10.10
SAM
1.0
2.43=0.13
14.87=0.16
0.1
2.63 2.63=0.02
15.83
0.2
2.58 20.06
15.67
0.3
2.45
15.33
0.4
2.37=0.07
15.30=0.19
RST
20.02
15.33
0.6
2.27=0.10
14.97
0.7
2.25_0.03
14.95+0.23
0.8
2.22
14.6840.14
0.9
2.23
14.79+0.05
Table 9: Testing error rate of WideResNet28-10 models on Cifar10 and Cifar100 datasets with Cutout regularization when training with constant scheduling functions.
Cifar-10 10 & 100
Cifar100 Error[%]
Cifar10
Training Scheme
Arje
Erro[[%]
SGD
2.81
16.91
SAM
1.0
2.4340.13
14.87
Peos2 (t) Psin1 (t)
0.5
2.35
15.02
0.5
2.27
14.85
B.4 ADDITIONAL EXPERIMENT RESULTS FOR OTHER MODELS
Other than ResNer18 and WideResNer28-100, we would also investigate another model architec- ture, including the VGG16 Simonyan & Zisserman (2015) with batch normalization and Vision Transformer. From the previous results, we could see that the constant scheduling functions would already provide representative results. So here we would only investigate the results when trained with constant scheduling functions to make comparisons with the baselines.
Table 10: Testing error rate of VGG16-BN models Cifar 10 and Cifar 100 datasets when training with constant scheduling functions.
Cifar- 10 100 Ari
Cifar10 Error[%]
Cifar Error[%]
Training Scheme
SGD
5.74
25.22+0.31
SAM
1.0
5.24=0.08
24.23+4.29
0.1
5.59
25.10.10.21
0.2
5.45
24.97
0.3
5.39
24.88+40.2
0.4
5.38
24.73
RST
0.5
5.35=0.05
24.50+40.17
0.6
50.07
24.42
0.7
5.2940.04
24.31 10.18
0.8
5.14
24.17
0.9
5.21
24.08
17

---

Under review as a conference paper at ICLR 2023
Table 11: Testing error rate of ViT-SI6 models on Cifar10 and Cifar100 datasets when training with constant scheduling functions. Note that the hyperparameters is different from those in the previous section. Here we only train for 300 epochs without mixup augmentation.
Cifar- 10 100
Cifarl0 Error[%]
Cifar
Training Scheme
Afic
Error[%] [%]
SGD
12.59
37.8210.31
SAM
1.0
10.51
36.40.10.26
0.1
11.9440.55
37.21 #:0225
0.2
11.79
37.10#0.29
0.3
11.40.10.5
36.6410.22
0.4
11.17:0.55
36.5840.17
RST
0.5
11.37
36.5210.20
0.6
11.31
36.10110117
0.7
10.85+0.11
36.3640.07
0.8
11.78
36.20.0..00
0.9
11.78+0.67
36.38
We could in the table that RST again could boost the computational efficiency and in the mean- time acquire better model generalization compared to that trained using the SAM scheme.
B.5 USING RST SCHEME ON OTHER SAM VARIANTS
In this section, we are going further show the effectiveness of our RST on SAM variants, where we would use ASAM (Kwon et al., 2021) and GSAM (Zhuang et al., 2022) as our investigation target. For both ASAM and GSAM, we would compare them with using our RST and G-RST schemes. Here, based on the previous demonstrations, the selecting probability in RST and G-RST set constantly to 0.5. And for G-RST, since the essence of these SAM variants is regularizing the gradient norm, would double the regularization effect in G-RST, the same the implementations in previous experiments. Table 12 shows the final results.
As we could see in the table, when using RST on ASAM and GSAM, could obtain similar results as using RST on SAM. Specifically, since RST and G-RST randomly selecting between sharpness-awee aware learning algorithm and the base learning algorithm, the computational efficiency could be largely improved for both ASAM and GSAM. And as previous demonstrations, RST would weaken the regularization effect, we could that the corresponding performance would be relatively lower than the standard sharpness-awar training. When doubling the regularization effect in G-RST, could get comparable results with the standard sharpness-aware training, which again confirms the effectiveness of our method.
B.6 MIXING RST SCHEME WITH OTHER EFFICIENT SAM TECHNIQUES
In RST, the optimizer would choose to perform the base learning algorithm and the tharpness-awarc algorithm. When selecting sharpness-awae aware algorithm, could meanwhile adopt other efficient techniques to further improve the training efficiency. Here, we would study the mixing effect of RST with separately LookSAM (Liu et al., 2022) and weight masking techniques (Mi et al., 2022; Du et al., 2021b). Table 13 shows the corresponding results.
As could in the table, for all these efficient techniques, RST could improve the compu- tational efficiency further. However, if the selecting probability RST relatively low (0.5 in the table), it may harm the mixing effect. On the other hand, properly raising the selecting probability (0.75 in the table), it is possible to acquire comparable results with these efficient techniques.
18

---

Under review as a conference paper at ICLR 2023
Table 12: Testing error rate of CNN models and VIT models on Cifar10 and Cifarl (0 datasets when training with SGD, SAM and the G-RST where p(t) = 0.5.
Learning Methods
C-10& Time[m]
Cifar10 Error[%]
C0 Error[%]
ASAM ASAM & RST
19.0
5.2811.06
24.08 10.14
14.4
5.50
24.49 10.16
ASAM & G-RST
+4.540.3
5.32
24.11
VGGI6BN
GSAM GSAM & RST
+9.8+0.4
5.74
25. 25.2224031
+4.7.0.3
5.24
24.23
GSAM & G-RST
+4.84u.4
5.21 40.08
24.37
ASAM ASAM & RST
+15.8+0.3
3.77 20.05
20.02
+8.0+0.2
3..09
20.31
ASAM & G-RST
+7.9+0.3
3.65
19. 19.95=0.18
ResNet18
GSAM GSAM & RST
+17.4
3.81 10.04
19.91 10.13
+8.9+0.4
3.99
20.43
GSAM & G-RST
+8.9+0.3
3.70+0.11
20.10+0.18
ASAM ASAM & RST
+27.610.3
3..3
16.40.10166
+14.440.3
2.7840.07
16.81
ASAM & G-RST
+14.340.5
2.68
16.59
WRN28-10
GSAM GSAM & RST GSAM & G-RST
+30.640.6
2.74
16.51
+15.540.5
2.95
16.95
+15.7
2.71
16.47
19

---

Under review as a conference paper at ICLR 2023
Table 13: Testing error rate of CNN models and VIT models on Cifar10 and Cifar datasets when training with SGD, SAM and the G-RST where p(t) = 0.5.
Learning Methods
C-10&100
Cifar10 Error[%]
Cifar100
Time[m]
Error[% [6]
SGD Loo)SAM(5))
9.920.2
5.74
20.22
+3.0+0.2
5.49
24.71
LookSAM(5) & G-RST[50%]
+1.640.3
5.56
24.88
LookSAM(5) & G-RST[75%)
+2.4+0.3
5.
24.34=0.23
SGD ESAM
16.7
6.12
25.5610.22
+13.510.6
5.50
14.49
VGG16BN
ESAM & G-RST[50%]
+6.940.4
5.92+0.06
24.91
ESAM & G-RST[70%]
+10.240.5
5.38
20.57
SGD
16.7
6.12
25.56+0.22
SSAM
+15.540.4
5.64
24.61
SSAM & G-RST[50%]
+8.1
5.99
25.03 25.0310.25
SSAM & G-RST(75%)
+12.010.6
5.59 10.07
24.66
SGD
15.6
10.10
20.79 10.12
LookSAM(5)
+5.610.4
4.06 10.09
20.30 10.16
LookSAM(5) & G-RST[50%]
+3.010.2
4.18
20.44
LookSAM(5) G-RST[75%]
+4.4+0.3
3.94
20.11
SGD
24.4
4.66
20.98 20.20
ESAM
+18.640.4
4.05
20. 20.14
ResNet18
ESAM & G-RST[50%]]
+9.540.6
40.04
20. 20.
ESAM & G-RST(75%)
+14.040.5
4.08
20.21
SGD
24.440.6
4.66
20.98
SSAM
+21.0+0.4
3.89 20.01
20.17
SSAM & G-RST[50%
+10.7
4.03 10.07
20.41 10.13
SSAM & G-RST[75%]
+15.810.8
3.83
20.19
SGD
33.5+0.5
3.53+0.10
18.99 18.9940.12
LookSAM(5)
+9.440.6
3.15
17.27
LookSAM(5) & G-RST[50%]
+4.940.4
3.22
17.55
LookSAM(5) G-RST[75%)
+7.1
3.04
17.09
SGD
109.9+1.2
3..7
19.13
ESAM
+91.440.9
2.96
16.90
WRN28-10
ESAM & G-RST[50%]
+45.9+1.4
3.20
17.58
ESAM & G-RST(75%)
+69.2
2.99 10.1
16.94
SGD
109.9+1.2
3.97
19.13
SSAM
+107.3+1.7
3.10+0.04
16.97
SSAM & G-RST[50%] SSAM & G-RST[75%]
+58.31.2.
3.24
17.11
+81.2+1.9
3.12+0.10
16.56
Following the paper (Liu et al., 2022), LookSAM(5) denotes that update the descent gradient in SAM
algorithm every five implementation iterations.
Unlike LookSAM, ESAM SSAM are both implemented on git repository http://github.com/ com/iMuuucoomM-----i-ii SetpSSparr--Shapneeeee Paappeess--wrre--aee Lwwar--iiimizztionn where one A100 GPU is used. And SGD baseline is also
obtained based on this repository.
20

---

Under review as a conference paper at ICLR 2023
C
TRAINING DETAILS
Code is available at gwtggb.cco//uustiso. parameters are deployed as below,
www.:/uttoooaay////hssiiii--.--. The basic training hyper-
Table 14: The basic hyperparameters for training CNNs on Cifar dataset.
SGD Scheme
SAM Scheme
RST Scheme
Epoch
200
200
200
Batch size
256
256
256
Base optimizer type
SGD
SGD
SGD
Basic learning rate
0.1
0.1
0.1
Learning rate schedule
cosine
cosine
cosine
Weight decay
0.001
0.001
0.001
Weight decay (PyramidNet) P in SAM
0.0005
0.0005
0.0005
0.1
0.1
Table 15: The basic hyperparameters for training ViTs.
Adam Scheme
SAM Scheme
RST Scheme
Data augmentation
mixup
mixup
mixup
Epoch
1200
1200
1200
Warmup epoch
40
40
40
Batch size
256
256
256
Base optimizer type
Adam
Adam
Adam
Basic learning rate
0.0005
0.0005
0.0005
Learning rate schedule
cosine
cosine
cosine
Weight decay in SAM
0.03
0.03
0.03
0.1
0.1
Table 16: The basic hyperparameters for training CNNs on ImageNet dataset.
SGD Scheme
SAM Scheme
RST Scheme
Epoch
100
100
100
Batch size
512
512
512
Base optimizer type
SGD
SGD
SGD
Basic learning rate
0.2
0.2
0.2
Learning rate schedule
cosine
cosine
cosine
Weight decay
0.0001
0.0001
0.0001
in SAM
0.05
0.05
21


Reviews:
[{"review_id": "riq8IbU7SzK", "reviewer": "Reviewer_eiyQ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Recently, sharpness-aware training such as SAM has drawn large attention, because of its provably guarantee its significant performance improvement. However, SAM requires a huge additional computation cost, and it is not easy to adopt SAM on a large-scale model or a real-time analysis system. This paper provides an efficient computation for SAM, RST, and G-RST. They adopt randomized sharpness-aware training. The idea is simple, and it works well.", "summary_of_the_review": "This paper raises an interesting concept, but similar ideas were already suggested in other venues. \n\nDiscussion and additional experimental comparisons are necessary.\n", "strength_and_weaknesses": "Strength points\n1. This paper aims to improve the efficiency of SAM, the important problems.\n2. This paper raises interesting ideas such as randomized sharpness and its general extension G-RST that adjust regularization degree freely for any scheduling function.\n\nWeakness points\n1. The concept of this paper is similar to the other papers [1]. This paper should cite and discuss the difference between this paper and another paper.\n2. This paper is necessary to include diverse related works and efficient computation of SAM. For example, there are three research works, [2], [3], [4]. This paper only cites and discusses the [4]. However, there is no experimental comparison with [4]. I suggest that this paper should discuss and compare the performance with [2], [3], [4].\n\n[1] Zhao, Yang, Hao Zhang, and Xiuyuan Hu. \"SS-SAM: Stochastic Scheduled Sharpness-Aware Minimization for Efficiently Training Deep Neural Networks.\" arXiv preprint arXiv:2203.09962 (2022).\n[2] Du, Jiawei, et al. \"Sharpness-Aware Training for Free.\" arXiv preprint arXiv:2205.14083 (2022).\n[3] Liu, Yong, et al. \"Towards efficient and scalable sharpness-aware minimization.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n[4] Du, Jiawei, et al. \"Efficient sharpness-aware minimization for improved training of neural networks.\" arXiv preprint arXiv:2110.03141 (2021).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "eTWQ1tIzit", "reviewer": "Reviewer_5xxY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper targets at improving the efficiency of sharpness aware optimizer (SAM) by assigning a probability where a normal SGD optimizer is used. The decision is based on Bernoulli trial. The authors further propose a general framework to make the proposed optimizer schedule usable for different architectures. The idea is intuitive and clearly expressed. The proposed method is simple. This would not be an issue if it is extremely effective. However, the paper has missed many efficient SAM baselines such as GSAM [1] and SAF [2]. Besides, the experiments on large dataset ImageNet is not solid. The selected baseline models are different from the experiments on CIFAR-10 and the are not representative enough.\n\n\n[1] Juntang Zhuang, Boqing Gong, Liangzhe Yuan, Yin Cui, Hartwig Adam, Nicha Dvornek, Sekhar Tatikonda, James Duncan, and Ting Liu. Surrogate gap minimization improves sharpness-aware training.\n\n[2] Du, Jiawei, et al. \"Sharpness-Aware Training for Free.\" arXiv preprint arXiv:2205.14083 (2022).", "summary_of_the_review": "In summary, the proposed method is not well supported by the presented experiments mainly due to two points:\n\n1. The missing baselines on efficient SAM algorithms and \n2. The missing architectures on ImageNet dataset.", "strength_and_weaknesses": "Strength:\n1. The motivation of the proposed method is clear. SAM is indeed suffering from low computational efficiency.\n2. The proposed method is clearly presented. The reproducibility is not an issue.\n\nWeakness:\n1. The proposed algorithm is intuitive and not novel enough. It is adding a probability to replace some of the SAM steps with normal SGD optimization steps. This would not be an issue if the proposed method is verified to be super-effective. However,\n2. The experiments do not show the superiority of the proposed method. The paper has missed some relevant baselines as mentioned in the summary section. Also, I was confused that why the selected architectures are different on CIFAR and ImageNet dataset?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Ua-w5CmuInh", "reviewer": "Reviewer_cVKR", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents new training methods called RST and G-RST to extend the geometry inspired training method SAM and improve its computational efficiency.\nBased on randomized gradient boosting RST randomly selects between SGD and SAM with probability based on parameterized Bernoulli distribution.\nThe authors explore different parameterization schemes for such a scheduling function and analyze their effect on computations and performance trade-off.\nThe authors also develop RST’s convergence properties for non-convex stochastic cases where the classes of objective functions are smooth and strongly convex.\nThe paper evaluates RST and G-RST on multiple image classification tasks showing that the proposed methods can save 50% computations while performing on-par or even better than the original SAM.\n", "summary_of_the_review": "Highly recommended for interested readers on SAM literature.", "strength_and_weaknesses": "- The paper is written in a very clear and professional way.\n- The paper is very solid with balanced views and analysis.\n- The paper provides convergence analysis which somehow lacks in SAM literature.\n- The resulting algorithm is simple and effective.\n- I literally didn’t find any flaws in the paper but thought adding more experiments on large scale and different domains could make the paper even stronger.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QHItPdGRQU5", "reviewer": "Reviewer_hEbB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Sharpness-aware minimisation (SAM) has been shown powerful to train high-performance deep learning models, but it also incurs at least double computational cost due to the extra back-propagation for the sharpness estimation. To improve the efficiency of the vanilla SAM, this paper proposes an training scheme, dubbed Randomized Sharpness-Aware Training (RST). RST performs a Bernoulli trial at each iteration to choose randomly from base algorithms (SGD) and SAM. The probabilities of Bernoulli trials at each time are  determined by a predefined scheduling function $p(t)$. The average extra time is reduced from 1 to $\\sum p(t) / T$.", "summary_of_the_review": "The paper is easy to follow. The empirical experiments are enough to support the arguments. However, the novelty of this paper is the concern. The idea of selecting a subset of parameters or iterations for implementing SAM has been explored in several recent works.", "strength_and_weaknesses": "Strength:\n- The proposed RST can reduce the extra computational cost of the vanilla SAM from 1 to $\\sum p(t) / T$ in average, and preserves similarly good performances.\n- The paper is easy to follow.\n\nWeakness:\n- The novelty of this paper is my only concern. The idea of applying SAM to a subset of parameters or iterations has been explored in several papers [Mi et al. 2022, Liu et al. 2022, Du et al. 2022]. For example, Liu et al. 2022 propose to only periodically calculate the inner gradient ascent across the training iterations; Liu et al. 2022 and Du et al. 2022 propose to select a subset of parameters to calculate the inner gradients in each iterations. Over expectations, randomly selecting a subset of parameters in iterations is same as randomly select interactions. And the later can be considered as a special case of the former --- alternatively masking out all parameters. \n\n\nMi et al. 2022, Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approach\n\nDu et al. 2022, Efficient Sharpness-aware Minimization for Improved Training of Neural Networks\n\nLiu et al. 2022, Towards Efficient and Scalable Sharpness-Aware Minimization", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject