An Adaptive Policy to Employ Sharpness-Aware Minimization
Authors: Weisen Jiang
Hansi Yang
Yu Zhang
James Kwok
Venue: ICLR 2023 poster (2023)
Keywords: Sharpness-aware minimization
model generalization
loss landscape

Abstract: Sharpness-aware minimization (SAM), which searches for flat minima by min-max optimization, has been shown to be useful in improving model generalization. However, since each SAM update requires computing two gradients, its computational cost and training time are both doubled compared to standard empirical risk minimization (ERM). Recent state-of-the-arts reduce the fraction of SAM updates and thus accelerate SAM by switching between SAM and ERM updates randomly or periodically. In this paper, we design an adaptive policy to employ SAM based on the loss landscape geometry. Two efficient algorithms, AE-SAM and AE-LookSAM, are proposed. We theoretically show that AE-SAM has the same convergence rate as SAM. Experimental results on various datasets and architectures demonstrate the efficiency and effectiveness of the adaptive policy.

Published as a conference paper at ICLR 2023
AN ADAPTIVE POLICY TO EMPLOY SHARPNESS-AWAR MINIMIZATION
Weisen J Hansi Yang', Yu Zhang!,3, James Kwok²
I Guangdong Provincial Key Laboratory of Brain-inspired Intelligent Computation
Department Computer Science and Engineering, Southern University of Science and Technology
2 Department of Computer Science and Engineering, Hong Kong University of Science and Technology
3 Peng Cheng Laboratory
(w: iangar, hyangbw, (osse. ust. hk, yu. za.zhangg.. wushanggsstteeeeeee wut.tusttaaaam.... comm
ABSTRACT
Sharpness-aware minimization (SAM), which searches for flat minima by min-max optimization, has been shown to be useful in improving model generalization. However, since each SAM update requires computing two gradients, its compu- tational cost and training time are both doubled compared to standard empirical risk minimization (ERM). Recent sate-oofthhe-rrt reduce the fraction of SAM updates and thus accelerate SAM by switching between SAM and ERM updates randomly or periodically. In this paper, we design an adaptive policy to employ SAM based on the loss landscape geometry. Two efficient algorithms, AE-SAM and A.,, are proposed. We theoretically show that AE-SAM has the same convergence rate as SAM. Experimental results on various datasets and architectures demonstrate the efficiency and effectiveness of the adaptive policy.
INTRODUCTION
Despite great success in many applications (He et al. 2016 Zagoruyko & Komodakis 2016 Han jt al. 2017), deep networks are often werrparameterriee and capable of memorizing all training data. The training loss landscape is complex and nonconvex with many local minima of different generalization abilities. Many studies have investigated the relationship between the loss surface's geometry and generalization performance (Hochreiter & Schmidhuber 1994; McAllester 1999
Keskar et al. 2017 Neyshabeyhabuu et al.) 2017]Jiang et al. (020), and found that flatter minima generalize
better than sharper minima (Dziugaite & Roy 2017 Petzka al. 2021 Chaudhari et al. 2017
Keskar et al. 2017 Jiang et 2020).
Sharpness-aware minimization (SAM) (Foret al. 2021 is the current offte-of-thh-arr seek flat minima by solving a min-max optimization problem. In the SAM algorithm, each update consists of Iwo warrwdd computations: one for computing the perturbation and the other for computing the actual update direction. Since these two computations are not parallelizable, SAM doubles the computational overhead as well as the training time compared to empirical risk minimization (ERM).
Several algorithms (Du et al. 2022al Zhao et al.] 2022b; Liu et al. 2022) have been proposed to improve the efficiency of SAM. ESAM (Du al. 2022a) uses fewer samples to compute the gradients and updates fewer parameters, but each update still requires two gradient computations. Thus, ESAM does not alleviate the bottleneck of training speed. Instead of using the SAM update at every iteration, recent state-of-the-arts (Zhao et al. 2022b Liu et al. 2022) proposed to use SAM randomly or periodically. Specifically, SS-SAM (Zhao al. 2022b) selects SAM or ERM according to Bernoulli trial, while LookSAM (Liu et al.) 2022) employs SAM every le step. Though more efficient, the random or periodic use of SAM is suboptimal as it is not geometry-aware. Intuitively, the SAM update is more useful in sharp regions than in flat regions.
In this paper, we propose an adaptive policy to employ SAM based on the geometry of the loss landscape. The SAM update is used when the model is in sharp regions, while the ERM update used in flat regions for reducing the fraction of SAM updates. To measure sharpness, we use
"oorresonndence Yu Zhang

---

Published as a conference paper at ICLR 2023
the squared stochastic gradient norm and model it by normal distribution, whose parameters estimated by exponential moving average. Experimental results on standard benchmark datasets demonstrate the superiority of the proposed policy.
Our contributions are summarized as follows: (i) We propose an adaptive policy to use SAM or ERM update based the loss landscape geometry. (ii) We propose efficient algorithm, called AE-SAM (Adaptive policy to Employ SAM), to reduce the fraction of SAM updates. We also theoretically study its convergence rate. (iii) The proposed policy is general and can be combined with any SAM variant. In this paper, we integrate it with LMokSAM (Liu et al. 2022) and propose AE-LookSAM. (iv) Experimental results on various network architectures and datasets (with and without label noise) verify the superiority of AE-SAM and AE-LooKSAM over existing baselines.
Notations.
Vectors (c.g., x) and matrices (e.g., X) are denoted by lowercase and uppercase boldface letters, respectively. For a vector its fo-n is ||x]| AV(; d²) is the univariate normal distribution with mean μı and variance o². diag(x) constructs a diagonal matrix with x on the diagonal. Moreover, (z) denotes the indicator function for given set A, i.e. (I))) 1 if x E A, and 0 otherwise.
RELATED WORK
We given a training set DD with i.i.d. samples {(x,, 91) : = 1, ..... n). Let f(x; w) be model parameterized by w. Its empirical risk on D is C(D; w) = 1/j - e( [(((x;)) vc). where ((., is a loss (e.g., cross-entropy loss for classification). Model training aims to learn a model from the training data that generalizes well on the test data.
Generalization and Flat Minima. The connection between model generalization and loss landscape geometry has been theoretically and empirically studied in (Keskar et al.] [017: Dziugaite & Roy 2017|Jian et al. 2020). Recently, Jiang et al. (2020) conducted large-scale experiments and find that sharpness-based measures (flatness) are related to generalization of minimizers. Although flatness can be characterized by the Hessian's 's cigenvalues (Keskar et al.
et al. (017). handling the Hessian explicitly is computationally prohibitive. To address this issue, practical algorithms
propose to seek flat minima by injecting noise into the optimizers (Zhu et al. 2019 Zhou et al. 2019
Orvieto et al. 2022 Bisla et al. 2022), introducing regularization (Chaudhari et al.
2017 et al.
2022a
Du et al. 2022b), averaging model weights during training (lzmailov et al.
2018] He et al.
2019
Cha et al.
2021). or sharpness-awan
minimization (SAM) (Foret al. 202
K Kwon et al.
2021
Zhuang et al. 2022 Kim et al. 2022).
SAM.
The state-of-the-ard SAM (Foret al. 2021 and its variants (Kwon et al. 2021 Zhuang et al.
2022:
Kim et al. [022 Zhao et al.) 2022a) search for flat minima by solving the following min- max
optimization problem:
min max L(D: w E),
w |ellsp
where p > 0 is the radius of perturbation. The above can also be rewritten as min L(D; w) +c) L(D;
R(D; w). where RD: w) max ells,p L(D; minimizers (Foret et al. 2021). ). As solving the inner maximization in (1) exactly is computationally infeasible for nonconvex losses, SAM approximately solves it by first-order Taylor approximation, leading to the update rule:
w) is regularizer that penalizes sharp
W4+ = W. 1/V L(B, W. + P: VLB. w:));
where B, is a mini-batch of data, is the step size, and P: TV2 Although SAM has shown to be effective in improving the generalization of deep networks, major drawback is that each update in (2) requires two waakkwdd calculations. Specifically, SAM first calculates the gradient of C(B,; w) at w. to obtain the perturbation, then calculates the gradient of L(B,; w) at w. Pt V (BB; w,) to obtain the update direction for w,. As a result, SAM doubles the computational overhead compared to ERM.
Efficient Variants of SAM. Several algorithms have been proposed to accelerate the SAM algorithm. ESAM (Du et al. 2022a) uses fewer samples to compute the gradients and only updates part of the model in the second step. but still requires to compute most of the gradients. Another direction is to reduce the number of SAM updates during training. SS-SAM (Zhao et al. 2022b) randomly selects
2

---

Published as a conference paper at ICLR 2023
ERM AE-SAM
25
Variance of gradlent
AE-SAM
Variance of gradlents
AE-SAM
varésitc of gradienls
30
25
15
20
15
16
13
10
5
5
200
200
50
150
250
epoch
spoch
epoch
(a) ResNer-18.
(b) WRN-28-10.
(c) PyramidNer-
Figure 1: Variance of gradient CIFAR-100. Best viewed in color.
SAM ERM update according to Bernoulli trial, while LookSAM (Liu al. 2022 employs SAM at every & iterations. Intuitively, the SAM update is more suitable for sharp regions than flat regions. However, the mixing policies in SS-SAM and LookSAM not adaptive to the loss landscape. In this paper, we design an adaptive policy to employ SAM based on the loss landscape geometry.
METHOD
In this section, we propose an adaptive policy to employ SAM. The idea is to use ERM when w, is in flat region, and use SAM only when the loss landscape locally sharp. We start by introducing sharpness measure (Section 3.1. then propose an adaptive policy based on this (Section 3.2p. Next, we propose two algorithms (AL-SAM and AB-LookSAM) and study the convergence.
3.1 SHARPNESS MEASURE
Though sharpness can be characterized by Hessian's eigenvalues (Keskar et al., 2017: Dinh et al. 2017). they are expensive compute. A widely-used approximation is based on the gradient magnitude diag([V (B, ) (Bottou et al. 2018 Khan et al., 2018), where [v]2 denotes the elementwise square of vector v. As [[V,; w equals the trace of diag([VC((,, ))) reasonable to choose [B;; wI) as sharpness measure.
[[V w 2 is also related the gradient variance Var(VVC(B; w.)). another sharpness mea- sure (Jiang et al. 2020). Specifically,
Var L(V, wx)) Es, VL(B,; w.) V w.)
v CCB. w. )|| [V )| (3)
With appropriate smoothness assumptions on L. both SAM and ERM can shown theoretically to converge to critical points of L(D; w) (i.e., VL(D; w) 0) Reddi et al. 2016 Andriushchenko Flammarionn 2022). Thus, follows from (3) that Var( (LLVLB, wi)) Eg, VVL;; w:)ll
when w. is critical point of C(D; w). Jiang et al. (020)) conducted extensive experiments and empirically show that Var CBC, w.)) is positively correlated with the generalization gap. The smaller the Var LBB w:)). the better generalization is the model with parameter w.. This finding also explains why SAM generalizes better than ERM. Figure [ shows the gradient variance w.r.t. the number of epochs using SAM and ERM on CIFAR-100 with various network architectures (experimental details are in Section 4.1). As can be seen, SAM always has a much smaller variance than ERM. Figure 2 shows the expected squared norm of the stochastic gradient w.r.t. the number of epochs on CIFAR 700. As shown, SAM achieves a much smaller Eg, (LB;; w.)]l than ERM.
3.2
ADAPTIVE POLICY TO EMPLOY SAM
As | VVC II? changes with (Figure 2). the sharpness at w, also changes along the optimization trajectory. As result, we need to estimate Es, VLB, wz) at every iteration. One can sample a large number of mini-batches and compute the mean of the stochastic gradient norms. However, this can be computationally expensive. To address this problem, with simple distribution and estimate the distribution parameters in an online manner. Figure ((a) shows ((B,; w 2 of 400 mini-batches at different training stages (epoch 60. 120, and 180)
model LLB, w. )|
3

---

Published as a conference paper at ICLR 2023
140
SAM AE-SAM
SAM
SAM
AEE SAM
V will
Fst [V, woll
100
(R, w)ll2
80
40
10
50
100 epoch
150
200
50
100
150
200
50
100
150
200
250
300
epoch
epoch
(a)
RexNer-18.
(b) WRN-28-10.
(c) SyramidNet-110
Figure 2: Squared stochastic gradient norms Eg VD w. wllll on CIFAR-I00. Best viewed in color.
CIFAR-/O0 using ResNer-18) As be seen, the distribution follows Bell curve. Figure (b) shows the corresponding quantle-quantile (Q-Q) plot (Wilk & Gnanadesikan 1968). The closer the curve to line, the distribution is closer to the normal distribution. Figure suggests that | LBB; w can be modeled with a normal distribution (W,,, (H),))) We use exponential moving average (EMA), which popularly used in adaptive gradient methods (e.g., RMSProp (Tieleman & Hinton 2012). AdaDelta Zeiler 2012), Adam [Kingma & Ba 2015)), to estimate its mean and
variance:
14 8p4 (1- (1-0))/V[(C () w:)ll
of 80 + (1 ()B we)ll 11)
(5)
where d E (0. 1) controls the forgetting rate. Empirically, we use ô = 0.9. Since VL(B, w.) is already available during training, this EMA update does not involve additional gradient calculations (the cost for the norm operator negligible).
0.04
epoch
50
epoch-180
epoch=120
epoch
sample le quantiles
epoch-60
density
30
10
0.01
0.00
10
40
-2
0
2
[V2(6% w:]l?
theoretical quantiles
(a) Distributions.
(b) Q-Q plots.
Figure 3: Stochastic gradient norms [IIC(B,; w.)]l B, ~ D} of ResNet-18 on CIFAR-100 are approximately normally distributed. Best viewed in color.
Using 14 and of. we employ SAM only at iterations where [/V;; w is relatively large (i.e., the loss landscape is locally sharp). Specifically, when VV, w2 2 ft + CjØ; (where e is threshold), SAM is used; otherwise, ERM is used. When 6 + -o0, it reduces to SAM; when > oc, it becomes ERM. Note that during the early training stage, the model still underfitting and w, is far from the region of final convergence. Thus, minimizing the empirical loss is more important than seeking a locally flat region. Andriushchenko & Flammarion (2022) also empirically observe that the SAM update is more effective in boosting performance towards the end of training. We therefore design a schedule that linearly decreases e, from A2 to X (which are pre-set values):
c = 99,,., (t) A + +) Ag. where the total number of iterations. The whole procedure, called Adaptive policy to Employ SAM (AE-SAM), is shown in Algorithm [
AE AE-LookSAM. The proposed adaptive policy can be combined with any SAM variant. Here, consider integrating it with LookSAM (Liu et al. 2022). When V C(B; w:)l| 2 at + C,d,, SAM
Results on other architectures and CIFAR-/O /0 are shown in Figures and of Appendix B.I
Note that normality is needed in the theoretical analysis (Section 3.3
4

---

Published as a conference paper at ICLR 2023
is used and the update direction for w, is decomposed into two orthogonal directions in LookSAM: (i) the ERM update direction to reduce training loss, and (ii) the direction that biases the model to a flat region. When || ( w,)ll < #4 + c, ERM is performed and the second direction of the previous SAM update is reused to compose an approximate SAM direction. The procedure, called AE-LookSAM, is also shown in Algorithm
Algorithm AE-SAM and AE-LookSAM
Require: training set D, stepsize radius x1 and X2 for (t); Wo, p-1 0, o², = = and for AE-LooKSAM:
1: for 2 = .
do
2: sample mini-batch data B, from D;
3: compute
V (B; wa):
4: update ftt by 44) and of by
5:
compute Ct DA1 A2 (t);
6:
if
V (B, w 2 2 Pr + CoO, then
7:
g+ V (B,, w, + pV L(B,; w:)):
8:
if AE-LookSAM. decompose g, g. g.
1
9:
else:
10:
AE-SAM: g. g:
11:
if AE-LookSAM:: -
g
g
12:
end if
13:
W++1 = Wt - 178s;
14:
end for
15:
return Wy.
3.3 CONVERGENCE ANALYSIS
In this section, we study the convergence of any algorithm AA whose update in each iteration can be either SAM or ERM. Due to this mixing of SAM and ERM updates, analyzing convergence more challenging compared with that of SAM.
The following assumptions on smoothness and bounded variance of stochastic gradients are stan- dard in the literature on non-convex optimization (Ghadimi & Lan| 2013 Reddi et al., 2016) and SAM (Andriushchenko Flammarion 2022 Abbas et al. 2022 Qu et al. 2022).
Assumption 3.1 (Smoothness). L(D; w) is 8-smooth in w, i.e., |VL(D; w) - vV(:vv)l - fl[w v|l-
Assumption 3.2 (Bounded variance of stochastic gradients). (tt.n>/D/V(((((:x//VVV
VL(D:w)] c
Let & be indicator of whether SAM or ERM is used at iteration / (i.e., & = 1 for SAM, and 0 for ERM). For example, & = [[ ww tcra)) (w,) for proposed AE-SAM, and & sampled from Bernoulli distribution for SS SAM (Zhao al. 2022b).
Theorem 3.3. Let be the mini-batch size. If stepsize nj 49VT and p A algorithm A satisfies
323 (C(D: wo) - EC((D; wr))
(1+ (1+
min 0S1ST-1
(6)
VT (7 (7-6C)
b/ 6C)
where Ç- 7 & E 10 is the fraction of SAM updates, and the expectation is taken over the
random training samples.
All proofs are in Appendix A Note that a larger leads to larger upper bound 6). When ( = 1, the above reduces SAM (Corollary A.2 of Appendix A.T..

---

Published as a conference paper at ICLR 2023
4 EXPERIMENTS
In this section, we evaluate the proposed AE-SAM and AE-LookSAM on several standard benchmarks. As the SAM update doubles the computational overhead compared to the ERM update, the training speed is mainly determined by how often the SAM update is used. Hence, we evaluate efficiency by measuring the fraction of SAM updates used: %SAM 100 #(iterations using SAM)/T /T. The total number of iterations, T. is the same for all methods.
4.1 CIFAR-1O AND CIFAR-100
Setup. In this section, experiments performed on the CIFAR-1O and CIFAR-100
datasets (Krizhevsky& & Hinton 2009) using four network architectures: ResNet-18 (He et al. 2016)
WideResNer-28- 10 (denoted WRN-28-10) (Zgoruyko & Komodakis 2016), PyramidNer- 110 (Han et al. 2017), and ViT-S16 (Dosovitskiy et al. 2021).
Following the setup in (Liu et al. 2022 Foret al. 2021 Zhao al. 2022a we use batch size 128,
initial learning rate of 0.1, cosine learning rate schedule, SGD optimizer with momentum 0.9 and weight decay 0.0001. The number of training epochs is 300 for PyramidNer- 100 1200 for VIT-S16, and 200 for ResNet-18 /8 and WideRexNer-28- As in Foret al. (2021), ) we perform grid search for the radius p over {0.01, 0.02, 0.05, 0.1, 0.2, using the validation set. Similarly, is selected by grid search over {0.1, 0.3, 0.6, 0.9}. For the cr schedule A2 (t). A1 = -1 and A2 = 1 for AE-SAM; λ = 0 and X2 = for AE-LooKSAM.
10% of the training set is used as the validation set.
Baselines. The proposed AE-SAM and AE-LooKSAM are compared with the following baselines: (i) ERM; (ii) SAM (Foret et al. 202T); and its more efficient variants including (iii) ESAM (Du et al. 2022a) which uses part of the weights compute the perturbation and part of the samples to compute the SAM update direction. These two techniques can reduce the computational cost, but may
not always accelerate SAM, particularly in parallel training (Li et al. 2020); (iv) SS-SAM (Zhao et al. 20226), which randomly selects SAM or ERM according to a Bernoulli trial with success probability 0.5. This is the scheme with the best performance in (Zhao et al.) 2022b); (v) Look- SAM (Liu et al. 2022) which uses SAM at every k = 5 steps. The experiment is repeated five times with different random seeds.
Results. Table D shows the testing accuracy and fraction of SAM updates (%SAM). Methods are grouped based on %SAM. As can be seen, AE-SAM has higher accuracy than SAM while using only 50% of SAM updates. SS-SAM and AE-SAM have comparable %SAM (about 50%), and AE-SAM achieves higher accuracy than SS-SAM (which statistically significant based on the pairwise t-test at 95% significance level). Finally, LookSAM and HE-LooKSAM have comparable %SAM (about 20%), and AE-LookSAM also has higher accuracy than LookSAM. These improvements confirm that the adaptive policy is better.
4.2 ImageNet
Setup. In this section, we perform experiments on the ImageNer (Russakovsky al. 2015). which contains 1000 classes and 1.28 million images. The RexNet-50 (He et al. (016)) is used. Following the setup in Du al. (2022a). ) we train the network for 90 epochs using SGD optimizer with momentum 0.9, weight decay 0.0001, initial learning rate 0.1, cosine learning rate schedule, and batch size 512. As Fooret al. 202 Du al. 2022a). P = 0.05. For the c schedule 9A (t). Å = -1 and Aa2 = for SE-SAM; X = U and A2 = 2 for AE-LooKSAM. k = 5 is used for LookSAM. Experiments are repeated with three different random seeds.
Results. Table shows the testing accuracy and fraction of SAM updates. As can be seen, with only half of the iterations using SAM, AE-SAM achieves comparable performance as SAM. Compared with LookSAM, AE-LookSAM has better performance (which is also statistically significant), verifying the proposed adaptive policy is more effective than LookSAM's periodic policy.
4.3 ROBUSTNESS TO LABEL NOISE
Setup. In this section, we study whether the more-efficiem SAM variants will affect its robustness to training label noise. Following the setup in Foret al. (2021) we conduct experiments corrupted

---

Published as a conference paper at ICLR 2023
Table 1: Means and standard deviations of testing accuracy and fraction of SAM updates (%SAM) on CIFAR-10 and CIFAR-I0O. Methods are grouped based on &SAM. The highest accuracy in each group underlined; while the highest accuracy for each network architecture (across all groups) is in bold.
CIFAR-10
CIFAR-100
Accuracy
%SAM
Accuracy
%SAM
ERM
95.41 10.03
0.0 10.0
78.17 10.05
0.0 10.0
ResNet-18
SAM
Foret et al.
2021
96.52 10.12
100.0 10.0
80.17 10.15
100.0 10.0
ESAM
Du al.
2022a
96.56 +0.08
100.0 =0.0
80.41 #0.10
100.0 #0.0
SS-SAM
Zhao et AE-SAM
2022b
96.40 10016
50.0 10.0
80.10 10.16
50.0 10.0
96.63 1.004
50.1 10.1
80.48 10.11
49.8 10.0
LookSAM (Lu al.
2022
96.32 10.12
20.0 #0.0
79.89 +0.29
20.0 =0.0
AE-LooKSAM
96.56 #0.21
20.0 #0.1
80.29 #0.37
20.0 =0.0
ERM
96.34 10.12
0.0 +0.0
81.56 +0.14
0.0 #0.0
WRN-28-11 10
SAM
Foret et al.
2021
97.27 #0.11
100.0 =0.0
83.42 +0.05
100.0 #0.0
ESAM
Du et al.
2022a
97.29 10.11
100.0 10.0
84.51 +0.02
100.0 10.0
SS-SAM
Zhao et al.
2022b
97.09 1:011
50.0 10.00
82.89 10.02
50.0 =0.0
AE-SAM
97.30 #0.10
49.5 #0.1
84.51 #0.11
49.6 =0.0
LookSAM (Liu et al. AE-LooKSAM
2022
97.02 1.012
20.0 10.0
83.70 1..12
20.0 10.0
97.15 1.008
20.0 11.0
83.92 10.07
20.2 10.0
ERM
96.62 4:0.1
0.0 10.0
81.89 10.15
0.0 10.0
PyramidNet-II
SAM
Foret et al.
2021
97.30 10.10
100.0 =0.0
84.46 =0.05
100.0 #0.0
ESAM
Du al.
2022a
97.81 10.01
100.0 +0.0
85.56 +0.05
100.0 =0.0
SS-SAM Zhao
2022b
97.22 10.10
50.0 +0.0
84.90 +0.05
50.0 =0.0
AE-SAM
97.90 #0.05
50.2 +0.1
85.58 #0.10
49.8 =0.1
LookSAM (Liu et al.
2022
97.10 10.11
20.0 10.0
84.01 10.06
20.0 =0.0
AE-LooKSAM
97.22 #0.11
20.3 #0.0
84.80 #0.13
20.2 =0.1
ERM
86.69 #0.11
0.0 #0.0
62.42 #0.22
0.0 #0.0
SAM
Foret et al. 202.
87.37 10.09
100.0 10.0
63.23 10.25
100.0 10.0
ViT-S16
ESAM
Du et al.
2022a
84.27 4:0.11
100.0 10.0
62.11 10.15
100.0 10.0
SS-SAM
Zhao et al.
2022b
87.38
#0.14
50.0 +0.0
63.18 10.19
50.0 =0.0
AL-SAM
87.77 10.13
49.7 +0.1
63.68 +0.23
49.5 =0.2
LookSAM Liu
2022
87.12 +0.20
20.0 :0.0
63.52 10.19
20.0 =0.0
AE-LoOKSAM
87.32 +0.11
20.2 #0.2
64.16
#0.23
20.3 =0.2
version of CIFAR-10, with some of its training labels randomly flipped (while its testing set is kept clean). The ResNer-18 and RexNer-32 networks are used. They are trained for 200 epochs using SGD with momentum 0.9, weight decay 0.0001, batch size 128, initial learning rate 0.1, and cosine learning rate schedule. For LookSAM, the SAM update is used every h = 2 steps 3 For AE-SAM and - and A2 in their o schedules gx1 As (t), such that their fractions of SAM updates (approximately 50%) are comparable with SS-SAM and LookS Experiments repeated with five different random seeds.
AE-LooKSAM, we
Results. Table 3 shows the testing accuracy and fraction of SAM updates. As can be seen, AE- LookSAM achieves comparable performance with SAM but faster as only half of the iterations use the SAM update. Compared with ESAM, SS-SAM, and LookSAM, AE-LookSAM performs better. The improvement is particularly noticeable at the higher noise levels (e.g., 80%).
'hhe performance of LookSAM can he sensitive the value of kt Table of of Appendix B.2 shows that using
k 2 leads to the best performance in this experiment.
7

---

Published as a conference paper at ICLR 2023
Table 2: Means and standard deviations of testing accuracy and fraction of SAM updates (%SAM) on ImageNer using ResNer-50. Methods are grouped based on %SAM. The highest accuracy each group is underlined; while the highest across all groups in bold.
Accuracy
%SAM
ERM
77.11 =0.14
0.0 #0.0
SAM
Foret et al.
2021
77.47
10.12
100.0 10.0
ESAM
Du et al.
2022a
77.25
20.75
100.0 10.0
SS-SAM
Zhao et al.
2022b
77.38
20.06
50.0 1:.0
SEM
77.43 #0.06
49.4 40.0
LookSAM [Luu et al. AE LooKKoM
2022
77.13
#0.09
20.0 #0.0
29 =0.08
20.3 #0.0
Table Testing accuracy and fraction SAM updates on CIFAR-IO with different levels of label noise. The best accuracy is in bold and the second best is underlined.
noise
20%
noise 40%
60%
80%
accuracy &SAM
accuracy &SAM
accuracy
&SAM
accuracy
%SAM
ERM
87.92
0.0
70.82
0.0
49.61
0.0
28.23
SAM
Foret et
2021
94.80
100.0
91.50
100.0
88.15
100.0
77.40
ResNer-11
ESAM
et
94.19
100.0
91.46
100.0
81.30
100.0
15.00
100.0
SS-SAM
Zhao al. 2022b
90.62
50.0
77.84
50.0
61.18
50.0
47.32
50.0
LookSAM
2022
92.72
50.0
88.04
50.0
72.26
50.0
69.72
50.0
AE-SAM
92.84
50.0
84.17
50.0
73.54
49.9
65.00
50.0
AE-LookSAM
94.34
49.9
91.58
50.0
87.85
50.0
76.90
50.0
ERM
87.43
70.82
46.26
29.00
SAM
Foret
2021
95.08
100.0
91.01
100.0
88.90
100.0
77.32
100.0
ResNeet-32
ESAM
Du
2022a
93.42
100.0
91.63
100.0
82.73
100.0
10.09
100.0
SS-SAM
Zhao al. 2022b
89.63
50.0
74.17
50.0
58.40
50.0
59.53
50.0
LookSAM
Liu
2022
92.49
50.0
86.56
50.0
63.35
50.0
68.01
50.0
AF-SAM
92.87
50.0
82.85
50.0
71.50
50.0
65.43
50.3
AE-LookSAM
94.70
50.0
91.80
50.0
88.22
50.0
77.03
49.8
80
SAM
70
SAM
ESAM
ESAM
training Rccuracy
LockSAM
50
LoOKSAM
AE-SAM
AE-SAM
AE-LooKSAM
80
AE-LookSAM
30
10
10
50
100
150
200
50
100
150
200
spach
epach
Training accuracy.
Testing accuracy.
Figure 4: Accuracies with number of training epochs on CIFAR-10 (with 80% noise labels) using R8sNet-18, Best viewed in color.
Figure 4 shows the training and testing accuracies with number of epochs at noise level of 80% using ResNet-184) As can be seen, SAM is robust to the label noise, while ERM and SS-SAM heavily suffer from overfitting, AE-SAM and LookSAM can alleviate the overfitting problem to a certain extent. AE- AE-LooKSAM. by combining the adaptive policy with LookSAM, achieves the same high level of robustness as SAM.
*Results for other noise levels and RexNet-32 shown in Figures 10 and
Appendix
B.3 respectively.
8

---

Published as a conference paper at ICLR 2023
100
100
87.0
80
4==00
0
Ag =
tossing accurncy
56.5
beeting Mncoure
NSMM
NSSM
A2 -
40
96,0
-8-
Ap A2
Ap -1 -
0
20
20
A2
86.5
78
-2
A2
A1
(a) CIFAR-10.
(b) CIFAR-100.
(a) CIFAR-1O.
(b) CIFAR-100.
Figure 5: Effects of Y and 22 on fraction of SAM updates using R8sNet-18. Best viewed in color.
Figure 6: Effects of X and X2 testing accuracy using ResNet-18. 18. Best viewed in color.
4.4 EFFECTS OF Å AND X2
In this experiment, study the effects of Yr and A2 on AE-SAM. We use the setup in Section 4.1, where λ and A2 (with A < A2) are chosen from {0, =1, +2}. Results on AE-LookSAM using the label noise setup in Section 4.3 are shown in Appendix B.4
Figure 3 shows the effect on the fraction of SAM updates. For a fixed X2, increasing A increases the threshold ce. and the condition | CVB,; w 2 je + Cg becomes more difficult satisfy. Thus, as can be seen, the fraction of SAM updates is reduced. The same applies when Aa increases. similar trend also observed on the testing accuracy (Figure 6
4.5 CONVERGENCE
In this experiment, we study whether w,'s (where / is the number of epochs) obtained from AE-SAM can reach critical points of L(D; w) as suggested Theorem 3.3 Figure 7 shows LVV w:)ll w.r.t. for the experiment in Section 4.1 As can be seen, in all settings, V CDD; wz)ll converges
to O. In Appendix B.5 we also verily the convergence of AE-SAM's training loss on CIFAR-
and CIFAR-1O0 (Figure 14), and that AE-SAM and SS-SAM have comparable convergence speeds
(Figure 15 which agrees with Theorem 3.3 as both have comparable fractions of SAM updates (Table
60
50
CIFAR-10
CIFAR-10 10
CIFAR-10
50
CIFAR-100 100
100
40
CIF CIFAR-100
1Ve wall
[0c|D) wolp
30
30
60
10
20
10
epoch
epoch
epoch
(a) R8.
(b) WRN-28-10.
(c) PyramidNet-110.
Figure 7: Squared gradient norms of AE-SAM with number of epochs. Best viewed in color.
5 CONCLUSION
In this paper, we proposed an adaptive policy to employ SAM based on the loss landscape geometry, Using the policy, proposed an efficient algorithm (called AE-SAM) to reduce the fraction of SAM updates during training. We theoretically and empirically analyzed the convergence of AE-SAM. Experimental results number of datasets and network architectures verify the efficiency and effectiveness of the adaptive policy. Moreover, the proposed policy is general and can be combined with other SAM variants, as demonstrated by the success of AF-LookSAM.
9

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work was supported by NSFC key grant 62136005, NSFC general grant 620761 and Shenzhen fundamental research program JCY20021022105000000 This research was supported in part by the Research Grants Council of the Hong Kong Special Administrative Region (Grant 16200021).
REFERENCES
Momin Abbas, Quan Xiao, Lisha Chen, Pin-Yu Chen, and Tianyi Chen. Sharp-MAML: Sharpness- aware model-agnostic meta learning. In International Conference Machine Learning, 2022.
Maksym Andriushchenko and Nicolas Flammarion. Towards understanding sharpness-aware mini- mization. In International Conference on Machine Learning, 2022.
Devansh Bisla, Jing Wang, and Anna Choromanska. Low-pass filtering SGD for recovering flat optima in the deep learning optimization landscape. In International Conference on Artificial Intelligence and Statistics, 2022.
Léon Bottou, Frank Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. SIAM Review, 2018.
Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. SWAD: Domain generalization by seeking flat minima. In Neural Information Processing Systems, 2021.
Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-SGD: Biasing gradient descent into wide valleys. In International Conference on Learning Representations, 2017.
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In International Conference on Machine Learning, 2017.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Representations, 2021.
Jiawei Du, Hanshu Yan, Jiashi Feng, Joey Tianyi Zhou, Liangli Zhen, Rick Siow Mong Goh, and Vincent Tan. Efficient sharpness-aware minimization for improved training neural networks. In International Conference Learning Representations, 2022a.
Jiawei Du, Daquan Zhou, Jiashi Feng. Vincent YF Tan, and Joey Tianyi Zhou. Sharpness-aware training for free. In Neural Information Processing Systems, 2022b.
Gintare Karolina Dziugaite and Daniel M. Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. In Uncertainty in Arlificial Intelligence, 2017.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations, 2021.
Saeed Ghadimi and Guanghui Lan. Stochastic first- and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization, 2013.
Dongyoon Han, Jiwhan Kim, and Junmo Kim. Deep pyramidal residual networks. In IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Haowei He, Gao Huang, and Yang Yuan. Asymmetric valleys: Beyond sharp and flat local minima. In Neural Information Processing Systems, 2019.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEFE Conference Computer Vision and Pattern Recognition, 2016.
10

---

Published as a conference paper at ICLR 2023
Sepp Hochreiter and Jürgen Schmidhuber. Simplifying neural nets by discovering flat minima. In Neural Information Processing Systems, 1994.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. In Uncertainty in Artificial Intelligence, 2018.
Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantas- tic generalization measures and where to find them. In International Conference on Learning Representations, 2020.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representations, 2017.
Mohammad Khan, Didrik Nielsen, Voot Tangkaratt, Wu Lin, Yarin Gal, and Akash Srivastava. Fast and scalable Bayesian deep learning by weight-perturbation in Adam. In International Conference on Machine Learning, 2018
Minyoung Kim, Da Li, Shell X Hu, and Timothy Hospedales. Fisher SAM: Information geometry and sharpness aware minimisation. In International Conference on Machine Learning, 2022.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, 2009.
Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi. ASAM: Adaptive sharpness aware minimization for ecale-invarian learning of deep neural networks. In International Confer- ence on Machine Learning, 2021.
Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, and Pritam Damania. PyTorch distributed: experiences on accelerating data parallel training. In Proceedings of the VLDB Endowment, 2020.
Yong Liu, Siqi Mai, Xiangning Chen, Cho-Jui Hsieh, and Yang You. Towards efficient and scalable sharpness-aware minimization. In IEEE Conference on Computer Vision and Pattern Recognition, 2022.
David McAllester. PAC-Bayesian model averaging. In Annual Conference on Computational Learning Theory, 1999.
Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring generaliza- tion in deep learning. In Neural Information Processing Systems, 2017.
Antonio Orvieto, Hans Kersting, Frank Proske, Francis Bach, and Aurelien Lucchi. Anticorrelated noise injection for improved generalization. In International Conference on Machine Learning. 2022.
Henning Petzka, Michael Kamp, Linara Adilova, Cristian Sminchiseseu, and Mario Boley. Relative flatness and generalization. In Neural Information Processing Systems, 2021.
Zhe Qu. Xingyu Li, Rui Duan, Yao Liu, Bo Tang, and Zhuo Lu. Generalized federated learning via sharpness aware minimization. In International Conference on Machine Learning, 2022.
Sashank Reddi, Ahmed Hefny, Suvrit Sra, Barnabas Poczos, and Alex Smola. Stochastic variance reduction for nonconvex optimization. In International Conference on Machine Learning, 2016.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 2015.
Tijmen Tieleman and Geoffrey Hinton. RMSProp: Neural networks for machine learning. Lecture 6.5, 2012.
11

---

Published as a conference paper at ICLR 2023
Martin B Wilk and Ram Gnanadesikan. Probability plotting methods for the analysis for the analysis of data. Biometrika, 1968.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference, 2016.
Matthew D Zeiler. AdaDelta: an adaptive learning rate method. Preprint 1212::1111. 2012.
Yang Zhao, Hao Zhang, and Xiuyuan Hu. Penalizing gradient norm for efficiently improving generalization in deep learning. In International Conference on Machine Learning. 2022a.
Yang Zhao, Hao Zhang, and Xiuyuan Hu. SS-SAM: Stochastic scheduled sharpness-aware minimiza- tion for efficiently training deep neural networks. Preprint arXiv:2203.09962. 2022b.
Mo Zhou, Tianyi Liu, Yan Li, Dachao Lin, Enlu Zhou, and Tuo Zhao. Toward understanding the importance of noise in training neural networks. In International Conference on Machine Learning, 2019.
Zhanxing Zhu, Jingfeng Wu, Bing Yu, Lei Wu, and Jinwen Ma. The anisotropic noise stochastic gradient descent: Its behavior of escaping from sharp minima and regularization effects. In International Conference on Machine Learning, 2019.
Juntang Zhuang, Boqing Gong, Liangzhe Yuan, Yin Cui, Hartwig Adam, Nicha C Dvornek, James S. Duncan, and Ting Liu. Surrogate gap minimization improves sharpness-awan training. In International Conference Learning Representations, 2022.
A PROOFS
A.1 PROOF OF THROREM 33
Theorem 3,3
Let b be the mini-batch size. If 43vT and p 1/T²,, algorithm A satisfies
W)) 323 (C(D; wo) EL(D; wT)) vT (7 6C)
(1+ 5) bVT (7 6C)
min 0S1ST-1
where ( = = E & E [0.
Lemma A.1 (Andriushchenko Flammarion (2022)). Under Assumptions 3.1 and 3.2 all t and p >> O, we have
EVL(B: DVL( VL(D:w) VL) (--08)
882822
n8) [V)
(8)
2b
Proof. Let g: = 1 (X..)E Ve(f(x;;w))au h, = 1 { Ve(f(x;;w, + +Pg.).
and g: VL(D wt).
By Taylor expansion and L(D; w) is 3-smooth, we have C(D;we41
<C(D; wi) + (Wi41 W:) 8 Wi+1 well
2
317²
<C(D; w.) 78. ((1 Egg. + +,,,,) +
|| - E.g + E,
2
317
=L(D; w:)- - 11 - 3. ET g: - 75. h4 +
(1-E:)]g 1| 2 & gg h. (9)
2
=0
=C(D; w.) 71 - -.) g: -7..8 h, +
((1
(10)
2
12

---

Published as a conference paper at ICLR 2023
where we have used Ee(1 Ex) 0 as & (0. E E.. and (1 - Ex) (1-)))²
to obtain (
Taking expectation w.r.t W, on both sides of (10). we have
Brf Et)
&t
BLCD: (1-111111166000000 E) Eg h.
Elggd
El
| 1l.
(11)
Claim Eg 2 Elgg 3 1| 0/' + g 11?, which follows from Assumption 3.2
Claim 2: E h. | < 2(1 + 8339 - (1 22 E g + 2Eg. h, which is derived as follows:
E B||b,|| B|[h, - g. - 2Eg. h,
-2E||h: grl1 + 2EI
g: 8r11 -E| 2 2Egg T he
22 lg 1 1 20
E 11 + 2Eg. h,
(12)
00²
20
<2p28²
+ Elg:
+
Elggr 2Eg h,
(13)
b
2(11 + 3²)
2p² 32 ) + 22g hr: b (1-2p²
(14)
where 12 follows from |bh - grll S p g and Assumption 3.2
13
follows from Claim 1.
Substituting Claims and 2 into 111. obtain
EC(D; Wo+1)
< EL(D; w:)) 11 (1- (1)))1))0 E| 2- Bg h; + BT - Et)
0
b + E E/
2
E
[2(1 + p2832 σ² (1 - 222 || 12 +2Eg h,
+ 2
(15)
=EC(D; w:) -11 (1 [- Bn(1-Et)
-
3nEa(1 - -2p233)
+
E El -nE, (1 n13) Egg h.
2
31/2(1
[t)
0²
-Bi72Ea1 + +p239)
2
b
EL(D; wi) - 11 (1
1 [- Bn(1 (+)
3n - 22?
+ & - (1-13) 1/ p3) E g
+ (1/2 - Et)
82p2
0²
-Ba + + + + (1 n13)
(16)
2
b
SE(D; wi) -11 17 (1 (1- (1+ + Bn 2p²) & Bn
Elig.11
2 2
31322
+ +n++ (1 + 2mp² 32 + 3p² - 373 p2))
(17)
2b
where (5) follows from Claims and 2, (16) follows from Lemma A.l and 1 1- 1-133 0. As 17
1
43
have Bn 2pB Z 3/2 and Bn 1/4, thus, (1 Bn 2pz) 8g >
13

---

Published as a conference paper at ICLR 2023
Summing over t on both sides of 17 and rearranging, obtain
L(D: EC(D;
min 0StST-1
Bl
B
(1 B7 - 2p/3) $4- 8n
7 1=
2
+ (1 + n2² + +3p²))
Bo²
(1 (1 -1 - - 9n
2b
2
[(D; wo) [C(D; WT)
T(n + nTς + 3)
(18)
(1 2 27)
7<
2b7 - 76
B1
2
[(D; Wo) EL(D; WT)
(1 + Kς + 4
(11 3
2b(1 16 On
?
C(D; wo) - EE[(;; WT)
(1 kς + 4)32
(19)
Th 75
81)
8byT(1 25 Bg
2
328 (C(D; wo) EL(D; wT))
(1+ + ()
(20)
(7 -6C)
byT (66)
where 7 = 1 + By- 2pB < 3/2, k 1 + p2B3, 1²83pp² p² 1/VT. and Ç = 7/2 Eto & E [0, 1]. We
thus finish the proof.
Corollary A.2. Let be the mini-batch size. If 13vT satisfies
and p 1/Th, SAM Foret et al.
2021
323 (C(D: wo) - (E(DDwT)) W1)
(2 + 5,39)0²
min
E VLL,
(21)
VT
byT
Corollary A.3. Let be the mini-batch size. If 4/
vt
and p 1/T algorithm satisfies
323(L(D; wo) EL((D; wT))
(1 + < 5,32
min E VL w.)l S 0S1ST-1
(22)
VTb(7 6C)
VTb(7 -6()
where Ç = = & E [0.
Proof. It follows from
18 that
L(D: wo) (.(
n7(1+ + Kς + )0
min 0S
g: 12
(23)
TT(1 - 25 4/))
2b(1 - 2/ 2/)
4B(C(D: wo) EL(D; WT))
(1 ( 5)2
(24)
VTb(3 -C)
8 8v - 3
32,(L(D; wo) - -EC(D; (ET(
(1 + ( + 5182)0²2
(25)
VTb(7 1 6C)
vTb(7 - 66)
A.2 CONVERGENCE OF BATCH GRADIENT DESCENT FOR AE-SAM
Theorem A4. Under Assumption 3.1
with full gradient descent, if p < 23
1 and < 7
algorithm A satisfies
L(D; w WT)
min 0-157
IVC(D, woll
(26)
Tn 1
8n -/. - 3pc
where <= += [0,
14

---

Published as a conference paper at ICLR 2023
Lemma A.5 (Lemma 7 in Andriushchenko & Flammarion (2022)) ) Let L(D: w) be a 3-smooth function. For any > O, we have
VL(D; w) T VL(D; w + PVL(D; 2 (1-13)][V((2,
(27)
Proof of Theorem A.4 Let g: = VL(D; w.) and h, = VL(D; w. + pV [D; w:)) be the update direction of ERM and SAM, respectively. By Taylor expansion and ((D; w) B-smooth, we have ((;:(+1)
8
& SD(D; wi) + + (Wi+1 wi) + 2
2
Wi+1 - - will?
<((D: - 7g. ((1 (1) + +cbba)
+
11(1 E.) + Echell
2
= w()) - 12 --n T h,t
(1 (1-6))lB:11 + +2ch,11222111111 h,
(28)
=0
w.): 17 (1 - 3m1(1 -E.)
Br ||; n5.gg h4,
||g 12
(29)
2
where we have used & Ex) = as & E {0, 10. S E,, and (1 - {,)² 1 - Ee to obtain (88
As |lb,l12 = ||h, ge
||g:|| + 2g. RigTh, h,, follows from 29 that
[(D; W(+1)
Brn(1 - E
=L(D; w.) -1
1 -
|g: 1
(lh, ||-1|:]]]] 2 1/++2g 2ggg11 h1) -1JE,g h
2
2
1 - E 3q(1 - 5
BIGE
Gn [h. gil n111 BBn) h.
<L(D; w.)
||g: 11 2
E Bm(1 1 Ee)
BnLE
p2n²p²t
<L(D; w.)
||g:11
||B:11 n111 Bn h; (30)
2
Bnn(1
Et)
BnKE
Er
-L(D; w.) 1
&t-
+ (1 (1--Bn)(1 -Bp)E:
||g. 1| (31)
2
2
2
Bn(11
Et)
BnGE
3TEEpp
-L(D; w.) 17
+
+
-BiEe - BpEc + B2nOSE)
l|ge|1
2
2
2
Bn
<C(D; - 1
-BpLc
||g.11.
(32)
2
where we have used
[h
g: 2
VL(D:: w.
PVL(D; w.))
FZ(D:w:)ll
S
8 (.: =
8 p²
g
to obtain (30 and Lemma A.5 to obtain 31
Summing over from to
on both sides of (32) and rearranging, we have
3n
Σ 7 1
3pf. || L(D: wo) L(D: wr).
(33)
2
0=
As p 23 and 17 #. follows that 1 -6- 87 - Bpp,, for a. t. Thus, (33 implies
1 2 < L(D; wo) - L(D: wT)
= L(D; wo)
[[(:; Wr)
min 0S1ST-1
(34)
ΣF 17 (1 On E13p)
Tn 1
8n
2 3pc
where
(= #
T-1 & E [0, 1] and we finish the proof.
B ADDITIONAL EXPERIMENTAL RESULTS
B.1 DISTRIBUTION OF STOCHASTIC GRADIENT NORMS
Figure 8 shows the distributions of stochastic gradient norms for ResNer-18. WRN-28-10 and PyramidNer- 110 on CIFAR-10 and CIFAR-100. As can be seen, the distribution follows a Bell curve in all settings. Figure 9 shows the Q-Q plots We can that the curves are close to the lines.
15

---

Published as a conference paper at ICLR 2023
opach 180
spach
0.24
0.03
genioo
0.03
denstty
Gonsity
0.03
0.02
0.02
0.01
0.00
0
15
20
25
0.00
10
20
30
40
50
69
0.00
10
20
30
40
(P00%] will"
PPetRic will"
(a) ResNet-18.
(b) WRN-28-10.
(c) PyramidNet-110.
0.01
spach 188
0.020
0.030
cpoch 240
spach -20
epech 150
0.03
spach 60
0.025
derdzy
dendlty
0,020
8.02
0.010
0.015
0.110
0.01
0.005
0.005
10
40
120
340
20
[0cta:
will
[Vccas
(d) R8.Net-18.
(e)
WRN-28-10
PyramidNet-110 10.
Figure 8: Distributions of stochastic gradient norms on CIFAR-10 (top) and CIFAR- 100 (bottom).
Best viewed in color.
epech=60
epoch=60
epoch=80
epech-120
50
epoch-120
epoch-160
20
sample quantiles
e8och-180
40
epoch-180
sample quantiles
epach-240
15
10
20
sample
20
5
10
0
-2
0
2
-2
0
2
-2
0
2
theoretical quantiles
theoretical quantiles
theoretical quantiles
(a) ResNet-18.
(b) WRN-28-10.
(c) PyramidNet-110.
50
epoch-180
epoch-180
epoch-240
epoch=120
epach=120
epach=160
sample quantiles
40
epoch=60
sample quantiles
100
epoch=60
sample quantiles
epoch=80
60
40
20
40
20
10
20
-2 theoretical quantiles
2
-2
0
2
-2
D
2
theoretical quantiles
theoretical quantiles
(d) ResNet-18.
(e) WRN-28-10.
(i) PyramidNet-1 10.
Figure 9: Q-Q plots of stochastic gradient norms on CIFAR-10 (top) and CIFAR-100 (bottom). Best
viewed in color.
B.2 EFFECT OF ON LOOKSAM
In this experiment, we demonstrate that LookSAM is sensitive to the choice of k. Table 4 shows the testing accuracy and fraction of SAM updates when using LookSAM noisy CIFAR-TO, with E {2, 3, 4, and the ResNet-18 18 model. As can be seen, Ao = 2 yields much better performance than k (3, 4,5, particularly at higher noise levels (e.g., 80%).
16

---

Published as a conference paper at ICLR 2023
Table 4: Effects of & in LookSAM on CIFAR-10 with different levels of label noise using ResNer--8.
noise accuracy
20%
noisc accuracy
40%
noise accuracy
60%
noisc accuracy
80%
%SAM
%SAM
%SAM
%SAM
92.72
50.0
88.04
50.0
72.26
50.0
69.72
50.0
3
89.07
33.3
75.38
33.3
63.79
33.3
53.87
33.3
89.00
25.0
74.12
25.0
58.17
25.0
52.28
25.0
88.57
20.0
73.90
20.0
56.80
20.0
51.82
20.0
B.3
MORE RESULTS ON ROBUSTNESS TO LABEL NOISE
Figure 10 rresp. shows the curves of accuracies at noise levels of 20%, 40%, 60%, and 80% with ResNet-18 (resp. ResNet-32). As can be seen, in all settings, AE-LookSAM is as robust to label noise as SAM.
800
100
138ingg 5000/00
Seating
tobling
200
100
200
spoch
spoch
apacc
epach
(a) 20% (Training).
(b) 20% (Testing).
(c) 40% (Training).
(d) 40% (Testing).
IRM
ERM
ERM
SAM
ESSAM
ESAM
5S:6AM
Masing acaniey
SAM
SSAAAM
BE-SAAM
LookSAM
CORSAM
LOOASAM
LOSESAM
AE:SAM
AT SAAT
LouiSAM
LoUGBAM
AE-LLMMAAM
10
10
103
102
150
100
150
200
150
200
opo
opoch
opocn
apoch
(e) 60% (Training).
(1) 60% (Testing).
(g) 80% (Training).
(h) 80% (Testing).
Figure 10: Accuracies with number of epochs on CIFAR-10 with 20%, 40%, 60%, and 80% noise
level using R8sNet-18. 18. Best viewed in color.
ERM
ERM SAM ESSAM S.S-AAM
SAM
ESAM
ESAM
bahino (001150)
SE-GAM
Sasdeng 10000
LookBAM
SC-SAM
SC-SAM
BAM
BAM
BAM
Lo:LooMAMM
3
LooSAM
20
23
0
102
150
250
0
92
152
200
100
0
52
100
150
200
spoch
spoch
spoch
epoch
(a) 20% (Training).
(b) 20% (Testing).
(c) 40% (Training).
(d) 40% (Testing).
ERM
SAM
SAM
EBAM
ESAM
ESAM
Im B1O0/173
E6 BM-3AM
SS BAM
LEae-AM
LookSSAN
SAM
AEE BE:5AMM
AF AE-SAM
LcoBBAM
AF- co65AMM
AF-L SE-LIAMMM
0
100
150
200
100
100
150
200
100 opoch
o0:00
090ch
opach
(e) 60% (Training).
(f) 60% (Testing).
(g) 80% (Training).
(h) 80% (Testing).
Figure 11: Accuracies with number of epochs on CIFAR-10 10 with 20%, 40%, 60%, and 80% noise
level using ResNer-32. Best viewed in color.
17

---

Published as a conference paper at ICLR 2023
B.4 EFFECTS OF X1 AND X ON AE-LOOKSAM
In this experiment, we study the effects of λ and λ2 on AE-LookSAM. Experiment is performed on CIFAR-IO with label noise (80% noisy labels), using the same setup as in Section 4.3
Figure 12 shows the effects of λ and X2 on the fraction of SAM updates. Again, as in Section 4.4 for fixed X2, increasing A always reduces the fraction of SAM updates. Figure shows the effects of λ and X2 on the testing accuracy of AE-SAM. As can be seen, the observations are similar to those in Section 4.4
-00 Ap
60
SSSAA
60
Ag
Ag -
Ag =
50
Ao =0
Au
(a) R8.Ner-18.
(b) ResNet-32.
(a) ResNet-18.
(b) RexNet-32.
Figure 12: Effects of X and X2 on fraction of SAM updates CIFAR-1O (with 80% noisy labels). Best viewed in color.
Figure 13: Effects of a and X on testing accu-
racy of CIFAR-10 (with 80% noisy labels). Note that the curves for X2 E {-2.-1} overlap com- pletely with that of X2 1. Best viewed in color.
B.5 ADDITIONAL CONVERGENCE RESULTS ON CIFAR-10 AND CIFAR-100
Figure shows convergence of AE AEM's'ss training loss the CIFAR- 10 and CIFAR-100 datasets. As can be seen, AE-SAM achieves convergence with various network architectures.
Figure 13 shows the training losses w.r.t. the number of epochs for AE-SAM and SS-SAM. As can be seen, AE-SAM and SS-SAM converge with comparable speeds, which agrees with Theorem 3.3 as both of them have comparable fractions of SAM updates (Table 1
4.0
4,0
4.0
CIFAR-10
CIFAR-10 CIFAR-100
CIFAR-16
CIFARc100
CIFAR-1C0
3.0
3.0
3.0
2.0
2.0
2.0
1.5
1.0
1.0
1.0
200
0.0
200
0.0
200
apoch
opach
opech
(a) ResNet-18.
(b) WRN-28-10. 10.
(c) Pyramiddet-- 10.
Figure 14: Training loss of AE-SAM with number epochs on CIFAR-10 and CIFAR-100. Best viewed in color.
AE
1.2
AE
AE
55-SAM
1.0
1.0
<(D:ww
600
0.6
0.4
0.2
0.00
50
100
1.5
200
0.0
0.0
100
150
epach
epoch
eposh
(a) R88Net-18.
(b) WRN-28-10.
(c) PyramidNet--
Figure 15: Training losses of AE-SAM and SS-SAM with number of epochs on CIFAR-10. Note that the two curves almost completely overlap. Best viewed in color.
18


Reviews:
[{"review_id": "xX7I1gH7Np", "reviewer": "Reviewer_4reb", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed an adapt scheme for sharpness-aware minimization (SAM), in which the algorithm simply perform ERM when landscape is flat and perform SAM which landscape is sharp. This method is based on couple of approximation strategies. The author also establish the theoretical guarantee for their proposed algorithm. Some empirical results are provided to verify the effectiveness of the proposed algorithm.", "summary_of_the_review": "Overall this paper is well-written and easy to follow. The high level idea of the algorithm proposed in this paper is intuitive and nature and should be interesting to the community. Since i didn't see obvious weakness in both theoretical and empirical results, I will give the score of 6 for now.", "strength_and_weaknesses": "Strength:\n(1) The proposed method is new and intuitive\n(2) The effectiveness of proposed algorithm is supported by both theoretical and empirical evidence\n\n\nWeakness:\n(1) It is not clear how the author come up with (4) (5) for mean and variance derivation. It seems that this formulation enables establishing convergence result but it is hard to see the intuition behind this formulation.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "hv9RjA6TOk", "reviewer": "Reviewer_rg2h", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this work, the authors propose a new variant of SAM optimizer, called AE-SAM by mixing SAM and SGD with a carefully designed condition. Moreover, the theoretical convergence of the proposed algorithm is also provided. Preliminary experiments demonstrate the efficacy of the proposed approach. ", "summary_of_the_review": "See the comments above. ", "strength_and_weaknesses": "The authors propose a new variant of the SAM optimizer. Preliminary experiments demonstrate its effectiveness. However, before this work is accepted, several concerns should be resolved:  \n(i)\tThe proposed AE-SAM falls into the scope of mixing SGD and SAM. Several existing works adopt a similar idea to improve the efficacy of SAM, including SS-SAM, and Gradient penalized SAM (Zhao et al, ICML 2022). The idea is not new, which limits the novelty of the proposed approach.\n(ii)\tAbout the practicality. Several hyperparameters are introduced in AE-SAM. In general, SAM-type methods are sensitive with respect to the perturbation parameters. More hyperparameters may make the tuning process more difficult. \n(iii)\tAbout the theory. In general, mini-batch SGD could achieve the linear speedup property with respect to the mini-batch size. (see Efficient Mini-batch Training for Stochastic Optimization, KDD). However, in Theorem 3.4, the linear speedup property is missing. In addition, can the author prove that the proposed AE-SAM theoretically outperforms randomized SAM (SS-SAM)? \n(iv)\tAbout the sensitivity of the adaptive policy. The proposed adaptive policy is highly related to mini-batch size. When training a deep neural network with a different mini-batch size, it shows different gradient variances. (see Not All Layers Are Equal: A Layer-Wise Adaptive Approach Toward Large-Scale DNN Training, WWW2022). The authors may conduct more experiments to show the consistency/sensitivity of the adaptive policy with respect to mini-batch size. \n(v)\tMore experiments verification on advanced neural network architecture are needed, such as vision transformer (such as Deit). \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PpgniGjX-j", "reviewer": "Reviewer_qUYU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Sharpness-aware minimization (SAM), which searches for flat minima by min-max optimization, has been shown to be useful in improving model generalization. However, SAM requires two gradient evaluations per iteration which makes it cost twice more expensive as SGD. One way to reduce this cost is mixing SAM with SGD in an adaptive manner. This paper proposes an adaptation based on the stochastic gradient’s norm.  The main idea is to keep an adaptive threshold (updating it per iterate), and then compare the gradient norm with this threshold: if it is bigger take a SAM update, otherwise take a SGD update.  They show the convergence of their proposed method. \n", "summary_of_the_review": "Comments: \n1- Assuming that the stochastic gradient follows a normal distribution is not well justified. For example, there exists research in the literature that shows this norm follows heavy-tailed distribution and not normal distribution. \n\n2- In your analysis, the adaptivity of the threshold doesn’t show up and the analysis is very similar to the method which takes SAM step randomly. Specifically, in the deterministic setting, there is no stochastic gradient and hence normal distribution assumption is incorrect. Besides in the empirical result, there is no significant difference between AE-SAM and SS-SAM. \n\n3- It is not explained in the paper why using AE-SAM can improve the robustness against label noises. \n", "strength_and_weaknesses": "Strength: Adaptive thresholding is a new approach to mixing SAM and SGD/ERM \nWeaknesses: choosing this threshold is not well justified. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster