Momentum-SAM: Sharpness Aware Minimization without Computational Overhead
Authors: Marlon Becker
Frederick Altrock
Benjamin Risse
Venue: NeurIPS 2025 poster ()
Keywords: Neural Networks
Optimization
Sharpness
Generalization
Overfitting
Primary area: deep_learning

Abstract: The recently proposed optimization algorithm for deep neural networks Sharpness Aware Minimization (SAM) suggests perturbing parameters before gradient calculation by a gradient ascent step to guide the optimization into parameter space regions of flat loss.
While significant generalization improvements and thus reduction of overfitting could be demonstrated, the computational costs are doubled due to the additionally needed gradient calculation, making SAM unfeasible in case of limited computationally capacities.
Motivated by Nesterov Accelerated Gradient (NAG) we propose Momentum-SAM (MSAM), which perturbs parameters in the direction of the accumulated momentum vector to achieve low sharpness without significant computational overhead or memory demands over SGD or Adam.
We evaluate MSAM in detail and reveal insights on separable mechanisms of NAG, SAM and MSAM regarding training optimization and generalization.

Momentum-SAM: Sharpness Aware Minimization without Computational Overhead
Marlon Becker
Frederick Altrock Benjamin Risse University of Muenster
(marlonbecker, altrook, b. rissee)uui-mmenstte
Abstract
The recently proposed optimization algorithm for deep neural networks Sharpness Aware Minimization (SAM) suggests perturbing parameters before gradient cal- culation by a gradient ascent step guide the optimization into parameter space regions of flat loss. While significant generalization improvements and thus reduc- tion of overfitting could be demonstrated, the computational costs doubled due to the additionally needed gradient calculation, making SAM unfeasible in case of limited computationally capacities. Motivated by Nesterov Accelerated Gradient (NAG) we propose Momentum SAM (MSAM), which perturbs parameters in the direction of the accumulated momentum vector to achieve low sharpness without significant computational overhead or memory demands over SGD or Adam. We evaluate MSAM in detail and reveal insights on separable mechanisms of NAG, SAM and MSAM regarding training optimization and generalization. Code is available at https: : //g:::/gtttttt thub. Foom/Marlowe hwt./MaaoooBBeeeeSSSSSS
Introduction
While artificial neural networks (ANNs) are typically trained by Empirical Risk Minimization (ERM), i... the minimization of predefined loss function finite set of training data, the actual purpose is to generalize over this dataset and fit the model to the underlying data distribution. Due to heavy overparameterization of state-of-the-art ANN models (Nakkiran et al. 2021). the risk of assimilating the training data increases. As a consequence, a fundamental challenge in designing network architectures and training procedures is ensure the objective of ERM to be adequate proxy for learning the underlying data distribution.
One strategy tackle this problem is exploit the properties of the loss landscape of the parameter space on the training data. A strong link between the sharpness in this loss landscape and the models generalization capability has been proposed by (Hochreitrr and Schmidhuber (1994) and further analyzed in the work of Keskar et al. (2017). Following these works, Forct er al. (2021) proposed an algorithm explicitly reduce the sharpness of loss minima and thereby improve the generalization performance, named Sharpness Aware Minimization (SAM). Built on top of gradient based optimizers such SGD or Adam (Kingma and Ba 2015). SAM searches for loss maximum in limited parameter vicinity for each optimization step and calculates the loss gradient at this ascended parameter position. To construct computationally feasible training algorithm, SAM approximates the loss landscape linearly so that the maximization is reduced to single gradient ascent step. Moreover, this step is performed on a single data batch rather than the full training set. Unfortunately, the ascent step requires an additional forward and backward pass of the network and therefore doubles the computational time, limiting the applications of SAM severely. Even though the linear approximation of the loss landscape poses vast simplification and Foret al. (2021) showed that searching for the maximum with multiple iterations of projected gradient ascent steps indeed yields higher maxima, these maxima, however, do not improve the generalization, suggesting that finding the actual maximum the local vicinity is not pivotal. Instead, it appears to be sufficient
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

to alter the parameters to find an elevated point and perform the gradient calculation from there. Following this reasoning, the ascent step can be understood as a temporary parameter perturbation. revealing strong resemblance of the SAM algorithm to extragradient methods (Korpelevich) 1976) and Nesterov Accelerated Gradient (Nesterov, 1983; Sutskever et al.) 2013) which both calculate gradients at perturbed positions and were also discussed previously in the context of sharpness and generalization (Lin et al.) 2020al Wenn et al. 2018).
Commonly, measures to address generalization issues are applied between the data distribution and the full training dataset (Keskar et al., 2017 ei al. 2018). Calculating perturbations on single batches, as done by SAM, results in estimating sharpness full training dataset. Thus motivated, we reconsider the effect of momentum in batch-based gradient optimization algorithms as follows. The momentum vector not only represents trace over iterations in the loss landscape and therefore accumulates the gradients at past parameter positions, but also builds an exponential moving average over gradients of successive batches. Hence, the resultant momentum vector can also be seen as an approximation of the gradient of the loss on a larger subset- in the limiting case the full training dataset.
single batches instead of the intended
Building on these observations and the theoretical framework of SAM, which assumes using the entire dataset for sharpness estimations, we present Momentum-SAn (MSAM), MSAM aims to minimize the global sharpness without imposing additional forward and backward pass computations by using the momentum direction as an approximated, yet less stochastic, direction for sharpness computations. In summary, our contribution is as follows:
We propose Momentum-SAA (MSAM), an algorithm to minimize training loss sharpness without computational overhead over base optimizers such as SGD or Adam.
The simplicity of our algorithm and the reduced computational costs enable the usage of sharpness-awar minimization for variety of different applications without severely compromising the generalization capabilities and performance improvements of SAM.
We discuss similarities and differences between MSAM and Nesterov Accelerated Gradient (NAG) and reveal novel perspectives on SAM, MSAM, as well as on NAG.
We analyze the underlying effects of MSAM, with particular focus the slope of the momentum vector directions, and share observations relevant for understanding momentum training beyond sharpness minimization.
We validate MSAM on multiple benchmark datasets and neural network architectures and compare MSAM against related sharpness-award approaches.
1.1 Related Work
Correlations between loss sharpness, generalization, and overfitting were studied extensively (Hochre-
iter and Schmidhuber 1994 Keskar el al., 2017 Lin el al.) [0200] Yao el al. 2018 Li el al. 2018)
Liu et al. [2020] Damian et al. 2021). all linking flatter minima to better generalization, while Dinh
et al. (2017) showed that sharp minima can generalize too. While the above-mentioned works focus
on analyzing loss sharpness, algorithms to explicitly target sharpness reduction were suggested by
Zheng et al. (202T); Wu et al. 2 Chaudhari et al. (017)) with SAM (Foret et al. 2021 being most prevalent.
SAM relies on computing gradients at parameters distinct from the current iterations position. This resembles extragradient methods (Korpelevich) 1976) like Optimistic Mirror Descent (OMD) (Judit-
sky el al.) 2011) or Nesterov Accelerated Gradient (NAG) (Nesterov) 1983 Sutskever el al. (2013)
which were also applied Deep Learning, either based on perturbations by last iterations gradients
DDakalakii et al. 2018: Lin el at.) (2200a) or random perturbations (Wen el al. 2018).
Adaptive-SAM (ASAM) (Kwon et al. (222I) accommodates SAM by scaling the perturbations relative to the weights norms to take scale invariance between layers into account, resulting in a significant performance improvement over SAM. Furthermore, Kim et al. (2022) refine ASAM by considering Fisher information geometry of the parameter space. Also seeking to improve SAM, GSAM (Zhuang et al.) (022) posit that minimizing the perturbed loss might not guarantee a flatter loss and suggest using a combination of the SAM gradient and the SGD gradients component orthogonal to the SAM gradient for the weight updates.
Unlike the aforementioned methods, several algorithms were proposed to reduce SAMs runtime, mostly sharing the idea of reducing the number of additional forward/backward passes, in contrast to approach which relies on finding more efficient parameter perturbations. For example, Jiang et al.

---

2023) are evaluating in each iteration if a perturbation calculation is to be performed. LookSAM Liu et al. 2022) updates perturbations only each k-thh iterations and applies perturbation components orthogonal to SGD gradients in iterations in between. Mi el al. (2022) are following an approach based on sparse matrix operations and ESAM (Du et al, (2022b) combines parameter sparsification with the idea to reduce the number of input samples for second forward/backward passes. Similarly, Bahri et al. (202I) and Ni et al. (2022) calculate perturbations on micro-batchess Not explicitly targeted at efficiency optimization, Mueller et al. (2023) show that only perturbing Batch Norm layers even further improves SAM. SAF and its memory efficient version MESA A were proposed by Du al. (2022a), focusing on storing past iterations weights to minimize sharpness output instead of the loss function. Perturbations in momentum direction after the momentum buffer update resulting in better performance but no speedup where proposed by Li and Giannakis (2023). Furthermore, several concepts were proposed to explain the success of SAM and related approaches going beyond sharpness reduction of the training loss landscape (Andriushchenko and Flammarion 2022; Möllenhoff and Khan 2023; Andriushchenko et al., (023b), with the influence on the balance of features/activations being a promising alternative explanation (Andriushchenko et al] 2023a] Springer et al.) (2024), which we also investigate in Appx. A.6 Zhang et al. (2024) analyze how SAM influences adversarial robustness, a question we explore for MSAM in Appendix A.16
the digits
2 Method
2.1 Notation
Given a finite training dataset S C X' x y where X' is the set of possible inputs and y' the set of possible targets drawn from joint distribution D, study a model fw X + 3) parameterized by w E W. an element-wise loss function / W x X' x ) + R. the distribution loss L (w) = E( ((w, x. v)) and the empirical (training) loss (s(w)
1/|8| (i..)es (w,,,y). If calculated on a single batch B c we denote the loss as Lg- We denote the L2- by -|
2.2 Sharpness Aware Minimization (SAM)
While for many datasets modern neural network architectures and empirical risk minimization algorithms, like SGD or Adam (Kingma and Ba 2015), effectively minimize the approximation and optimization eтror (i.c. finding low Ls(w), reducing the generalization error (Lo (w) Ls(w))
remains a major challenge. Following ideas of Hochreiter and Schmidhuber (1994) Keskar et al.
(2117) observed link between sharpness of the minimized empirical loss Ls (w) with respect to the parameters and the generalization error. Intuitively, this follows from the observation that perturbations in inputs (ef. adversarial training (Goodfellow et al., 2015)) and perturbations in parameters have a similar effect on network outputs (due to both being factors in matrix-vector products) and that the generalization error is caused by the limitation to a smaller input subset which resembles an input perturbation.
Without giving an explicit implementation, Keskar et al. (2017) sketches the idea of avoiding sharp minima by replacing the empirical loss minimization with minimization of the highest loss value within a ball in parameter space of fixed size p:
min max Ls(w + E) |e
Foret et al. (2021) propose computationally feasible algorithm to approximate this training objective via so-called Sharpness Aware Minimization (SAM). SAM heavily reduces the computational costs of the inner maximization routine of Eq. by approximating the loss landscape in first order, neglecting second order derivatives resulting from the min-max objective, and performing the maximization on single batches (or per GPU in case of m-sharpness). These simplifications result in adding one gradient ascent step with fixed step length before the gradient calculation, i.c., reformulating the loss as
V LB (w)
I (w) = LI(Ww + CSAM) where
SAM P
V L (
The parameters are temporarily perturbed by SAM in the direction of the locally highest slope with the perturbation removed again after gradient calculation. Thus, the parameters are not altered permanently. While performance improvements could be achieved (Foret et al. 2021 Chen et

---

al. )022), the computation of eSAM demands an additional backward pass and the computation of L ESAM) an additional forward pass, resulting roughly doubling the runtime of SAM
compared to base optimizer like SGD or Adam.
Minimizing Eq. 2 can also interpreted as jointly minimizing the unperturbed loss function (u))
and the sharpness of the loss landscape defined by
Sg(w (w) == Ls(w + c) - Ls(w).
(3)
2.3 Momentum and Nesterov Accelerated Gradient
Commonly, SGD is used with momentum, i.e., instead of updating parameters by gradients di- rectly (W1++1 Wt r/V Lg, (w;) with learning rate 17), an exponential moving average of past gradients is used for the updates. Given the momentum factor μ and the momentum vector Vt-1 = jar + VLs, (wy) the update rule becomes W++1 = wi 170++1.
The momentum vector has two averaging effects. First, averages the gradient at different positions in the parameter space w. and second, it averages the gradient over multiple batches B, which can be interpreted as an increase of the effective batch size.
The update step consists of the momentum vector of the past iteration and the present iterations gradi- ent. While this gradient is calculated prior to the momentum vector update in standard momentum training, NAG instead calculates the gradient after the momentum vector step is performed. The update rule for the momentum vector thus becomes Ut+1 = jet; + V LB, (w, - mHwe)). Analogously to Eq. 2 NAG can be formulated in terms of perturbed loss function
NAG (w) LB + ENAG where NAG -npeer
Since the perturbation vector E NAG neither depends the networks output nor its gradient at step t no additional forward or backward pass is needed.
2.4
Momentum-SAM
Foret et al. 2021 show that performing multiple iterations of projected gradient ascent in the inner maximization does result in parameters with higher loss inside the p-ball (cf. [. However, and counterintuitively. this improved inner maximization does not yield better generalization of the model. We conclude that finding the exact (per batch) local maximum is not pivotal to SAM. Inspired by NAG and given that the theoretical framework of sharpness minimization is based on calculating the sharpness on the full training dataset, we propose using the momentum vector as the perturbation direction and call the resulting algorithm Momentum-SA.M (MSAM) (further perturbations are discussed in Appx. A.7. Following the above notation, this yields the loss objective
MSAM (w) := Ls(w + LB
e MSAM) where E MSAM
(5)
Contrary to SAM, we perturb in the negative direction. While this seems countcrintuitive at first glance, negative momentum directions actually cause a loss increase and are thus suitable for sharpness estimation. Since we use the momentum vector before it is updated, a step in the negative direction of the momentum has already been performed in the iteration before. We observe that this update steps overshoots the local minima in the direction of the former iterations momentum vector. Thus, when evaluated on the batch of the new iteration, the momentum direction exhibits negative slope, caused by the high curvature in this direction. The stepsize by MSAM is typically at least order of magnitude higher than learning rate steps, occurring local minima in momentum direction and reach an increased perturbed loss which is used for sharpness minimization. We empirically validate this in detail in Sec. 4.1 Sec. A.10 and Appx. A.3]
additionally overshoot eventually
For an efficient implementation, we shift the starting point of each iteration to be the perturbed parameters W; = w. pe:/1|o/ll (in analogy to common implementations of NAG) and remove the final perturbation after the last iteration (see Alg. D All mentioned optimization strategies are depicted in detail in Fig. [ Since SGD with momentum as well as Adam store a running mean of gradients, MSAM does not take up additional memory and comes with negligible computational overhead.
Furthermore, we confirm that a similar theoretical generalization bound as reported by Foret el al. (202I) also holds for directions of high curvature as the momentum direction (see Appx. A.I)

---

SGD
SGD with momentum
NAG
Dt
(LB,(w)))
Wi
Vt+1
100
VLB,(W (101))
00++1
Wy
9t
w++++1
W1+1
1
U+41
SAM
MSAM
MSAM
(eff. impl.)
9SAM
Et P
9MSAM
W
JSAM
MSAA
1T
91
D1+1
W1+1
9MSAM
1000
9MSAM
Wt
9t
Dt++1
Wi+1
Wi
V++1
W4+1
W++1
Figure I: Schematic illustrations of optimization algorithms based on SGD. NAG calculates gradients after updating parameters with the momentum vector. SAM and MSAM calculate gradients perturbed positions but remove perturbations again before the parameter update step. See Alg. [ detailed description of the efficient implementation of MSAM.
for
Algorithm 1: SGD with Momentum- -SAM (MSAM; efficient implementation)
Input: training data S, momentum learning rate 7), perturbation strength p Initialize: weights we ranndmm, momentum vector vo <-
for t- 0 to 7 do
sample batch B, C LB, (w)) = 1/ ((w))E 1(W,, 2,
9MSAM V LB, (w))
// inc pert. // remove last pert. // update momentum // SGD step // next pert.
w, = wt + P |milll
th
Ut+1 μv 9MSAM
W++1 = wt 1/1++1
Vesl Wi+1 = wi+1 P ||
end
vr Wr WT + P [ur
// remove pert.
return WT
3 Experimental Results
3.1 Speed and Accuracy for ResNets on CIFAR100
In Tab. [ we show test accuracies for MSAM and related optimizers for WideResNer-28. 10, WideResNNet11-44 (Zagoruyko and Komodakis 2016) and ResNe150 (He ef al.) (0066) on CIFAR100 (Krizeesky and Hinton| (2009) and vision transformers (Dosovitskiy el al. 2021 and ImageNet- 1k Deng al. (2009) next the training speed. We first tuned the learning rate and weight decay for SGD/Adam W and then optimized for each model (see Appx. A. for more details), Additionally,
Table 1: Comparison against multiple (share)ess-aware) optimizers. Baseline optimizers are SGD for CIFAR 100 and Adam for ImageNet. Please see Appx. [..x|AA.14ffr for experimental details. MSAM outperforms optimizers of equal speed (AdamW/SGD and NAG) and alternative approaches for faster sharpness reduction.
CIFARI00
ImageNet ViT-S/32
Optimizer
Speed
WRN-28-10
WRN-16-4
ResNet-50
SAM
84.1620.12
79.25
83.3610.17
69.1
0.52
Bascline
81.51
76.90
81.46+0.13
67.0
1.00
NAG
82.00=0.11
77.09
82, 10.12
0.99
LookSAM
83.31+0.1 10.12
79.00+o.08
82.24+t.11
68.0
0.84
ESAM
82.71=0.38
77.7940.11
80.49
66.1
0.62
MESA
82.75=2.00
78.3210.08
81.94
69.0
0.77
MSAM (ours)
83.21 40.07
79.11+o.00
82.65
69.1
0.99
5

---

we conducted experiments with related approaches which seek to make SAM more efficient, namely ESAM (Du et al. 20226), LookSAM (Liu et al. 2022) and MESA (Du et al. 2022a). While et al. (2022a) also proposed second optimizer (SAF), we decided to compare against the memory-eficient version MESA (as recommended by the authors for e.g. ImageNet). Note that LookSAM required tuning of an additional hyperparameter. See Appx. A.14 for implementation details on the related optimizers. Optimizers of the same speed as MSAM (i.e. SGD/Adam W and NAG) are significantly outperformed. While SAM reaches slightly higher accuracies than MSAM, twice much runtime is needed, Accuracies of MSAM and LookSAM do not differ significantly for WideResNets, however, MSAM performs better ResNet-50, is faster, and does not demand additional hyperparameter tuning. For ESAM we observed only a minor speedup compared to SAM and the accuracies of MSAM could not be reached. MESA yields similar results to MSAM for ViT ImageNet but performs worse on all models on CIFAR 100 and is slower compared to MSAM.
3.2 ResNet and ViT on ImageNet Results
Moreover, we test MSAM for ResNets (He et al. 2016) and further VIT variants (Dosovitskiy et al. 2021) on ImageNet- 1k (Deng et al. 2009) and report results in Tab. Due to limited computational resources, we only run single iterations, but provide an estimate of the uncertainty by running iterations of baseline optimizers for the smallest models per category and calculate the standard deviations. During the learning rate warm-up phase commonly used for ViTs we set PMSAM = O. SAM also benefits from this effect, but less pronounced, we kept SAM active during warm- up phase to stay consistent with related work (see Appx. 12 for detailed discussion). While performance improvements are small for ResNets for MSAM and SAM, both optimizers achieve clear improvements for ViTs. Even though slightly below SAMs performance for most models, MSAM yields comparable results while being almost twice as fast.
In addition, we conducted experiments for ViT-S/32 on ImageNet when giving MSAM the same computational budget as SAM (i.e. training for 180 epochs) yielding a test accuracy of 70.1 1% and thus clearly outperforming SAMs 69.1% 1% (also see Appx. A.9)
Table 2: Test accuracies ImageNet for baseline optimizers (SGD AdamW), SAM and MSAM. Estimated uncertainties: ResNet: +-008, ViT (90 epochs): +0.177 ViT (300 epochs): +0.24. Im- provements baseline are given in green. MSAM yields results comparable to SAM for most models while being ~ 2 times faster in all our experiments.
Model
Epochs
Baseline
SAM
MSAM
KesNat-50
100
SGD
76.3
76.6
70.0+02
ResNes-101
100
SGD
77.9
78.8
182:003
VIT-S/32
300
AdamW
67.2
7444
70.5e3.3
90
Adamm
67.0
69.1421
60.1+2a
VIT-S/16
300
AdamW
73.0
78.243.
75.8695
90
Adamm
72.6
75.8432
74.9223
WT-
90
AdamW
06.9
70.4
70.2:3
B/6
90
Adamn
73.0
77.1447
75.7427
3.3 Combination with other SAM Variants
As shown by Kwon et al. (021). weighting the perturbation components by the parameters signifi- cantly improves SAM. Similarly, Mueller et al. (2023) showed that applying the perturbations only to the Batch Norm layers (loffe and Szegedy, 2015) yields further enhancements. Both of these techniques can also be applied to MSAM, yielding test results similar to SAM (see Tab. 3
Properties of MSAM
In the following we analyze the perturbation direction of MSAM, its similarities to SAM and the
loss sharpness in more detail. Additional experiments are presented in the appendix. These include the observations that the perturbation normalization is not pivotal to MSAM and instead the binding to the learning rate 7) discriminates NAG and MSAM (Appx. A.2) and that MSAM reduces feature ranks more effectively than SAM (Appx. A.6).

---

Table 3: Test accuracy for different variants of MSAM/SAM on CIFARIO0. Adaptive refers to ASAM (Kwon et al. [02T) and BN-only to applying the perturbance only to Batch Norm layers (cf. Mueller et al. 2023) ). MSAM performs well with both variants.
Optimizer
WRN-28-10
WRN-I6-4
ResNet-50
SGD
81.32
70.00
81.40600.0
SAM
81. 11.16=0.1
79.25
83.36e0.17
vanilla
MSAM
84.2
19.11
82.6510.12
adaptive
SAM
84.74
79.00.0.11
83.301g0.0
MSAM
84.15
19.89
83.4810.08
BN-only
SAM
84.37g007
79.73an.54
84.5129.17
MSAM
83.622009
79.73.0.14
83.49aa.1
4.1 Negative and Positive Perturbation Directions
Instead of ascending along the pasitive gradient as in SAM, we propose perturbing along the negative momentum vector (positive p MSAA in our notation) as it is also done by extragradient methods as by Lin et al. (2020a). Counterintuitvelly, the cosine similarity between the momentum vector e,-1 and the gradient dr V LB, (w:) is negative. Thus, the negative momentum direction actually has positive slope, so that perturbing in this direction resembles an ascent on the per-batch loss. An update step in the momentum direction was already performed and the momentum direction is typically of high curvature (see Appx. The positive slope of the negative momentum direction can thus be explained by the parameter update overshooting local minima if evaluated on the new iterations batch. As a consequence, moving further in this direction yields valid sharpness estimates. not already caused by the SGD update step, the overshooting perturbation step will overshoot the minima since the used perturbations optimization step sizes. Positive momentum perturbations, instead, yield improper estimates of the local sharpness since the minima which overshot is approached again. We analyze this effeet in detail in Appx. A.33 In addition, we provide direct evidence that the proposed perturbation results in loss increase suitable for sharpness estimations in Appx. A0 The increased generalization of MSAM directly follows from this phenomenon, which we show in our theoretical consideration in Appx. based on PAC-Bayes bound.
typically at least order of magnitude larger than the
We did not observe any increase in test performance when perturbing in the positive momentum
A
B
(l:B uzi))
16.6
KAG
PHISAM
--- Tran (riphe aoda!
SCID
70
79-
VAL4
Cortte Starbortyy (66-4)
NAG
SGD
-14:2
-90.0
77-
77-
-6.8
Renning mean (36 Hteration)
-0.50
Per Ilcatioo
70
Pertzzlation Struutth
110 -30
Pesturatto
10"33
100
100
200
Figure 2: WideRestNeet1664 CIFAR100 A: Test accuracy for positive and negative P compared against SGD and NAG. B: Train and test accuracy on logarithmic scale. C: Cosine similarity between momentum vector 0t-1 and gradient St v Lg, (ux.). Momentum vector direction has mostly negative slope during training and approaches zero at the end (caused by cosine learning rate scheduler).
direction (negative م MSAM). In fact, negative o values cause a rapid decrease in test accuracy, whereas positive values cause a gain in test accuracy of more than 2% for a WideResNet- 16-4 trained on CIFARION 100 as depicted in Fig. _B, while NAG only provides minor improvements.
Fig. 2C shows the same data on logarithmic scale next to the test accuracy. The ordinate limits are chosen such that baseline (SGD) accuracies as well as maximal gains by MSAM align. NAG improves the training accuracy greatly, especially compared to the gains in test accuracy. This underlines that NAG is designed to foster the optimization procedure (ERM) but does not improve the generalization capabilities of the model. Similarly, for MSAM the maximal test accuracy is reached for high values of p where the train accuracy dropped far below the baseline, emphasizing the effect of MSAM on the generalization instead of optimization.
7

---

Furthermore, small negative values of induce a steep decrease in training accuracy while the test accuracy significantly affected, but drops for higher negative p. So the generalization improves also for negative p values, however, the test performance does not, since the improved generalization overlaid by the decreasing training performance. This offers an additional explanation why MSAM does not improve the test accuracy for positive momentum perturbations. A perturbation in the positive momentum vector direction resembles step back to past iterations which might result in the gradient not encoding appropriate present or future information about the optimization direction and thus seems to be ill-suited to reduce the training loss effectively, counteracting the benefits of the increased generalization (smaller test-train gap). SAM might not suffer from this effect, since the local (per batch) gradient does not encode much of the general optimization direction (which is dominated by the momentum vector), hence, the perturbed parameters disagree with parameters from previous iterations.
4.2 Similarity between SAM and MSAM
To support our hypotheses that MSAM yields valid approximation for SAM's gradient calculations, we investigate the similarity between the resulting gradients. After searching for the optimal value, keep PSAM 0.3 fixed, train model with SAM, and calculate the gradients
ISGD V LLS, (we)
9SAM = LLB, (we + PSAM V L,g,
9MSAM (PMSAA)
VLB, (w, PMSAM //
while we keep PMSAM free parameter. To eliminate gradient directions which do not contribute in distinguishing between SAM and SGD gradients, we first project gMSAM into the plane spanned by 9SAM and IsGD and then calculate the angle 0 9SAM (see Fig. BA). By repeating this every 50 iterations for various values PMSAM and calculating the value of zero- crossing pu. we determine when the maximal resemblance SAM is reached (see Fig. (3B). As shown in Fig. BC. Po reaches values close to the optimal regime of PMSAM x 3 (ef. Fig. A for most epochs. While this correlation does not yield strict evidence offers additional empirical support for the similarity between SAM and MSAM gradients.
A
B
ISAM
Stasbertty Augl
Busac
2
-107
$500
50
Epoch
100
160
200
Figure 3: A: Projecting IMSAM into the plane of JSAM and JSGD to measure SAMMSSAM similarity. B: Varying PMSAM until maximal similarity is reached (i.e. 0 = 0) and determine Po- C: PMsAM at maximal similarity Po is close to generalization optimality (PMSAM- cf. Fig. 2 for most epochs.
4.3 Loss Sharpness Analysis
As mentioned above, the SAM implementation performs the loss maximization step on a single data batch instead of the full training set (Lg vs. Ls). To analyze the efficacy of SAMs sharpness minimization, we therefore compare the sharpness (cf. Eq. 33 in the direction of local (per batch) gradients for models after full training with SGD, SAM and MSAM as a function of the perturbation
scale in Fig. 4A. The minima in local gradient directions are shifted from O, since parameters
found after training are usually not minima but saddle points (Dauphin et al.) 2014). Compared to the other optimizers, SAM successfully minimizes the sharpness, especially at optimal Popt (as used during training).
The sharpness in momentum direction (Fig. represents the MSAM objective (Eq. 5 Here we do not include the negative sign the definition of (as in Eq. 5). hence Popt negative. As expected, MSAM reduces this sharpness best. In contrast to the definition before, the sharpness is

---

A
B
SGD
SAM
Stapping
Sterring
Persahattion
Steraghh
-10
Partstlacion
Figure 4: Sharpness (Eq.) after full training (200 epochs) for WideResNet- along different directions e scaled by p. A: Gradient direction (as used as perturbation in SAM). B: Momentum direction (as in MSAM). C: Random normalzeddl direction in Li et al. 2018). Vertical line at na marks values of optimal test performance (cf. Fig. A. MSAM and SAM reducing their respective optimization objective best while MSAM reaches the lowest sharpness along random directions.
CIFARI00
symmetric now for positive and negative signs. While MSAM only minimizes the sharpness in the negative direction explicitly, the positive direction is reduced jointly, further supporting the validity of perturbations in the negative momentum direction.
In Fig. ECC, we choose filter-normalized random vectors perturbations as in Li el al. (208)). Since the loss landscape is rotational symmetric around the origin for multiple directions (as used in the original paper), we confine our analysis to one perturbation direction allowing easier comparison (see Appx. A.IT for two perturbation directions). MSAM reaches the lowest sharpness, while both MSAM and SAM, significantly flatten the loss. This might be caused by MSAM approximating the maximization of Ls better due to the momentum vector v2 being an aggregation of gradients of multiple batches. Interestingly, this relates to the findings Foret et al. (2021) regarding m-shaappness, where the authors performed the maximization on even smaller data samples (fractions of batches per GPU in distributed training), yielding even better generalization. In this sense MSAM, reduces further over ordinary SAM (m = 1). In the same line of argument and despite being more efficient, MSAM oftentimes does not improve generalization compared to SAM. However, contradicting the general idea behind correlations of generalization and sharpness, MSAM yields flatter minima (if defined in Fig. 4C), hence, indicating that explanations for the improved generalization of SAM/MSAM go beyond the reduction in sharpness. Additionally, we show that SAM guides the optimization through regions of lowest curvature compared SGD and SAM Appx. A.4 and yields low Hessian eigenvalues Appx. A.5
5 Conclusion
In this work we introduced Momentum-SAM (MSAM), an optimizer achieving comparable results to the SAM optimizer while requiring no significant computational memory overhead over optimizers such as Adam or SGD, hence, halving the computational load compared to SAM and thus reducing the major hindrance for widespread application of SAM-like algorithms when training resources are limited. Alternative efficient approaches (which are mostly based on simple sparsifications) are all outperformed in speed and accuracy. Instead, showed that perturbations independent of local gradients can yield significant performance enhancements. In particular perturbations in the negative momentum buffer direction yield substantial generalization improvements. While the negative sign of the perturbations seemed counterintuitive at first glance, we analyzed the loss landscape in momentum direction in detail show that negative perturbation actually causes loss ascent (please also see the appendix for further analysis). In summary, we not only proposed analysis yielding multiple new perspectives on sharpness aware optimization and momentum training in general.
optimization method, but also offered detailed empirical

---

Acknowledgements
This work funded by the German Research Foundation (DFG CRC 1459 Intelligent Matter Project-ID 433682494).
References
Maksym Andriushchenko and Nicolas Flammarion. Towards understanding sharpness-aware mini- mization. In International Conference on Machine Learning, 2022.
Maksym Andriushchenko, Dara Bahri, Hossein Mobahi, and Nicolas Flammarion. Sharpness-awest aware minimization leads to low-rank features. In Advances in Neural Information Processing Systems, 2023.
Maksym Andriushchenko. Francesco Croce, Maximilian Müller, Matthias Hein, and Nicolas Flam- marion. A modern look at the relationship between sharpness and generalization. In Proceedings of the 40th International Conference on Machine Learning, 2023.
Dara Bahri, Hossein Mobahi, and Yi Tay. Sharpness-aware minimization improves language model generalization. In Annual Meeting of the Association for Computational Linguistics, 2021.
Lucas Beyer, Xiaohua Zhai, and Alexander Kolesnikov. Better plain vit baselines for imagenet- kk.
CoRR, abs/2205.01 580000 2022.
Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-SGD: Biasing gradient descent into wide valleys. In International Conference on Learning Representations, 2017.
Xiangning Chen, Cho-Jui Hsieh, and Boqing Gong. When vision transformers outperform resnets without pre-training strong data augmentations. In International Conference on Learning Representations, 2022.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free free attacks. In Proceedings of the 37th International Conference on Machine Learning, 2020.
Yan Dai, Kwangjun Ahn, and Suvrit Sra. The crucial role of normalization in sharpness-aware minimization. In Advances in Neural Information Processing Systems, 2023.
Alex Damian, Tengyu Ma, and Jason D. Lee. Label noise SGD provably prefers flat global minimizers. In Advances in Neural Information Processing Systems, 2021.
Constantinos Daskalakis, Andrew Ilyas, Vasilis Syrgkanis. and Haoyang Zeng. Training gans with optimism. In International Conference on Learning Representations,, 2018.
Yann N. Dauphin, Razvan Pascanu, Çaglar Gülçehre, KyungHyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. In Advances in Neural Information Processing Systems, 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In Proceedings of the 34th International Conference on Machine Learning, 2017.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Jiawei Du, Zhou Daquan, Jiashi Feng, Vincent Tan, and Joey Tianyi Zhou. Sharpness-aware training for free. In Advances in Neural Information Processing Systems, 2022.
Jiawei Du, Hanshu Yan, Jiashi Feng, Joey Tianyi Zhou, Liangli Zhen, Rick Siow Mong Goh, and Vincent Tan. Efficient sharpness aware minimization for improved training of neural networks, In International Conference on Learning Representations, 2022.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. International Conference on Learning Representations, 2021.
10

---

Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015.
lan J. Goodfellow, Jonathon Shlens, and Christian Szeged. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Simplifying neural nets by discovering flat minima. In Advances in Neural Information Processing Systems, 1994.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Francis R. Bach and David M. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, 2015.
Weisen Jiang, Hansi Yang. Yu Zhang, and James Kwok. An adaptive policy to employ sharpness- aware minimization. In International Conference Learning Representations, 2023.
Anatoli Juditsky, Arkadi Nemirovski, and Claire Tauvel. Solving variational inequalities with stochastic mirror-prox algorithm. Stochastic Systems, 1(I), 2011.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference Learning Representutions, 2017.
Minyoung Kim, Da Li, Shell X Hu, and Timothy Hospedales. Fisher SAM: Information geometry and sharpness aware minimisation. In Proceedings of the 39th International Conference on Machine Learning, 2022.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
G. M. Korpelevich. The extragradient method for finding saddle points and other problems. In Ekonomika Matematicheskie Metody, volume 12, 1976.
Alex Krizhevsky and Geoffrey Hinton. Learnnng multiple layers of features from tiny images. Technical report, University of Toronto, Toronto, Ontario, 2009.
Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi. Asam: Adaptive sharpness-an aware minimization for inver-inv learning of deep neural networks. In Proceedings of the 38th International Conference on Machine Learning, 2021.
Bingcong Li and Georgios B. Giannakis. Enhancing sharpness- optimization through variance suppression. In seirth Conference on Neural Information Processing Systems, 2023.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In Advances in Neural Information Processing Systems, 2018
Tao Lin, Lingjing Kong, Sebastian Stich, and Martin Jaggi. Extrapolation for large-batch training in deep learning. In Proceedings of the 37th International Conference on Machine Learning, 2020.
Tao Lin, Sebastian U. Stich, Kumar Kshitij Patel, and Martin Jaggi. Don't use large mini-batches, use local SGD. In International Conference on Learning Representations, 2020.
Chen Liu, Mathieu Salzmann, Tao Lin, Ryota Tomioka, and Sabine Süsstrunk. On the loss landscape of adversarial training: Identifying challenges and how to overcome them. In Advances Neural Information Processing Systems, 2020.
Yong Liu, Siqi Mai, Xiangning Chen, Cho-Jui Hsieh, and Yang You. Towards efficient and scalable sharpness-aware minimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. International Conference on Learning Representations, 2018
11

---

Peng Mi, Li Shen, Tianhe Ren. Yiyi Zhou, Xiaoshuai Sun, Rongrong Ji, and Dacheng Tao. Make sharpness-awe aware minimization stronger: A sparsified perturbation approach. Advances in Neural Information Processing Systems, 2022.
Thomas Möllenhoff and Mohammad Emtiyaz Khan. SAM as an optimal relaxation of bayes. In International Conference on Learning Reprexentations, 2023.
Maximilian Mueller, Tiffany Vlaar, David Rolnick, and Matthias Hein. Normalization layers all that sharpness-aware minimization needs. In Advances in Neural Information Processing Systems, 2023.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and llya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021(12), 2021.
Y. Nesterov. A method for solving a convex programming problem with convergence rate o(1/k2), 1983.
Renkun Ni, Ping yeh Chiang, Jonas Geiping, Micah Goldblum, Andrew Gordon Wilson, and Tom Goldstein. K-sam: Sharpness-aware minimization at the speed of sgd, 2022.
Jacob Mitchell Springer, Vaishnavh Nagarajan, and Aditi Raghunathan. Sharpness-aware minimiza- tion enhances feature quality via balanced learning. In International Conference on Learning Representations, 2024.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance initialization and momentum in deep learning. In Proceedings of the 30th International Conference on Machine Learning, Proceedings of Machine Learning Research, 2013.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
Wei Wen, Yandan Wang, Feng Yan, Cong Xu, Yiran Chen, and Hai Li. Smoothout: Smoothing out sharp minima for generalization in large-batch deep learning. CoRR, 000p//80500788, 2018.
Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust general- ization. In Advances in Neural Information Processing Systems, 2020.
Zhewei Yao, Amir Gholami, Qi Lei, Kurt Keutzer, and Michael W. Mahoney. Hessian-based analysis of large batch training and robustness to adversaries. In Advances in Neural Information Processing Systems, 2018
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Proceedings of the British Machine Vision Conference, 2016.
Yihao Zhang, Hangzhou He, Jingyu Zhu, Huanran Chen, and Yifei Wang amd Zeming Wei. On the duality between sharpness-aware minimization and adversarial training. In Proceedings of the 41th International Conference on Machine Learning, 2024.
Yaowei Zheng, Richong Zhang, and Yongyi Mao. Regularizing neural networks via adversarial model perturbation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Juntang Zhuang, Boging Gong, Liangzhe Yuan, Yin Cui, Hartwig Adam, Nicha C Dvorne,, sekhar tatikonda, James Dunean, and Ting Liu. Surrogate gap minimization improves sharpness-aware training. In International Eleventh Conference on Learning Representations, 2022.
12

---

Appendix
A.1 Theoretical Analysis of MSAM
We build our analysis in analogy to Foret et al. (021)) While Foret et al. (2021) proofed the existence of upper generalization bound if the parameters with the highest loss fixed p-ball are found, we show that a similar bound can also be derived by simply assuming perturbations in directions of high curvature. In practice this first assumption is fulfilled by the momentum vector Uts since the directions of gradients are directions of high curvature. Secondly, if properly tuned learning rate used, the slope in momentum direction after the parameter update is either close zero or even negative caused by overshooting marginally.
We state these two assumptions Setring While we empirically validate the first assumption in Appx. A.4. we already showed and discussed evidence for the second assumption in the main text (cf. Fig. 2C).
Proposition 1 Let E, W with i.i.d. components E, NO o) for some 0, then for any
p>0 0
(l o Hess(Ls(w))) s p²n,
(A.1)
where K. = TwT tr (Ls)sss
Proof W.L.O.G. assume Hess (Ls))) to be diagonal. The claim then follows from the linearity of the expectation and symmetry.
Setting Let w. v E W with llla| such that:
* a Hesss (wss))
VLs(w) < O.
Lemma Assume Setting 1 and let E E W with e, (O, or), then it holds for any p >> 0 that
E (Udlaa))) Lsw E)] s Ls(w pv) + O(p²).
Proof Applying Taylor expansion around w yields:
(( + E)] I Ls(w - pv)
+ E 1 (l VLs(w) c|+ E|1 ([e]laas)) e" HeHHesssLs (w)))
Spk
(VLs(w) +p2 Hess
T (ssss))s(w))rr)) +
k o HesseeH(LL(u))i + O(p²)
subtracting the -term from the initial inequality then yields the claim.
Theorem Assume Setting 7 then for any distribution D, with probability 5 over the choice of the training Set S D.
Lo(w)= < Ls(w-poo pw)) dim(W) log (1
||
1 +
[S)([S]) WW]
2)
+ log
[S] 3 + +11
+0(p3)
[S]
Proof Using the bound from Lemmal 2 we adapt the proof of Theorem 2 in Foret et al. 2027 (i.e
Eq. 13 and following) show the claim.
A.2 Normalization
To gain a better understanding of the relation between MSAM and NAG, we conducted an abla- tion study by gradually altering the MSAM algorithm until it matches NAG, Firstly, drop the
13

---

normalization of the perturbation e (numerator in Eq. 5 then we reintroduce the learning rate scale e, and finally set p to arrive NAG. Train and accuracies functions p are shown in Fig. A. for all variants. Dropping the normalization only causes a shift of indicating that changes the momentum norm during training are negligible (contradicting Dai al. (2023)) However, scaling by 1y drastically impacts the performance. Since the model is trained with cosine learning rate scheduler (Loshchilov and Hutter, 2017), p decays jointly with 7, The train accuracy is increased significantly not only for p = (NAG), but even further for higher p. while the test performance drops at the same time when compared to MSAM. Thus, optimization is improved again while generalization is not, revealing separable mechanisms for test performance improvements of MSAM and NAG. High disturbances compared to the step size at the end of training appear to be crucial for increased generalization. Extensively investigating the effect of SAM/MSAM during different stages of training might offer potential to make SAM even more effective and/or efficient (i.e. by scheduling p to only apply disturbances for selected episodes).
A
73.5
MSAAE
MSAM
97.5
MSAM k/jo nortetyscaling
79.2
NAG
Tialk Aevuuacy
37.0
75.0
36.0
96.0
770
79.5
98.5
Petarbetion Struggh
Pattebliii Stonge
Figure A.1: Test (A) and train (B) accuracy for WideResNet- 16-4 on CIFAR100 for different normalization schemes of MSAM in dependence on p. MSAM without normalization works equally well. If the perturbation is scaled by learning rate y train performance (optimization) is increased while performance (generalization) benefits only marginally.
A.3 Detailed Analysis of Negative Perturbation Direction
A.3.1 Overshooting of Local Minima
As discussed in the main text, negative momentum perturbations lead to loss increase, which in turn results in effective sharpness minimization. This effect caused by overshooting local minima in the momentum direction during the SGD update. We validate this by depicting the ID-loss landscape along the momentum direction 0 (before being updated) calculated on the next iterations batch in Fig. A.2 As can be seen the SGD update from w;-1 to does not approach the local minima on the new batch but instead overshoots and results in negative slope of the momentum buffer direction (i.e. negative cosine similarity to the gradient). Thus, further perturbations (dashed arrow) in the negative momentum direction result in an increase of the loss, allowing for effective sharpness estimations. is important to note that the loss is calculated on the new unused batch and the momentum buffer is not updated yet. After updating the momentum buffer with the new batches gradient to u. the slope of the new momentum direction turns positive again allowing for SGD to behave properly (ef. Appx. A.3.2). Additional analyses validating that overshooting indeed the cause of negative momentum slopes are given in Appx. A.3.2
14

---

0020
Epech - 16
Fpech
Fpoch
Epock 120
MSAJJ
SGD
M441
Fpoch 300
0010
0005
0000
-0.01
0.00
0.01
0.00
Monrettm Reffee Direriion
Figure A.2: Loss change (i.e. sharpness) in momentum buffer direction 1/-1 during optimization. WRN-16-4 16-4 on CIFAR100 averaged over 30 runs. Minima are overshot resulting loss increase for further perturbations in negative momentum direction done by MSAM. Effect occurs for low (constant) learning rate of v 0.01 used here while we use 0.5 in the main manuscript.
We measure the effect of by the negative cosine similarity between the momentum vector u--1 (before being updated) and the gradients. Additionally, to Fig. 2C we make similar observations for all of our studied models and datasets in Fig. A.3
WRN-16-4 CIFAR100
WRN-28-10 CIFAR100
0.03
0103
0.00
0.05
0.05
-0.13
-0.13
Rusning, arau Seniisto
Running urou. (50 Berei
Pee Eremation
The Eversaass
30
75
100
133
150
175
300
10
75
100
150
175
ResNeet-5 CIFAR100
VIT--/322 /22 ImageNet
0.03
0.00
Stuilally (mo)to)
-0.00
-0.0
0.15
0.15
Timaiing mean (50 meneione:
Trrning mear. (10 inerariene
Ps: trension
Re Itention
75
100
150
175
900
10
30
30
Figure A.3: Cosine similarity between momentum vector #:-1 and gradient ge L.B, (we) for different models/datsets. all with cossine scheduler
A.3.2 Overshooting Cause of Negative Momentum Slope
To validate that the effect of negative slope of the momentum direction (i.e. negative cosine similarity to the gradient) is caused by the overshooting of minima, we perform the following experiments depicted in Fig. A.4
15

---

1.0
Iso, LR d.Cee
LRt
6.5
oller apdnie (vs) mi val botchen
Stalakyy
6.6-
Exod vil bands
6.4
6.2
6.0
30
75
100
155
1.00
175
700
Finch
Figure A.4: Cosine similarity between momentum and gradient with and without LR decay (cosine scheduler), after performing the momentum update with the new batches gradient, for random and a single fixed batch sampled from the validation dataset. WRN-16-4 CIFAR100. Running average over 50 iterations.
Only when using learning rate scheduler, the cosine similarity approaches the end of the training. Since the learning rate decays close to zero at the end of the training, minima are not overshot anymore.
After updating the momentum buffer with the current gradient, the slope turns positive with the cosine similarity to the gradient being slightly below 0.5 since the updated momentum buffer e, is a composition of the gradient itself (weighted by μ = 0.9) and the former momentum buffer. This allows for an effective Empirical Risk Minimization (ERM) by SGD despite the overshooting of minima along the not-updated momentum direction.
For the last two experiments, we sample batches from the validation dataset, which are not used for parameter updates, and thus yield gradients independent of the directions used for optimization. We use these to construct an additional pair of gradients and moving average of gradients (as the momentum buffer) and calculate the cosine similarity between these two. Note that the validation batches are sampled additionally to the training samples and are not used in any parameter updates. When sampling only single fixed batch from the validation set for gradient calculation as well as construction of the moving average, the cosine similarity is close (fully correlated). Thus, the parameter updates of the loss landscape (which not correlated the sampled batch), only have minor impact on the gradient change.
However, if new random additional batches sampled from the validation dataset in each iteration, the cosine similarity stays positive, since no parameter updates are performed these directions and thus no overshooting occurs. In the first epochs the correlation the of the moving average and gradients of new sampled batches even increases. We hypothesize that the common optimization direction, when general information is learned in the early stages, causes this effect. In the later stages of training, however, gradient information becomes more sample-specific.
A.3.3 Dependence of Cosine Similarity on Learning Rate Momentum Factor and Batch Size
0.7
0 30001
V 0.001
04
3.002
0-0000
0/00
0-0000
0.3
0.00
0 0.089
V 0.029
0.4
0/06
Stuilelty
0.03
g 0.03
0.00
0003
-0.10
0.13
25
50
100
125
20
30
100
Rgoch
Fpoch
Figure A.5: WRN-16-4 on CIFAR100. Constant learning rate. Moving average over 50 iterations. Negative cosine similarity (and thus overshooting) even for very small learning rates. Runs with large learning rates still show overshooting when training is not possible anymore.
16

---

0.03
0.1
01
0.00
0.3
40.0
0.5
Conta Sintkeity
-0.00
Tel
0.15
0- 0.83
11.00
0.5
-0.25
119
25
501
1000
125
1.50
175
1000
75
50
73
100
125
150
175
700
Fincc
Fpoch
Figure A.6: WRN-16-4 CIFAR100, Constant learning rate. Moving average 50 iterations.
Small Jr, ie., faster forgetting of the momentum buffer, results more overshooting.
68
81
0.03
88 61
65
0.00
65
000
364
354
Datuu Stulledy
R8
00.3
1ett
0.3
1004
0.2
1646
BS 1699
ES
2018
-0225
RS 2008
0.1
00
B8
6141
B8 6151
25
101
2141
125
150
175
20
5132
25
330
2140
220
130
200
BS 5332
Fgoch
Finchh
Figure A.7: WRN-16-4 CIFAR 100. Constant learning rate. Moving average over 50 iterations. While larger batch sizes result in less stochasticity, the cosine similarity is reduced. This supports that the negative slope is property of the wrining-dataset loss landscape and not caused by limited sampling.
A.4
Curvature
We calculated the loss curvature momentum direction, gradient direction and in random direction in Fig. A.8 if training with SGD, SAM and MSAM for WRN-1664 on CIFAR100. For this calculate 2 Hess (. direction vectors e (normalized ||<||2 1) every optimizer steps.
The curvatures in momentum directions are larger than the curvature random direction (which tends towards the mean curvature as amount of parameters increase) for all optimizers and epochs, validating Setting [ in Appx. A. and thus the suitability of momentum directions for sharpness estimation (especially compared to random perturbations; cf. Appx. A.7).
Additionally, the curvature in these directions offers measure for the loss sharpness. Since local minimum of high curvature is approached, all three curvatures increase at the end of the training for SGD. Similarly to Fig. 4 SAM and MSAM reducing the curvature best in their corresponding perturbation direction and MSAM yields lower curvatures than SAM in random directions.
5GD
XS5AM
107
10
RND
10-4
10-5
100 Rpodhs
Figure A.8: Curvature of random directions (RND), momentum (MOM) and gradient (GRAD) for different optimizers.
17

---

A.5 Eigenspectrum Analysis
We calculated the top 5 eigenvalues of the Hessian using the Lanczos algorithm for WideResNet-16-4 trained by SGD, SAM and MSAM on CIFAR-100.
Table A.1: Top
Hessian eigenvalues.
X1
X2
X
A
SGD
15.19
13.79
13.23
9,98
9.23
MSAM
4.15
3.03
2.30
1.88
1.81
SAM
5.18
4.85
4.43
3.72
3.36
While SAM already effectively reduces the eigenvalues of the Hessian (and thus the curvature), MSAM results in even lower Hessian eigenvalues. These findings align with the other sharpness metrics discussed and further support the conclusion that MSAM leads to lower sharpness than SAM (and SGD).
A.6 Feature Rank Analysis
Although MSAM approximates the loss calculation the full training dataset more closely and thus yields lower sharpness (measured as in Fig. and lower curvature (Fig. A.8 RND direction), we do not see positive correlation with generalization compared to SAM. Similarly to other authors (Andriushchenko er al., 2023b Dinh et al.) 2017 Springer et al. 2024), we also affirm, that explanations for the improved generalization of SAM/SSMM go beyond direct causal relations to sharpness minimization. In analogy to Mueller et al. (2023) and Andriushchennko et al. (2023a), we investigated the hypothesis, that the improved generalization of SAM stems from low rank features. For WRN- IRN-1664 trained on CIFAR 100 for features after the last convolutional block (before pooling) found feature ranks of 5019 for SGD, 4775 for SAM and 3791 for MSAM. MSAM reduces the feature rank significantly more than SAM and thus helps to find more distinct and task specific features.
A.7 Random and Last Gradient Perturbations
Instead of the momentum vector v. in MSAM, we also tried to use other perturbations which are independent of the current gradient and thus not bring significant computational overhead, namely
the last iterations gradients 9t-1 (cf. Daskalakis et al. (2018); Lin el al, (2020a)) with positive and
negative sign as well as Gaussian random vectors (cf. Wen et al. (2018)). For each variation, we tested absolute perturbations (Fig. A.9A)
6
ABS
0 |18|
(A.2)
and relative perturbations (Fig. A.93)
8/ul
REL
(A.3)
||ō2o||
with weights w (multiplied element-wise) and, e.g., 5 = - for MSAM.
MSAM provides the only perturbation reaching SAM-like performance without inducing relevant computational overhead.
18

---

A
(MSAA))
0.90
0.50
(5AM)
The Aestrer
0.78
The
0.75
0.77
0.70
10"
10
10
101
Pormeruction
Figure A.9: Random perturbations and last gradient perturbations compared SAM and MSAM. A: Absolute perturbations, B: Relative perturbations, i.e., scaled by we before normalization. All perturbations normalized by L2-norm and scaled by p. WideResNet 16-4 trained on CIFAR100. MSAM always better than other current gradient-independent perturbations.
A.8 Hyperparameter Stability
To show the stability of MSAM and its hyper parameter p. varied the learning rate 7) and the momentum factor μ when optimizing a WRN-16-4 on CIFAR100 for fixed = 2.2 and depict results in Fig. A. MSAM yields stable performance increases compared SGD and NAG over wide ranges of hyperparameters. We made similar observations when comparing SGD and MSAM for different number of epochs (cf. Fig.A.11.
A
SCID
18
0.79
Text: Arstrayy
0.77
KAG
MIAA5
0.022
0.06
0.1
0.22
0.5
0.6
0.7
0,5
0.5
0.96
Lounting Rate
Mausriuu Facloc
Figure A.10: WRN-16-4 on CIFAR100 fixed 2.2. A: Learning rate ablation (log-scale). B: Momentum factor μ ablation.
A.9 Comparison with Same Computational Budgets
We compare MSAM and SAM (and SGD and NAG) when given the same computational budget for WRN-16-4 on CIFAR 100 for a wider range of epochs (up to 1200) in Fig. A..I Iec. running SAM for half the number of epochs compared to other optimizers, resulting in the same number of network passes for all optimizers. MSAM performs similar to NAG (and SGD) for short training times, however, if trained until convergence of SGD/NAG or even longer (overfitting occurs; SGD/NAG results decrease again) MSAM reaches higher test accuracies and overfitting prevented. Due to the additional forward/backward passes SAM performs worse compared to MSAM for limited computational budgets. For long training times MSAM and SAM do not differ significantly. We further support these observations by training VIT-S/32 with MSAM with doubled number of epochs (180) on ImageNet where we reach 70.1 1% test accuracy and thus clearly outperform SAMs 69.1% (cf. Tab. 2.
19

---

SASE
SCDD
Text ArTtacy 12
10a
(Nemenel
Figure A.11: Comparing different optimizers when given the same computational budget. WRN-16-4 on CIFAR100.
A.10 Loss Ascent in Momentum Direction
To validate that the perturbation of the loss results in a loss increase, we show the perturbed and unperturbed loss during training for different learning rates in Fig. A.12
21
Unperturbed
Petutbed
14
12
0.0
0.4
50
1001
Epoch
Figure A.12: Loss before (LB, (w,)) and after (LB, (w, por/ (l)) perturbation in momentum
direction as done by MSAM (p 3) for WRN 16-4 CIFAR100 for different learning rates.
A.11 3D Loss Landscapes
We show loss landscapes after training with SGD, SAM and MSAM for two random flter-normalin perturbation directions as done by Li et al. (018) in Fig. A. The sharpness/loss landscapes show rotational symmetry. SAM and MSAM both reduce sharpness effectively, while MSAM reaches an even flatter minimum.
SGD
SAM
MSAM
Sharpioo 8
Prrinbiin Pc
2 All
2g Rg 2g 20 hottthation
(ill
Perisn
S10 Will
Figure A.13: 2D sharpness landscapes for two random filter-normalicn normalized perturbations (cf. Liet (018)) as shown in Fig. 4C in ID.
A.12 Effects of MSAM/SAM During VIT's Warm-up Phase
We further investigate the effect of SAM/MSAM during warm-up phase in Tab. A.2 As described above, we not apply MSAM during the warmup phase by default (i.e. setting p 0) since doing so, observe drop in test accuracy from 69.1% to 66.1% which is below the AdamW
20

---

baseline. We assume fluctuations of the momentum vector, that determines the perturbation direction for MSAM, to cause instabilities during the warmup phase. A similar effect can be seen for SAM, however, it is less pronounced, so that applying perturbations during the warm-up phase does not thwart SAM hugely. Since we focused on proposing a computationally more efficient variant and not on improving the generalization of SAM in this work, we thus decided to stay consistent with related work and conduct our extensive experiments in Tab. while applying SAM also during the warm-up phase. Nevertheless, ViT models to further improve SAM. We think further investigating effects of p-scheduling for SAM and MSAM is of high interest. E.g., Zhuang et al. (2022) investigated to reduce p during training (contrary to what our findings suggest) by binding it to the learning rate scheduling for SAM and they did not notice benefits. Despite the discussion in Appx. A.2 could not observe analogous effects for ResNets (though we did not study these extensively).
would generally propose to apply SAM only after the warmup phase for
Table A.2: Impact of application of SAM/MSAM during warm-up phase. ViT S/32 on ImageNet. By default MSAM is applied after warmup phase only while SAM is always applied.
AdamW
SAM
SAM (after warmup only)
MSAM
MSAM (during warmup)
67.1
69.2
69.8
69.1
66.1
A.13
Training and Implementation Details
If not stated differently, we calculate uncertainties of mean accuracies by 68% CI estimation assuming Student's t-distribution.
We tuned weight decay and learning rates for our baseline models (SGD/AdamW) and did not alter these parameters for the other used optimizing strategies. We used basic augmentations (horizontal flipping, cutout and cropping) for CIFAR 100 trainings and normalized inputs to mean 0 and standard deviation 1. For ImageNet trainings we used Inception-like preprocessing (Szegedy el al. 2015) with 224x224 resolution, normalized inputs to mean 0 and std and clipped gradients 1.2-norms to 1.0. We used ViT variants proposed by Beyer et al. (2022). A full implementation comprising all models and configuration files is available at http:://tttbb.ccommmmmaabbbbeees performed on up to 4 NVIDIA A100 GPUs.
Experiments were
Table A.3: Training Hyperparameters
CIFAR100
ImageNet
WideResNets
ResNet50
ResNets
ViTs
Base Optimizer
SGD
SGD
SGD
AdamW
Epochs
200
200
100
90/300
Learning Rate
0.5
0.1
le-3
L.R-cceeduler
cos
cos
cos
cos linear warm-up (8 epochs)
Label Smoothing
0.1
0.1
0.1
Batch Size
256
256
1024
1024
Weight Decay
5e-4
1e-3
le44
0.1
A.14 Details on Optimizer Comparison
We report experimental details on the results presented in Tab. in this section. To calculate the speed, we conducted a full optimization on single GPU for each model and dataset combination, normalized the runtime by SGDs runtime and report the average over all runs per combination.
We trained ViT-S/32 on ImageNet for 90 epochs for all models. Further hyperparameters not specific to SAM variants are reported above in Appx. A.13 Due to limited computational capacities and inline with related work, we did not perform runs for multiple random seeds for ImageNet trainings.
Thus, we did not report standard deviations for these runs.
We adapted official implementations of ESAM (Du et al, (202bb) and MESA (Du el al. 2022a) while
no official implementation was available for LookSAM (Liu el al. 2022).
For LookSAM, we fixed the trade-off parameter k = 5 and conducted a thorough search on the
21

---

additional hyper parameter a, since the value suggested for ViTs by the original authors (a = 0.7) was not suitable for our experiments (also see Fig. A.16). We decided to set 0.1, while runs for >> 0.3 did not yield further performance increases. Full hyperparameter search results are reported
Fig. A4.44
ESAM comprises two hyperparameter (7 and (3 that steer the performance/runtine tradeoff which we set to match those of the original paper (i.e. y = 3 = 0.5).
For MESA we tuned the regularization factor À instead of the perturbation strength
Please also note the full p scan results presented the next section (Fig. A.15 and Fig. A. 16
A
B
SISAE
BAM
DJ
D.5
Tese
Teat
1072
11
10"2
50%S
Pertuclation Steomcll
Peitubatio Steeagth
Strungli
Figure 1414: Full results for P and a search for lookSAM (k CIFARI00. Shaded cone: 68% CL. Dots: Random Seeds. A: WideResNet-16-4, B: WideResNet-28-100 10. C: ResNet-50.
A.15 Full Hyper Parameter Search Results
We report our full o-hyperparameter search results in Fig. A.15 Fig. A.16 and Fig, A.17 In consistency with related work, we report results for best p only, in the main text. We sam- pled p with approximately even spacing on logarithmic scale with 6 datapoints per decade, i.e, E O.1.,.0., 0.1, 0.15, 0.22, 0.34, 0.5, 0.67, 1.5, for experiments on CIFAR100 and with data- points per decade, i.e, {..., 0.1, 0.17, 0.3, 0.55. 1, 1.7... on ImageNet for experiments in Sec. We used a slightly denser sampling for the visualizations in Sec. but did not use those results for comparisons against baselines other methods.
While optimal values for vary slightly between models and datasets, we do not observe higher susceptibility changes in p of MSAM compared SAM.
Over all models and datasets we find higher optimal p values for MSAM compared to SAM. Per- turbation vectors are normalized (L2-norm), so we conjecture components for parameters of less importance to be more pronounced for momentum vectors compared to gradients on single batches. For VIT models, find optimal o values to be higher compared to ResNets. If chosen even higher, heavy instabilities occur during training, up to models not converging, limiting performances espe- cially for MSAM. Similar to the observations during warm-up phase discussed above, this effect is more pronounced for MSAM. Notably, MSAM looses most performance against SAM on the biggest ViT models and trained for 300 epochs, when highest values are optimal for SAM. This suggests, that even better performances might be achievable for MSAM if the instability problems are tackled. Strategies do so might include e.g. clipping e or scheduling of p. which we intend to pursue in future work.
22

---

A
50-
ESAM
SAM
Thd hecmmel
Teall Arriney
A85-Ak--odyyB BSS
5AM 31MM-ooll8S BXX
AE-RAM
SCD
XAG
MERA
19
11
10
10a
10
Strouggt MESA X)
Pestutatii Steerggh p
Portciation Steougth
Figure A.15: Full results for p-search for different SAM/MSAM variants CIFARI00. AM- SAM/ASAM refer to Migaiiie-MMMMaaaaaaiiieeeeeeee SAM as in Kwon et al. (202T). For LookSAM we set k and report only the best performing value of a 0.1 (ct. Fig. A.14D. For MESA: A-search results plotted on same axis. Shaded cone: 68% CI. Dots: Random Seeds. A: 16de B: WideResNet-28-10. 10. C: ResNet-50.
AwASAM 0.7
SwASSAM 0.1
ESAM
0.66
(196 up)
0.81
MERA
6.62
0.60
1.0
10a
30
Pertuzbatare Stoonagth p (Jor MEBSA: A)
Figure A.16: Full results for p-search for different SAM/MSAM variants for ViT-S/32 trained for 90 epochs on ImageNet. For MESA: A-search results plotted on same axis.
A
(300)
VIT-8/16 (300)
TeeNNee-300
70
VVT-8/16 (90)
SAM Arlan/W
VIZ-B/12 (90)
(90)
SISAM
SAM
AdonW
Prrselation Streatt
Pesturiation Steeageh P
Figure A.17: Full results for pearch for all models tested on ImageNet (see Tab. 2 A: Vision Transformer (epochs in parentheses). ResNets.
23

---

A.16 Adversarial Robustness
We evaluated the adversarial robustness of MSAM under attacks by PGD (Madry et al. 2018). FGSM Goodiellow et al. (2018) and APGDT Croce and Hein (2020). We test several attack methods on WideRes<Net-16-4 trained on CIFAR- 100. We used default hyperparameters and 0.1 for gaussian noise and 1 1/255 for the other attack methods. Results shown in A.4 were averaged over 3 randomly initialized runs per optimizer. MSAM improves adversarial robustness similarly to SAM. In particular, under an undirected Gaussian noise attack, MSAM slightly outperforms SAM, further supporting its effectiveness in sharpness minimization. These findings are consistent with those of Zhang et al. (2024).
Table A.4: Adversarial Robustness of MSAM compared to SAM and SGD
Gaussian Noise
PGD
FGSM
APGDT
SGD
22.55
24.05
31.80
20.72
MSAM
26.86
28.92
35.46
27.06
SAM
25.77
30.22
35.30
26.08
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
All claims and contributions reflected in the abstract.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses limitations to the studied datasets, models and methodol-
ogy.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
25

---

Justification:
All assumptions are stated clearly and additionally justified empirically.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All hyperparameter sean results are given, code is provided and detailed instructions of the method presented.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
26

---

Answer: [Yes]
Justification: Code can be found in the supplementary material and will be released upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
hetpolliicggiddeeCoooooo SoweeeeeeeooeSbubiicoooe ssionlolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All hyperparameter sean results are given, code is provided and detailed instructions of the method are presented.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: Error bars well statistical significance analyses are given.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
27

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [NA]
Justification: We conduct experiments on different hardware settings, mainly NVIDIA A100 GPUs.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httph:t/////// ptttp:://neurips.oomm http:/ppww.......oo.ccccccccacaa
Answer: [Yes]
Justification:
No ethically problematic experiments are presented.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
No social impact is expected.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
28

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
No models or data is released.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: No licenced assests where used.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
29

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Documented code can be found the supplementary material and will be released upon acceptance.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
30

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: LLM were not used.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy ottp:e////errripc.cc///oootroooe for what should or should not be described.
31


Reviews:
[{"review_id": "yGnX7XhxY2", "reviewer": "Reviewer_xcYq", "summary": "This paper proposes Momentum-SAM (MSAM), a variant of the Sharpness-Aware Minimization (SAM) optimizer. The main idea is to replace the  gradient ascent step with a perturbation with the momentum vector. The empirical insight is that the momentum vector negatively correlated with the batch gradient. Authors use this as proxy for the gradient ascent step in SAM, thus relacing the need for 2 separate gradient calls. The paper provides empirical evidence for models and datasets to validate the approach.", "questions": "1. It would be useful to add an ablation study that isolates the source of the benefit. For example, comparing against a version of MSAM where the SGD step is performed with the raw gradient $g_{MSAM}$ instead of the momentum vector $v_{t+1}$ would clarify whether the performance gain comes purely from the momentum-based perturbation or if the momentum-based weight update is also essential.\n\n\n2. How does the the eigenspectrum of the Hessian look after MSAM when compared to SGD and SAM? Showing that MSAM, like SAM, suppresses the top eigenvalues of the Hessian would provide stronger evidence that MSAM is proxy for SAM and is exploring flat minimas.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\n1. The idea of using the existing momentum vector as a proxy for the gradient ascent step in sharpness-aware perturbation is useful. \n2. Overall paper is well written and authors empirically show strong generalization performance of MSAM when compared to baselines like SGD, and comparable performance to SAM and other variants of SAM. \n\n\n## Weaknesses\n- The momentum vector ($v_t$) is used in two distinct places in the proposed algorithm (Algorithm 1):\n   * In the \"SGD step\" to update the weights (w_t+1 = w_t - η * v_t+1).\n   * In the \"next pert\" step to define the perturbation for the next iteration ($w̃_{t+1} = w_{t+1} - ρ * v_{t+1} / ||v_{t+1}||$).\nIt's not clear where the improved generalization comes from momentum based perturbation, or momentum-based weight is also necessary.\n\n- The momentum vector is negatively correlated with the batch gradient is interesting but an empirical heuristic. I can imagine simple optimization problems where this wouldn't hold. This should perhaps be made clearer in the paper.", "was_revised": true, "final_justification": "Post rebuttal updates:\n\n- Authors isolated the role of the momentum buffer $v_t$​ by removing its use from the weight update and showed that MSAM retains strong performance even when momentum is excluded from the update, highlighting the benefit of momentum in the perturbation step alone.\n\n- Authors conducted a Hessian analysis, computing the top 5 eigenvalues for SGD, SAM, and MSAM and showed that MSAM achieves the lowest curvature, further confirming its effectiveness. \n\nAuthors propose to include these as part of the manuscript and thus I raised my score for the paper.", "rebuttal": "Thank you for your positive and constructive feedback.\n\nWeaknesses:\n1. Thank you for highlighting this observation. We hope that our response to your first question resolves your concerns.\n2. We fully agree that the negative correlation between the momentum vector and the gradient may not hold for many optimization problems beyond deep learning. We will revise the manuscript to emphasize that our proposed algorithm is specifically designed for optimization in the context of deep learning. Additionally, we will clarify that our focus is on  providing a thorough empirical evaluation of SOTA network architectures, rather than on deriving results from a simplified theoretical model.\n\n Questions:\n 1. Thank you for suggesting the additional ablation experiment. In MSAM, the first effect of the momentum buffer $v_t$ in the weight update step is the same as in standard SGD with momentum. We made no changes in this regard and built MSAM on top of SGD with momentum as well as on top of Adam. As you recommended, we conducted an experiment where we isolated this effect by removing the use of $v_t$ from the weight update step. For the baseline optimizer, this corresponds to training with SGD without momentum (i.e. setting $\\mu = 0$). For MSAM we kept $\\mu = 0.9$ to compute $v_t$, but used it only in the perturbation step, not in the weight update. For a WideResNet-16-4 trained on CIFAR-100 we obtained the following test accuracies: SGD $75.96 \\pm 0.19; MSAM: $77.16 \\pm 0.13$. While the performance of SGD drops significantly when momentum is removed, MSAM still delivers substantial improvements. This confirms that the benefit of the momentum-based perturbation in MSAM can be disentangled from the momentum use in the weight update, and still results in significant performance gains.\n2. We calculated the top 5 eigenvalues of the Hessian using the Lanczos algorithm for WideResNet-16-4 trained by SGD, SAM and MSAM on CIFAR-100.\n\n| Optimizer  | $\\lambda_1$  | $\\lambda_2$   | $\\lambda_3$ |  $\\lambda_4$ | $\\lambda_5$ |\n|---|---|---|---|---|---|\n| SGD  | 15.19  | 13.79  | 13.23  | 9.98  | 9.23  |\n| MSAM  |  4.15 | 3.03  | 2.30  | 1.88  | 1.81  |\n| SAM  |  5.18 | 4.85  | 4.43  | 3.72  |  3.36 |\n\nWhile SAM already effectively reduces the eigenvalues of the Hessian (and thus the curvature), MSAM results in even lower Hessian eigenvalues. These findings align with the other sharpness metrics discussed in our manuscript and further support the conclusion that MSAM leads to lower sharpness than SAM (and SGD). We appreciate your suggestion to include this experiment, which we think is a valuable addition to our work, particularly as a complement to the curvature analysis presented in Section A.4, and will incorporate it into the revised manuscript.\n\nPlease let us know if you have any unresolved concerns about our manuscript."}, {"review_id": "4GYVQyuFU9", "reviewer": "Reviewer_Vj9r", "summary": "- This paper aims to improve the efficiency of Sharpness-aware minimization.\n- This paper proposes the MSAM, which mimics the framework of Nesterov Accelerated Gradient, using the momentum vector computed in each iteration to approximate the adversarial weight perturbation.\n- Empirically, this paper conducts experiments on CIFAR and ImageNet to demonstrate the effectiveness of MSAM (faster than SAM and better than SGD). The paper also provides empirical analysis to show the similarity between gradient directions of MSAM and SAM. Theoretically, the paper provides a loose bound which shows the loss of MSAM upper-bounds the expected loss of general perturbation.", "questions": "In terms of performance of MSAM, it highly depends on the perturbation \\rho and learning rate. Can you provide more insights on the coupling effect of the two hyper-parameters?  How to ensure the similarity of momentum vector and graident?", "limitations": "Better theory analysis", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "- The paper is well-written. From method, to experiments, to analysis, they are well-organized\n- The observation is counter-intuitive but also interesting. The similarity between momentum vector and gradient is negative. This observation are supported by empirical results. The performance is comparable with SAM and the efficiency is good.\n\n- (Not a weakness at the stage) The theory analysis is not perfect. I encourage more solid analysis on this observation and more extensive experiments (larger model and diverse dataset). If the observation can be further supported by large-scale experiments and more solid theory, It would be perfect.", "was_revised": true, "final_justification": "Keep score unchanged, boardline accept.", "rebuttal": "Thank you for your positive and constructive feedback.\n\nWhile we agree that a solid theoretical model to explain the effects of MSAM would be valuable, we would like to emphasize the intentional empirical focus of our study.  Our central finding — a negative correlation between momentum and gradient — is not adequately explained by most existing theoretical frameworks. Instead of relying on heavily simplified assumptions that may hinder the transferability of results to real-world scenarios, we chose to conduct a comprehensive and rigorous empirical investigation of SOTA network architectures and tasks. We will clarify this empirical focus, as well as its limitations, more explicitly in the conclusion section of our manuscript.\n\nThank you for highlighting the potential coupling between $\\rho$ and the learning rate $\\eta$. We examined this coupling, and we found that, as illustrated in Figure A.10, no strong interdependence can be observed.\nA fixed value of $\\rho = 2.2$ consistently improves test accuracy across a wide range of learning rates. Additionally, a grid search over $\\rho$ and $\\eta$ did not reveal any significant dependency between the two parameters (not shown in the current manuscript, but we will add these full results to the appendix). \n\nAdditionally,  as shown in Figure A.5, a negative correlation between the momentum buffer and the gradient is consistently observed across the full range of learning rates from $\\eta = 0.001$ to $\\eta = 2.2$. Figures A.6 and A.7 further illustrate that the similarity between the momentum vector and the gradient is a stable phenomenon, occurring independently of both the momentum coefficient $\\mu$ and the batch size.\n\nPlease let us know if you have any unresolved concerns about our manuscript."}, {"review_id": "VDJeOSL7l5", "reviewer": "Reviewer_pE5s", "summary": "This paper proposes Momentum-SAM (MSAM), an optimizer that reduces sharpness (improving generalization) without the extra compute cost of SAM. Instead of gradient-based perturbations, MSAM uses the momentum direction to estimate sharpness, avoiding extra forward/backward passes. Experiments on CIFAR100 and ImageNet show MSAM matches or exceeds the performance of faster SAM variants while being nearly as efficient as SGD.", "questions": "1. For the comparison with LookSAM, which k did you use? For example, LookSAM-5, LookSAM-10?\n2. How is the training speed calculated?\n3. How does this approach affect model robustness to adversarial attacks compared to SAM?\n4. Are there directions to more formally link MSAM to the original SAM objective in terms of an explicit bound, rather than heuristic similarity?", "limitations": "limitations regarding stability on large-scale ViT training, potential underperformance in certain configurations, and the heuristic nature of the theoretical connection to SAM are discussed.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1.  The exposition is clear, including motivating diagrams and an algorithm listing.\n2. All hyperparameter scans, code references, and implementation details are provided.\n\nWeaknesses:\n1. Although well-discussed in the appendix, the intuition behind perturbing in the negative momentum direction could be more concisely explained in the main text.\n\n2. It would be better to report explicit compute time.", "was_revised": true, "final_justification": "Thank the authors for the detailed rebuttal. I appreciate the experiments about adversarial robustness of MSAM. In the rebuttal part, there is something not very clear to me about the training time. And I still hope to see some theoretical connection, since for the paper about SAM or its variants, it is usually carefully proved. I will keep my score.", "rebuttal": "Thank you very much for your overall positive and constructive review.\n\nWeaknesses:\n1. Thank you for considering our detailed appendix. We agree that providing a more concise explanation of the intuition behind perturbing in the negative momentum direction in the main text would improve clarity. We will incorporate it into the main text accordingly.\n2. We originally reported normalized compute times to simplify comparisons across methods with different configurations, but we agree that providing exact compute times offers a more comprehensive view of our specific experiments. Therefore, we will add additional columns to Table 1 reporting the exact compute times per model. For example, the exact times for ViT-S/32 on ImageNet for 90 epochs are: 11h 55m, SAM: 22h 54m, SGD: 11h 48m.\n\n\n Questions:\n 1. We set $k=5$ as it was also done by the original authors. We also experimented with $k=10$, which resulted in a faster training speed but worse test accuracy compared to MSAM. For more details about our experiments with other SAM variants, please refer to Section A.14.\n 2. To calculate the training speed, we first average the runtime per optimizer and task across the 5 randomly initialized runs.  We then normalize these times by dividing the baseline optimizer’s runtime (SGD or AdamW, depending on the model) by the runtime of the optimizer under consideration. This normalization results in a speed value of 1.0 for the baseline. Since the variation of these values across different tasks is minimal (typically around $\\pm 0.02$), we average them across tasks and report a single speed value per optimizer in Table 1 to improve readability. We will add this detailed description to our concise explanation of the training speed calculation in Section A.14.\n 3. Following your suggestion, we evaluated the adversarial robustness of MSAM. We tested several attack methods on a WideResNet-16-4 trained on CIFAR-100. (We used default hyperparameters and $\\sigma=0.1$ for gaussian noise and $\\epsilon=1/255$ for the other attack methods. Results were averaged over 3 randomly initialized runs per optimizer.)\n| Optimizer  | gaussian noise  | PGD (Madry et al.)   | FGSM (Goodfellow et al.) |  APGDT (Croce et al.) |\n|---|---|---|---|---|\n| SGD  | 22.55  |  24.05 | 31.80  |  20.72 |\n| MSAM  |  26.86 |  28.92 |  35.46 | 27.06  |\n| SAM  |  25.77 | 30.22  |  35.30 |  26.08 | \n\nMSAM improves adversarial robustness similarly to SAM. In particular, under an undirected Gaussian noise attack, MSAM slightly outperforms SAM, further supporting its effectiveness in sharpness minimization. These findings are consistent with those of Zhang et al. (2024) , which we will add to the related work section.\nThank you for suggesting this experiment - it aligns very well with our work, and we are pleased to include it in the manuscript.\n\n4. We formally link MSAM to the original SAM objective by showing in Section A.1 that a generalization bound, similar to that of SAM, can be derived for MSAM. While our main focus is on the empirical study, we appreciate the reviewer's interest in the theoretical connection and will highlight it more in the main text.\n\nPlease let us know if you have any unresolved concerns about our manuscript.\n\nCroce et al. \"Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks\" ICML 2020\\\nMadry et al. \"Towards Deep Learning Models Resistant to Adversarial Attacks\" ICLR 2018\\\nGoodfellow et al. \"Explaining and harnessing adversarial examples\" ICLR 2014\\\nZhang et al. \"On the Duality Between Sharpness-Aware Minimization and Adversarial Training,\" ICML 2024"}, {"review_id": "Nm1USpBASU", "reviewer": "Reviewer_Uekb", "summary": "This paper introduces Momentum Sharpness-Aware Minimization (MSAM), a novel optimizer designed to reduce the sharpness of training loss surfaces for deep neural networks without incurring the significant computational and memory overhead of SAM. Unlike SAM, which perturbs parameters along the gradient direction and requires an extra gradient computation, MSAM leverages the accumulated momentum vector for perturbation, achieving sharpness-aware updates with virtually no additional cost over standard optimizers like SGD or Adam. The authors provide a detailed empirical evaluation of MSAM on multiple benchmark datasets and architectures, demonstrating that MSAM achieves generalization performance comparable to SAM while being much more computationally efficient. The paper also offers an in-depth analysis comparing MSAM, Nesterov Accelerated Gradient (NAG), and SAM, and provides new insights into the effects of perturbing in the momentum direction, including the impact of negative momentum perturbations on the loss landscape.", "questions": "1. Although the appendix contains some theoretical analysis for MSAM, the content and proofs are rather brief, and there does not appear to be a convergence proof. I suggest that key theoretical assumptions and results should be presented in the main text, as they are more convincing than purely empirical conclusions.\n2. Since MSAM is easy to implement and incurs computational costs similar to SGD or Adam, why not conduct experiments on different types of tasks, such as natural language processing or large language models, instead of focusing mainly on standard image classification benchmarks? This would help further demonstrate the effectiveness and generalizability of the method across a broader range of application domains.\n3. The font sizes in Table 1 compared to Tables 2 and 3 are quite inconsistent, which affects the readability. It would be better to adjust the layout for consistency. Additionally, in Table 2, why are the baselines for ResNet models always SGD and for ViT models always AdamW? Is there a specific reason for this choice?", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength\n1. MSAM maintains the generalization improvements of SAM but removes its major computational bottleneck, making sharpness-aware optimization feasible even when training resources are limited.\n2. The algorithm is simple to implement and can be seamlessly integrated into existing training frameworks based on popular optimizers such as SGD or Adam.\n3. The authors offer insightful analyses by comparing MSAM, NAG, and SAM, and shed new light on the role of momentum-based perturbations in improving generalization.\n\nWeaknesses\n1. The paper lacks rigorous theoretical analysis or convergence guarantees for MSAM, relying instead on empirical results, assumptions, and intuitive explanations.\n2. Although the experiments are extensive, the study mostly focuses on standard benchmarks and architectures. The performance of MSAM on larger or more specialized tasks remains unclear.", "was_revised": true, "final_justification": "In the rebuttal, the authors explained the empirical focus of their study and supplemented more experimental results of an NLP task. They have fully addressed my concerns, and thus I raised the score.", "rebuttal": "Thank you very much for your positive and constructive feedback.\n\nWhile we agree that a solid theoretical model to explain the effects of MSAM would be valuable, we would like to emphasize the intentional empirical focus of our study.  Our central finding — a negative correlation between momentum and gradient — is not adequately explained by most existing theoretical frameworks. Instead of relying on heavily simplified assumptions that may hinder the transferability of results to real-world scenarios, we chose to conduct a comprehensive and rigorous empirical investigation of SOTA network architectures and tasks. We will clarify this empirical focus, as well as its limitations, more explicitly in the conclusion section of our manuscript.\n\nQuestions:\n1. Since our method is designed to improve generalization, our theoretical analysis in Section A.1 focuses on a generalization bound rather than a convergence proof, in line with the original SAM paper by Foret et al. (2021). We agree that presenting the main theoretical results in the main text will enhance the clarity and impact of our work, and we will update the manuscript accordingly.\n2. We conducted additional experiments in the context of language processing. Due to time constraints, we selected a simple task and model: English-to-Romanian machine translation using the WMT 2016 dataset. We fine-tuned a T5-tiny model (efficient variant by Tay et al.) starting from a publicly available checkpoint pretrained on the C4 dataset. We scanned $\\rho \\in {0.01, 0.03, 0.1, 0.3, 1}$ and found that both MSAM and SAM performed best at $\\rho = 0.1$. The resulting BLEU scores were 23.35 for AdamW, 23.57 for SAM, and 23.64 for MSAM. Notably, MSAM slightly outperformed both SAM and AdamW while requiring approximately half the computational cost. We would be happy to include these results to our manuscript.\n3. Thank you for pointing out the inconsistent font size, we will fix this. Regarding the choice of the baseline optimizers, we used SGD for ResNets and AdamW for ViT, as these give the best performance, and our aim is to compare against the strongest respective baselines.\n\nPlease let us know if you have any unresolved concerns about our manuscript.\n\nTay et al. \"Scale Efficiently: Insights from Pretraining and Finetuning Transformers\" ICLR 2022"}]

Author rebuttal:

Meta-review: This paper introduces Momentum-SAM (MSAM), an optimizer that addresses the computational overhead of Sharpness-Aware Minimization (SAM). The core contribution is the use of the existing momentum vector for parameter perturbation, which avoids the need for a second gradient computation. The authors provide a comprehensive empirical study demonstrating that MSAM achieves generalization performance comparable to SAM with significantly better efficiency.

The primary strength of this work lies in its practicality and efficiency. MSAM offers a powerful solution to a major bottleneck in SAM, making sharpness-aware optimization more accessible. The method is built on a compelling empirical observation—the negative correlation between momentum and gradient, which is well-supported by extensive experiments.

The main weakness noted by reviewers is the lack of a deep theoretical analysis or convergence guarantees. The paper primarily relies on empirical evidence and intuitive explanations. While a generalization bound is briefly mentioned, the theoretical connection to the original SAM objective is not as rigorously proven as in some prior works.

The decision to Accept this paper is based on its significant and timely empirical contribution. The proposed method is highly effective, well-validated, and addresses a crucial, real-world limitation of SAM. The core idea is simple and impactful, giving the method potential for wide adoption. This paper presents a solid contribution to the field of optimization and is well-suited for a poster presentation.

During the rebuttal period, reviewers raised key questions about the source of MSAM's benefits (Reviewer xcYq), its theoretical foundation (Reviewer pE5s), and its generalizability to other tasks (Reviewer Uekb). The authors provided an excellent rebuttal. They conducted additional experiments on Hessian analysis and adversarial robustness, which satisfied Reviewer xcYq. They also provided new results for a language processing task, which convinced Reviewer Uekb. While Reviewer pE5s remained unconvinced on the theoretical front and maintained their borderline reject score, the authors' proactive engagement and the quality of their additional experiments were pivotal in the final decision to recommend acceptance.
Decision: Accept (poster)