Revisiting Sharpness-Aware Minimization: A More Faithful and Effective Implementation
Authors: Jianlong Chen
Zhiming Zhou
Venue: ICLR 2026 Poster (2026)
Keywords: Sharpness-Aware Minimization
Optimization
Generalization
Primary area: optimization
TL;DR: Based on empirical and theoretical analysis, we propose a novel interpretation of a key component of Sharpness-Aware Minimization (SAM) and introduce XSAM to address two limitations revealed by this analysis.

Abstract: Sharpness-Aware Minimization (SAM) enhances generalization by minimizing the maximum training loss within a predefined neighborhood around the parameters. However, its practical implementation approximates this as gradient ascent(s) followed by applying the gradient at the ascent point to update the current parameters. This practice can be justified as approximately optimizing the objective by neglecting the (full) derivative of the ascent point with respect to the current parameters. Nevertheless, a direct and intuitive understanding of why using the gradient at the ascent point to update the current parameters works superiorly, despite being computed at a shifted location, is still lacking. Our work bridges this gap by proposing a novel and intuitive interpretation. We show that the gradient at the single-step ascent point, when applied to the current parameters, provides a better approximation of the direction from the current parameters toward the maximum within the local neighborhood than the local gradient. This improved approximation thereby enables a more direct escape from the maximum within the local neighborhood. Nevertheless, our analysis further reveals two issues. First, the approximation by the gradient at the single-step ascent point is often inaccurate. Second, the approximation quality may degrade as the number of ascent steps increases. To address these limitations, we propose in this paper eXplicit Sharpness-Aware Minimization (XSAM). It tackles the first by explicitly estimating the direction of the maximum during training, and addresses the second by crafting a search space that effectively leverages the gradient information at the multi-step ascent point. XSAM features a unified formulation that applies to both single-step and multi-step settings and only incurs negligible computational overhead. Extensive experiments demonstrate the consistent superiority of XSAM against existing counterparts across various models, datasets, and settings.

Published as a conference paper at ICLR 2026
REVISITING SHARPNESS-AWAR MINIMIZATION: A MORE FAITHFUL AND EFFECTIVE IMPLEMENTATION
Jianlong Chen, Zhiming Zhou'
Key Laboratory of Interdisciplinary Research of Computation and Economics Shanghai University of Finance and Economics chen. j anallon oonnonn@@su.... www..euu.cc......oommoommm edu zhouzhiming@ma suf..edu.co
ABSTRACT
Sharpness-Aware Minimization (SAM) enhances generalization by minimizing the maximum training loss within predefined neighborhood around the parame- ters. However, its practical implementation approximates this as gradient ascent(s) followed by applying the gradient at the ascent point to update the current param- eters. This practice can justified as approximately optimizing the objective by neglecting the (full) derivative of the ascent point with respect to the current parameters. Nevertheless, a direct and intuitive understanding of why using the gradient at the ascent point to update the current parameters works superiorly, despite being computed at a shifted location, still lacking. Our work bridges this gap by proposing novel and intuitive interpretation. We show that the gradient at the single-step ascent point, when applied to the current parameters, provides better approximation of the direction from the current parameters toward the maximum within the local neighborhood than the local gradient. This improved approximation thereby enables more direct escape from the maximum within the local neighborhood. Nevertheless, our analysis further reveals two issues. First, the approximation by the gradient the single-step ascent point is often inaccu- rate. Second, the approximation quality may degrade as the number of ascent steps increases. To address these limitations, we propose in this paper eXplicit Sharpness-Aware Minimization (XSAM). It tackles the first by explicitly estimat- ing the direction of the maximum during training, while addressing the second by crafting search space that effectively leverages the gradient information the multi-step ascent point. XSAM features a unified formulation that applies to both single-step and multi-step settings and only incurs negligible computational overhead. Extensive experiments demonstrate the consistent superiority of XSAM against existing counterparts across various models, datasets, and settings. Code is available at https: /tttps://qithub... http://hhhh..coo////////////////
INTRODUCTION
The success of modern machine learning relics heavily on overparameterization. This necessitates strong regularization, either implicit explicit, from the training procedures (Srivastava et al. 2014
Gidel et al. 2019 Karakida et al. 2023) to ensure generalization beyond the training set (Zhang et al.
2021). In recent years, Sharpness- Aware Minimization (SAM) (Foret al. 2020 Kwon et al. 202 I
Liu et al. 2022b Kim et al.) 2023 Mordido et al.) 2024) has attained significant attention for its
potential enhance the generalization of machine learning models, in a direct optimization manner.
SAM seeks to minimize the maximum training loss within a predefined neighborhood around the parameters, thereby promoting flatter minima and better generalization. Its effectiveness is evidenced by empirical successes across various domains (Bahri et al.) 2021 Rangwani et al.) 2022b a Fan al. 2025). However, its practical implementation approximates this as: carry out one few steps of gradient ascent, and then apply the gradient from the ascent point to update the current parameters.
Though being justified as approximately optimizing the objective by neglecting the Jacobian matrix of the ascent point with respect to the current parameters (Foret et al. 2020), the underlying mechanism
"Corresponding author.

---

Published as a conference paper at ICLR 2026
remains poorly understood. A body of research (Wen et al. 2023 Bartlett et al. 2023 Andriushchenko et al. 2023a Andriushchenko & Flammarion 2022 Andrushchenko et al. (023b) has sought to demystify SAM after such approximations. However, direct and intuitive understanding of why applying the nonlocal gradient at the ascent point to update the current parameter works superiorly is still lacking. This gap necessitates deeper investigation into SAM's fundamental mechanisms, which motivates our work.
Common misinterpretation. A prevalent misunderstanding must be clarified before we proceed: ap- plying the gradient at the estimated maximum point DOES NOT necessarily lead to the minimization of the maximum loss within the local neighborhood. The key here is that there is a shift in location: the gradient is computed at the estimated maximum point, but applied to the current parameters. The nuisance can be clear on considering the extreme case: the gradient at a point arbitrarily distant from the current parameters provides vanishingly little information about the local loss geometry.
To unravel the mystery of the SAM update, we commence by visualizing the local loss surface during SAM training. As shown in Figure further illustrated in Appendix [A] our visualization analysis reveals an important underlying mechanism. Specifically, the gradient at the single-step ascent point, when applied to the current parameters. generally provides a better approximation of the direction from the current parameters toward the maximum within the local neighborhood than the gradient at the current parameters. Therefore, updating the current parameters along the direction opposite to the gradient at the single-step ascent point enables a more direct escape from the maximum. It thereby more effectively reduces the worst-case loss in the neighhorhood, leading improved generalization.
The above interpretation rationalizes the application of the gradient at the single-step ascent point to the current parameters. Nevertheless, our visualizations simultaneously reveal two limitations. First, the approximation by the gradient at the single-step ascent point is often inaccurate (as exemplified in Figure Tap. The approximation quality also unstable, exhibiting large variations as the local loss landscape evolves (evidenced by further visualizations in Appendix (A). Second, as illustrated by Figure Ib (and Figure in Appendix AA). the approximation quality may get worse as the number of ascent steps increases, explaining the unexpectedly inferior performance of multi-step SAM.
Motivated by these observations, we propose in this paper eXplicit Sharpness- Minimization (XSAM), which fundamentally tackles the approximation inaccuracy issue of the SAM gradient by explicitly estimating the direction from the current parameters toward the maximum. This achieved by probing the loss values in different directions at the neighborhood boundary. To ensure its high quality throughout training, XSAM dynamically updates this estimation.
Probing the entire high-dimensional neighborhood for estimating the direction can be computationally intractable. We therefore constrain the probe to a two-dimensional hyperplane spanned by the gradient at the final ascent point (i.e., the point reached after k > ascent steps) and the vector from the current parameters to that point. This definition is crucial. ensures that the point with the highest known loss, i.e., the one pointed to by the gradient at the final ascent point, lies within the hyperplane. Such a definition also simultaneously addresses the inaccuracy issue of directly applying the gradient the multi-step ascent point to the current parameters, while fully leveraging its informational value.
We express the estimated direction in terms of the spherical interpolation factor of the two spanning vectors, which, according our experiments, changes slowly during training. Therefore, it requires only infrequent updates and incurs negligible computational overhead. With this improved estimate of the direction toward the maximum, XSAM escapes the nearby high-loss regions more effectively, thereby achieving better generalization. Extensive experiments demonstrate that XSAM consistently outperforms existing counterparts across various models, datasets, and settings.
The primary contributions of this work are threefold:
We provide a novel, intuitive interpretation of the fundamental mechanism of SAM: the gradient at the (single-step) ascent point offers a superior approximation of the direction from the current parameter toward the maximum within the local neighborhood than the local gradient; thereby, enables a more direct escape from the maximum within the local neighborhood.
Our analysis further reveals that the approximation by the gradient at the single-step ascent point is often inaccurate, and its quality varies largely during training. Moreover, the approximation quality may degrade as the number of ascent steps increases, explaining the inferior performance of multi-step SAM. These collectively demonstrate the sub-optimality of the SAM gradient.
2

---

Published as a conference paper at ICLR 2026
20.00
2.0
0.20
15.00
0.10
0.00
81
0.3
Loss
0.10
01
3,00
0.20
21899t
-2.0
0.00
20.0
-0.10
B.00
0.30
0.30
X
(a) Visualization of single-step SAM
(h) Simulation of multi-step SAM
Figure I: (a) Visualization of the local loss surface of single-step SAM on the hyperplane spanned by the gradient go the current parameter il and the gradient 91 at the single-step ascent point d1. dn is set as the origin, the -axis is defined along the direction of go. and the X-axis is aligned with the component of 91 perpendicular to 9o- The visualized arrows of gradients are set to have length p. We see that gi @g (i.e., 91 applied to θo) points clearly closer to the direction from do toward the maximum within the local neighborhood than go- The targeted direction roughly from origin to the upper-right corner in the figure. The loss along g (i.e., L(Vo- + Pm 91. //lll II) is higher than that along go (i.c., L( + Pm 90/ll90 ()), for sufficiently large Pm- (b) simulation of multi-step SAM get worse as the number of ascent steps increases. grOdd inferiorly identifies the direction from do toward the maximum within the neighborhood (the upper-left high-loss region in yellow) than g g(
2D test function. The approximation quality by the SAM gradient may
We propose XSAM, which addresses these limitations of SAM by explicitly estimating the direction from the current parameter toward the maximum, within novel, principled search space during training. This leads to a more faithful and effective implementation of sharpness-aware minimization. Extensive experiments demonstrate the consistent superiority of XSAM.
2
REVISITING SHARPNESS-AWAR MINIMIZATION
This section reviews the objective of Sharpness-Awaan Minimization (SAM) and its classical approxi- mate optimization method, followed by our novel interpretation of its underlying mechanism.
2.1 THE OВJECTIVE AND CLASSICAL APPROXIMATION OF SAM
SAM (Foret et al. 2020) aims to find parameters that minimize the maximum training loss (i.e., worst-case loss) over predefined p-neighborhood around the parameters. The formal objective is:
min max L(e + 6),
where L is the training loss, E. is the model parameters, and 8 e R the perturbation vector
Since exactly solving the inner maximization in Equation 1 is computationally expensive, SAM approximates by performing one or few steps of gradient ascent from the current parameters.
Assuming the procedure involves à 2 1 successive gradient ascent steps, it proceeds as follows: initialize 90 = 6, and then for each step 1.1.....4 .
1) Compute the gradient at the current point 0;: Ji V.. T((0):; 2) Ascend along the direction of gi by a distance of p:: 0.41 0. + P: # I
This formulation unifies the single-step (k- = 1) and multi-step (k > 1) settings, with the constraint Ek-1 S ensuring the total perturbation remains within the p-bal.. The procedure yields the final perturbed parameters directly as 0z. while approximating the best perturbation 8* as O2 0g-
Data is collected at the first iteration of the 150thh epoch in training ResNet- 18 on CIFAR-100.
For simplicity, we default all norms fz.

---

Published as a conference paper at ICLR 2026
After such approximation of the best perturbation, the SAM objective in Equation reduces to:
min L( + 8*);
equivalently min I(Oa)-
To optimize this objective efficiently, SAM employs key approximation. It assumes N = 0, equivalently, V 02 = I, thereby avoiding involving expensive higher-order derivatives. Formally,
V T( + V = Vo, T(xx) (0) Vo, I(Dx)-
Approximated as identity matrix I
The resulting algorithm essentially applies the gradient at the final ascent point ih& to 8:
0++1 02 T/e V T(0x)-
2.2 A NOVEL INTERPRETATION SAM'S UNDERLYING MECHANISM
Despite the key approximation in the classical SAM algorithm being justified as assuming V d) = I, leads to an unusual gradient operation, applying the gradient at another point (1s) to the current parameters (0). It is apparent that applying the gradient at an arbitrarily distant point to the current parameters makes no sense, since it brings vanishingly little information about the local loss geometry around the current parameters. This contradiction raises fundamental question: How is tx special? Why does applying this nonlocal gradient tend to outperform the local gradient in practice?
While a body of literature has sought explain how SAM works after such approximation (Wen et al. 2023; Bartlett et al. 2023; Andriushchenko et al. 2023a b). they often attribute it to implicit bias or regularization. None of them directly addresses our core inquiry: the underlying mechanism that enables this specific nonlocal gradient operation to be effective, which is the focus of this work.
2.2.1 EMPIRICAL ANALYSIS THROUGH VISUALIZATIONS
To unravel the underlying mechanism, we start by visualizing the gradients at the ascent point on the local loss surface during SAM training. For a tractable analysis and clear comparison between the gradient at the ascent point and the gradient at the current parameters, we focus on the loss surface over the hyperplane spanned by these two gradient vectors. We begin with the single-step setting.
Better Approximation. As depicted in Figure Ta the gradient at the single-step ascent point, when applied to the current parameters, can better approximate the direction toward the maximum within the local neighborhood than the gradient at the current parameters (i.e., the local gradient). More specifically, @ points clearly closer to the high-loss region around the upper-right corner than go. and the loss value along g @ is also literally higher. This phenomenon is consistently observed in practice, shown by additional visualizations in Appendix A
Inaceuracy and Instability. Although @ provides better approximation than Go. we can clearly see Figure Ia that the approximation by gr can still be rough and inaccurate. fact, according to the additional visualizations in Appendix A the approximation quality by 9r is also unstable, exhibiting large variations during training. This suggests that such an approximation by 9r000 can not well adapt to the evolving local loss landscape.
Multi-Step Degradation. We further extend the visualization analysis to multi-step settings. To approximate the complexity of high-dimensional landscapes, where multi-step ascent gradients deviate from a 2D plane, we simulate the process on a suitably complex 2D test function. As shown in Figure bb the gradient at the multi-step ascent point, when applied to the current parameters, may act as an unexpectedly poorer approximation compared to the gradient at the single-step ascent point. Specifically, 92 &oD inferiorly indicates the nearby high-loss region for 9n than Φ1 anh- Notably, Ø2 at its original position 12 indeed points toward the nearby high-loss region; however, when applied to Sn the resulting vector @ points toward relatively flat region. This offers visual explanation for why multi-step SAM does not work as well as expected (Foret et al., 2020 Andriushchenko & Additional simulation results supporting this finding are included in Appendix
Flammarion 2022).
2.2.2 THEORETICAL CONFIRMATION UNDER SECOND-ORDER APPROXIMATION
In this section, we substantiate our empirical observations with the following results:
4

---

Published as a conference paper at ICLR 2026
Proposition 1. Let L R" + R be twice continuously differentialle function that admits second-order approximation at 9u with:
V((0o) 9o. which does not equal to 0:
VL.
(o + P 1oo0
90 91, which is not parallel to go:
.
Hessian H V (((0)) positive definite.
Then there exists Po > 0 such that for all pm > Po:
1) SAM better approximates the direction toward the maximum in the vicinity than SGD
91
90
00 + Pm
L
0o + Pm
1
lgol
2) There exist better approximations than SAM there exists a E IR such that
L 00 + Pm
Jcc
L
do + Pm
Ja = agi + (1 (1-a)go-
The first result in the proposition delivers that for any fixed distance that is relatively large, the loss along the direction of the gradient at the single-step ascent point is higher than that along the gradient at the current parameters. This confirms, from the loss-value perspective, that the gradient of single- step SAM better approximates the direction toward the maximum within its local neighborhood than
that of SGD. Note that a relatively large distance is necessary for the second-order term to dominate the first-order term. For a distance that is too small, 9o by definition the steepest ascent direction.
detailed proof is provided in Appendix B We additionally compare the losses of L(1o + Pm. 91/ l])
and L L + Pm bo/llooll) across different P and Pmn in actual experiments. See Figure 11 and 12 Appendix which provides further empirical evidence of this result.
The second result in the proposition implies that there exist better approximations than the gradient of single-step SAM even in the two-dimensional hyperplane spanned by and 91. This confirms our observation that the approximation by the gradient at the single-step ascent point is often inaccurate.
2.2.3 HEURISTIC EXPLANATION AND DEDUCTIVE ANALYSIS
To help establish a more intuitive understanding of why gr provides a better approximation for the direction of the maximum, we further provide the following heuristic explanation. Assuming the Hessian matrix of the loss function exhibits sufficiently slow variation within the local neighborhood. ie., the gradient field evolves smoothly. Then, if is not parallel to go. the directional change from Go to 91 reveals how the gradient field evolves in the surroundings. Considering additional virtual ascent steps within the local region, e.g., 12 and 92. The directional change from 91 to J2 will tend to follow a similar trend as that from go to 91- The same pattern persists for all subsequent virtual ascent steps, i.e., the virtual ascent trajectory will tend to curve in consistent manner. Therefore, the high-loss region identified by the virtual ascent trajectory will likely be located at position that is further shifted from the one-step ascent point 01 along the direction of 91. but curves further in the evolving direction of the gradient. Its direction relative to 9g is thus better captured by gı L than by Jo- Nevertheless, such an approximation inherently inaccurate.
In multi-step settings, a crucial observation is that each adjacent pair of steps (i. (1.++) recapitulates the configuration of single-step SAM. Consequently, the conclusion from the single-step analysis holds inductively for each step. That is, gi+ @+;; better approximates the direction toward the maximum than g, Av, for directly applies 9k to Do. but it remains unclear whether gr @n stands better approximation of the direction from do toward the maximum than 91 @igy (or even go). The core difference here is that Φ1 is evaluated along the ray defined by 9o and 9g. whereas Ok may substantially deviate from the ray defined by Yu and up. Because the entire multi-step trajectory can curve significantly. This renders the direct application of 9e to tn potentially suboptimal unjustified.
[0...., , 1 1]. However, critical discrepancy arises in multi-step SAM:
As a final remark, a simple deduction reveals the inherent inaccuracy of the SAM gradient approxima- tion: Consider SAM operating fixed loss surface. Regardless of how accurately gi @dg currently
5

---

Published as a conference paper at ICLR 2026
Algorithm XSAM
Table Training time comparison. Values are presented as hours/200 epochs, SAM/ XSAM.
Input: Initial parameters 8g. number iterations T, number of ascent steps &: perturba- tion radius } neighborhood radius Pm. a' update frequency To, learning rate {
CIFAR 10
CIFAR- 100
Tiny ImageNNet
VGG-11
0.93/
0.98/ 1.03
2.18/2.22
ResNet-18 DenseNet-121
2.33
2.40/ 2.43
4,95 /4.98
Output: Final parameters Or
/ 8.08
8.05 8.07
16.50/ 16.55
1: for to T
do
2:
do = 0g
Best Atoha
3:
for i 0 to k do Single-step:
Start Point
End Point.
4:
9i Vo, (0t)
5:
oi 01+1 = 03 + Pt Taal
6:
end for
7:
gh = Va, T((k))
8:
Db 1o dall"
-a
llxxl
ot
1.00
9:
v = arccos( - v1)
-090
0.75
10:
if mod To = 0 then
Value 550T
11:
of arg max. L(to + v(a)),
0.50
100
12:
else
where v(a)= sin((1-a)s Up + (om()))))
sin(()
sl) 01
0.25
160
120
140
14:
ai = =
0.00
200
15:
end if
0.0
1.0
2.0
16:
0.+1 0c Tt v(a5) Iool
end for
Figure 2: Slow variation of a* during training.
approximates the direction, as long as we continuously decrease {pt} } (for all i E [0, k- 1l) toward O, Ie will reduce to Jo- Consequently, the approximation quality of gu @g, will get reduced arbitrarily close to that of the original gradient 9o- This sensitivity to the choice of {p;} also implies that, for an arbitrary {ps). it typically suboptimal (even for a certain fixed loss surface). D On the other hand, we can also tune {p:} } to make it the best possible approximation, which could have played role in the practical effectiveness SAM. Nevertheless, given the evolving local loss landscape during training, the approximation with any fixed {p:} can hardly remain relatively accurate throughout.
3 EXPLICIT SHARPNESS-AWAL MINIMIZATION
As shown in the above section, the approximation by the SAM gradient is often inaccurate and lacks adaptivity to the evolving local loss landscape. Moreover, the approximation quality may degrade as the number of ascent steps increases. To provide integrated solution that simultancously addresses all these limitations, we propose in this section eXplicit Sharpness-ANe Awaree Minimization (XSAM).
XSAM addresses the inaccuracy issue by explicitly probing the location of the maximum within the local neighborhood, thereby providing a more accurate update direction. By dynamically performing this probe during training, it further enhances adaptivity the evolving local loss landscape.
Probing the maximum within the entire high nigh-dimenssioal neighborhood can be computationally intractable. We therefore assume that the maximum is located at the neighborhood boundary, while further constraining the probe to a two-dimensional hyperplane. The 2D hyperplane is spanned by the gradient at the final ascent point (i.e., the point reached after k > ascent steps) and the vector from the current parameters to that point, Formally, the two spanning vectors are defined as:
dx Do
9k
Uy
U1
(5)
10
Poll
[g
This definition of the two-dimensional hyperplane is crucial and provides four key advantages. First, ensures that the point with the highest known loss (the one pointed to by gs. standing at 9z) lies within the hyperplane. Second, it avoids the inaccuracy issue of directly applying the gradient at the multi-step ascent point to the current parameters, while fully leveraging its informational value. Specifically, use 0& and gs define search space that encompasses all the information they contain, instead of directly applying Ie to 0g- Third, it offers a unified formulation for both single-step and multi-step settings. Note that when 1, Up and in correspond the directions

---

Published as a conference paper at ICLR 2026
of gu and 91, respectively. Fourth, normalization is applied to separate direction from magnitude, allowing us to manage them independently.
To probe within the two-dimensional hyperplane, generate directions as the spherical linear interpolation between vpp and U1:
v(a)= = sin((1 - a)e)
20 + sin az)) (in
U1
(6)
sin(1)
where di arocos(vo v1) and a is the interpolation factor. It has [|v(a)] = 1 for any a, c(0) vo. v(1 = V1. More generally, v(a) is a unit vector that rotates from vo by an angle of a along the direction toward f1- It can span all possible directions in the search space.
We then determine the direction, parametrized by a', that maximizes the loss at a predefined distance:
o' arg max L(0o + Pm =((a)): [e(0,,]
where Pm is hyperparametter specifying the radius of the true (in contrast to the perturbation radius) sharpness-aware neighborhood. In each dynamic search, we uniformly sample a values from [0, a]. In practice, setting a to 2 or 4 and sampling 20-40 samples is typically sufficient.
Once a" is identified, the model parameters are updated using -v(a") as the descent direction. The gradient scale, by default, is set to |gx|]| to make it consistent with SAMS Formally,
Oz+1 02 Th- v(a") ||
(8)
by which v(a") steers the parameters away from the estimated maximum within the neighborhood.
Faithfulness and Effectiveness. Since we use L(0p + Pm ( as a proxy F the method explicitly identifies the maximum within neighborhood of radius Pm. Although restricted to a hyperplane, this approximation relies only on the boundary assumption. It thus more faithfully identifies the maximum in the local neighborhood, in contrast directly regarding Or the maximum or approximating its direction by Ob Ld.. XSAM thereby more authentically realizes the sharpness-aware minimization.
The Cost of Explicit Estimation. The evaluation of each requires forward pass. Thus, the cost of explicit estimation scales with the number of sampled a values times the cost of a forward pass. If performed every iteration, this would introduce substantial overhead. Fortunately, frequent updates of a" are unnecessary. Our experiments show that a remains relatively stable and varies smoothly during training (Figure 2). By default, we adopt an epoch-wise update strategy: a* is updated at the first iteration of each epoch and then fixed for the remainder. Runtime comparison is shown in Table I indicating the additional overhead is negligible. Further details are provided Appendix C
4 RELATED WORK
SAM has been extended in several distinct directions. One line of work focuses on improving the gradient ascent (i.e., perturbation) step, addressing issues such as parameter scale dependence
(ASAM (Kwon et al. 202T); Fisher SAM (Kim et al. 2022 ) approximation quality (RSAM (Liu
et al.
20226); CR-SAM Wu et al. 2024)), ), and perturbation stability (VasSO (Li & Giannakis FSAM Li al. 2024)). These approaches largely complementary ours; for instance, E. demonstrates that integrating XSAM with ASAM yields additional performance gains.
2024
Appendix
Another line of research targets the parameter update step. GSAM (Zhuang et al. 2022) combines the perturbed gradient with the orthogonal component of the local gradient. GAM (Zhang et al.) (2023) simultaneously optimizes empirical loss and first-order flatness. In particular, WSAM Yue et al.) 2023) and Zhao et al. (2022a) derive their update rules as linear combination of go and gi through weighted sharpness regularization and gradient norm penalization, respectively. While their superior performance over SAM is readily explained by our interpretation, this very perspective reveals a critical weakness: their dependence on a fixed combination weight, treated as a hyperparameter, is inherently suboptimal. In contrast, XSAM explicitly estimates the optimal interpolation factor
'Alternative gradient scaling strategies are examined in Appendix F OOu implementation uses only the current batch, consistent with the standard SAM procedure.
7

---

Published as a conference paper at ICLR 2026
Table 2: Test accuracies on classification tasks in the single-step setting.
Dataset
CIFAR-I0
CIFAR-100
Tiny-ImageNet
Model
VGG-11
ResNet-18
DenseNet-121
VGG-11
ResNet-18
DenseNes121
VGG-11
ResNee-18
DenseNet-12
SGD
93.19
96.1510.c
96.34_n.11
11.4610.17
18.55
81.78
47.44.0.3
57.02
61.93
SAM
20.00
96.59
=0.97
20.00
80.93
83.81 40.02
41,96
62.81 62.81-0.00
=0.09
66.31 10:00
XSAM
94.25.o.11
96.7410.04
97.15
74.21 10.54
81.24_a.0r
83,96
52.5810.30
63.82 0.25
66.81 10.0w
45,00
82:50
82.50
ResNet-18,
Testt Accuraty
81:84
Test Accuraccy
01:84
11:8
00.52
Start PoRe Shopp Meana
79.86
79.86
5.00
79.20
J
0
(a)
(b)
(c)
Figure 3: (a) Training trajectory comparisons on 2D test function. (b)-(c) Test accuracy comparisons
of ResNet-18 18 trained on CIFAR-100 in single-step and multi-step (k: 3) settings with varying p.
dynamically during training and naturally extends this principle to multi-step settings. More funda- mentally, our approach is derived from a reformulation of the sharpness-aware objective itself, rather than introducing an auxiliary regularization term, offering more general and principled solution.
Multi-step SAM variants are discussed in Section 5.3 while additional related work on topics such as flatness, efficiency, and long-tail learning is deferred to Appendix G
5 EMPIRICAL RESULTS
In this section, we empirically compare SAM and its related variants with the proposed XSAM. Due to space limitations. detailed experimental settings are deferred to Appendix D
5.1 2D TEST FUNCTION
Following ( Yuue et al. 2023 Kim et al. 2022), we first evaluate methods on a 2D function featuring a sharp and flat minimum within certain distance, serving as an ideal testbed for sharpness-aware minimization. We compare SGD, SAM, and XSAM across different initial points and hyperparame- ters. XSAM consistently converges to the flat minima when Pm is sufficiently large, whereas SAM and SGD are more prone to get trapped in the sharp minima. Representative training trajectories for each method shown in Figure Ba| Both SAM and XSAM are evaluated in their single-step form.
5.2 EVALUATION UNDER THE SINGLE- STEP SETTING
In this section, we evaluate the methods under the single-step setting across variety of classification datasets and model architectures. To stress-test the methods, we first tune SAM's S learning rate, weight decay, and o to achieve its optimal performance on each dataset. Other methods are then tuned using the same hyperparameters whenever feasible. To isolate the effect of different gradient directions and eliminate the influence of gradient scaling, all methods adopt SAM's gradient scale, i.e., ||9h||-
We evaluate the methods across diverse neural network architectures and datasets to ensure broad applicability. The experiments cover architectures ranging from VGG-11 (Simonyan & Zisserman 2014) and ResNet-18 (He et al. 2016) to DenseNet-121 (Huang et al.) (2017), encompassing classic models of increasing capacity. The datasets include CIFAR-10 10, CIFAR- 100, and Tiny- ImageNet, which span increasing complexities. As shown in Table 2 SAM consistently outperforms SGD, confirming the superiority of the gradient direction of 9i compared to go- Meanwhile, XSAM consistently outperforms SAM, highlighting the benefit of explicitly estimating the direction.
8

---

Published as a conference paper at ICLR 2026
To provide a more thorough comparison, evaluate performance under varying p on CIFAR-100 using ResNet-18 18, For this experiment, we further include a WSAM-like baseline, which implements method with fixed but tunable a, to highlight the benefit of dynamically estimating a compared to a static choice. The best fixed a for the WSAM is determined via grid search over [-1.0. 3.0) with a step size of 0.25. As shown in Figure 3b the WSAM improves over SAM, while XSAM consistently achieves further and significant improvements over the WSAM.
Having established XSAM's potent performance under varying P. we further assess XSAM's general- ity on larger-scale and more diverse tasks. We conduct experiments on ImageNet with ResNet-50, neural machine translation task with a Transformer /Vaswani et al. (017), and CIFAR- 1-00 with ViT-Ti (Dosovitskiy et al.) (2020). The results in Table 3 that XSAM consistently outperforms SAM, demonstrating its broad applicability across diverse tasks and models.
Table 3: Comparison of SAM and XSAM on larger-scale and more diverse tasks.
Table 4: Multi-step results on CIFAR-100 R00 with ResNet-18, = p² /k with p= for single-step.
ImageNet ResNet-50 (Accuracy)
Transformer IWSLT2014 (BLEU)
VIT-i
Methods
==
==
E: 4
CIFAR-100 100 (Accuracy)
SAM
80.93 + 0.11
80.91 + 0.10
80.65 + 0.26
LSAM
80.93 : 0.11
80.94 1: 0.09
80.74 1 0.18
LSAM+
80.61 - 0.20
80.83
0.11
80.41 = 0.03
SAM
77.04 +: 0.09
35,30 + 0.04
67.80 + 0.22
MSAM
80,93 0.11
81.18
81.01 0.09
MSAM+
80.83 = 0.05
80.86 0.33
80.77 = 0.08
XSAM
77.22 + 0.07
35.63 + 0.12
68.32 = 0.18
XSAM
81.27 0.07
81.44 + 0.09
81.37 = 0.24
5.3
EVALUATION UNDER THE MULTI-STER SETTING
We proceed to evaluate and compare methods in multi-step setting. We use a constant perturbation magnitude p for all steps (i.e., A p for all i), therefore omitting the subscript for clarity. All experiments in this section are conducted on CIFAR- 100 using a ResNet-18.
As the first experiment, we compare XSAM with multi-step SAM variants across different values of k. The considered methods include: MSAM (Kim et al. 2023), which updates parameters with E Di. and LSAM (Mordido et al. 2024 which employs Σ 9 0t/llg/ll- To ensure a thorough comparison, we further introduce two augmented variants that incorporate the initial gradient go: MSAM+ E2 9)) and LSAM+ (E 95/ /l://)) Consistent with our previous protocol, we isolate the effect of gradient direction by readjusting all gradients to have the norm [[gk]] ll- The perturbation radius is set to p p² /k. where p" is the optimal value for single-step SAM, as suggested by Kim ct al. (023)); all other hyperparameters remain unchanged from the single-step setup.
As shown in Table the performance of SAM tends to decline k increases. This phenomenon can be attributed to the growing deviation of gb from the original ascent direction go big as the single ascent step is subdivided. As result, when applied to dg. it leads to poorer approximation of the direction toward the maximum in the vicinity. In contrast, XSAM is not affected by this issue and typically benefits from more steps, demonstrating its superior ability leverage multi-step ascent.
LSAM and MSAM, which incorporate intermediate ascent gradients (g; for 0 < i is k), generally surpass SAM. The decline in SAM's performance with large A suggests substan- tial deviation of gt from the ideal direction, which makes earlier, less-deviate gradients 9; valuable. Notably, LSAM+, which essentially moves away directly from the identified maximum point by multi-step ascent, even underperforms SAM. This highlights the value of an extra explicit estimation of the direction toward the maximum. Nevertheless, XSAM consistently outperforms all these methods across all settings.
02.00
CIFAR-100, BesNet-18, k=1
81.64
Test Accuracy
80.92
80.56
-
XSAN ALCUIOKY SAM Accuraty
Unee bpsc
80.20
10a
10a
10*
Update period of best
We further evaluate SAM and XSAM under multi-step setting (k- = 3) with a varying perturbation radius p. A multi- step extension of WSAM, which combines the gradients 9k and 9n with a fixed interpolation factor, is also compared. The results in Figure Be indicate that while the WSAM variant outperforms SAM, XSAM consistently outperforms WSAM.
Figure 4: XSAM robustness to the a" update frequency.

---

Published as a conference paper at ICLR 2026
Figure shows the robustness of XSAM the a" update frequencies. We observe consistent pattern in performance when varying the update frequency of a". Additional ablation results further visualizes the loss surface convergence,
presented in Appendix E.4 Appendix furthrr illustrating that XSAM finds flatter minima than SAM.
6 CONCLUSION
In this paper, have studied the underlying mechanism of SAM and provided novel, intuitive explanation of why it valid and effective to apply the gradient at the ascent point to the current parameters. We have shown that the SAM gradient in its single-step version can provably better approximate the direction of the maximum within the local neighborhood than that of SGD. We have further demonstrated that such approximation local loss landscape, and may degrade as the number of ascent steps increases. To address these limitations, have proposed XSAM that explicitly and dynamically estimates the direction of the maximum within the local neighborhood during training. XSAM thereby more faithfully and effectively moves the current parameters away from high-loss regions. Extensive experiments across various models, datasets, tasks, and settings have demonstrated the effectiveness of XSAM.
be inaccurate, lacks adaptivity the evolving
ACKNOWLEDGMENTS
This work was supported in part by the National Natural Science Foundation of No. U22B2020.
China under Grant
REPRODUCIBILITY STATEMENT
We have provided the code as supplementary material, along with detailed instructions for reproducing our experiments. The experimental settings and hyperparameters are described in the Appendix D The datasets used in this paper are publicly available and can be downloaded online. Detailed proofs of the proposed proposition are included in the Appendix B
REFERENCES
Maksym Andriushchenko and Nicolas Flammarion. Towards understanding sharpness-aware mini- mization. In International Conference on Machine Learning, pp. 639-668. PMLR, 2022.
Maksym Andriushchenko, Dara Bahri, Hossein Mobahi, and Nicolas Flammarion. Sharpness-aware minimization leads to low-rank features. Advances in Neural Information Processing Systems, 36: 47032-47051, 2023a.
Maksym Andriushchenko. Francesco Croce, Maximilian Müller, Matthias Hein, and Nicolas Flam- marion. A modern look at the relationship between sharpness and generalization, 2023b. URL
httts::/arxiv.. org/ abs 2302. 07011
Dara Bahri, Hossein Mobahi, and Yi Tay. awaree minimization improves language model generalization. arXiv preprint arXiv:2110.08529, 2021.
Peter L Bartlett, Philip M Long, and Olivier Bousquet. The dynamics of sharpness-aware minimiza- tion: Bouncing across ravines and drifting towards wide minima. Journal of Machine Learning Research, 24(316):1-36 2023.
Mauro Cettolo, Jan Niehues, Sebastian Stüker,, Luisa Bentivogli, and Marcello Federico. Report on the 11th iwslt evaluation campaign. In Proceedings of the 11th International Workshop on Spoken Language Translation: Evaluation Campaign, pp. 2-17, 2014.
Pratik Chaudhari. Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-sgdt Biasing gradient descent into wide valleys, 2017. URL crttps://ant wwwp://.g////aot.........o...e/t 61 01838
10

---

Published as a conference paper at ICLR 2026
Terrance Vries Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04555 04552, 2017.
Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. /929, 2020.
Jiawei Du, Hanshu Yan, Jiashi Feng, Joey Tianyi Zhou, Liangli Zhen, Rick Siow Mong Goh, and Vincent YF Tan. Efficient sharpness-aware minimization for improved training of neural networks. arXiv preprint 2r.0v:2/00.0314/ 2021.
Chongyu Fan, Jinghan Jia, Yihua Zhang, Anil Ramakrishna, Mingyi Hong, and Sijia Liu. Towards IIm unlearning resilient relearning attacks: A sharpness-aware minimization perspective and beyond, 2025. URL
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-awar minimization for efficiently improving generalization. In International Conference on Learning Representations, 2020.
Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization via hessian eigenvalue density, In International Conference Machine Learning, pp. 2232-2241. PMLR, 2019.
Gauthier Gidel, Francis Bach, and Simon Lacoste-Julien, Implicit regularization of discrete gradient dynamics in linear neural networks. Advances in Neural Information Processing Systems, 32, 2019.
Shreyank N Gowda and David A Clifton. Cc-sam: Sam with cross-feature attention and context for ultrasound image segmentation. European Conference on Computer Vision, pp. 108 124. Springer, 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Simplifying neural nets by discovering flat minima. Advances in neural information processing systems, 7, 1994.
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural computation, 9(1):1-42, 1997.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Av- eraging weights leads to wider optima and better generalization. arXiv preprint (.X.i:1880.00444 2018.
Stanislaw Jastrzebski, Maciej Szymczak, Stanislav Fort, Devansh Arpit, Jacek Tabor, Kyunghyun Cho, and Krzysztof Geras. The break-even point optimization trajectories of deep neural networks, 2020. URL
abwp:///bs//000 09572
Weisen Jiang, Hansi Yang, Yu Zhang, and James Kwok. An adaptive policy to employ sharpness aware minimization. arXiv preprint MxXi:23304.146477 2023.
Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization measures and where to find them. arXiv preprint arXiv: 1912.0277,,, 2019.
Ryo Karakida, Tomoumi Takase, Tomohiro Hayase, and Kazuki Osawa. Understanding gradient regularization in deep learning: Efficient knite-difference computation and implicit bias. In International Conference Machine Learning, pp. 15809-15827. PMLR, 2023.

---

Published as a conference paper at ICLR 2026
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batell batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836 2016.
Hoki Kim, Jinseong Park, Yujin Choi, Woojin Lee, and Jaewook Lee. Exploring the effect of multi-step ascent in sharpness-aware minimization. arXiv preprini arXiv:2302. 2023.
Minyoung Kim, Da Li, Shell X Hu, and Timothy Hospedales. Fisher sam: Information geometry and sharpness aware minimisation. International Conference on Machine Learning, pp. 11148 11161. PMLR, 2022.
Jungmin Kwon, Jeongseop Kim, Hyunsco Park, and In Kwon Choi. Asam: Adaptive sharpness-awar minimization for scale-invariant learning of deep neural networks. In International Conference on Machine Learning, pp. 5905-5914 PMLR, 2021.
Bingeong Li and Georgios Giannakis. Enhancing sharpness-aware optimization through variance suppression. Advances in Neural Information Processing Systems, 36, 2024.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances neural information processing systems, 31, 2018.
Sicong Li, Qianqian Xu, Zhiyong Yang, Zitai Wang. Linchao Zhang, Xiaochun Cao, and Qingming Huang. Focal-sam: Focal sharpness-aware minimization for long-tailed classification. arXiv preprint arXiv:2505.0060 2025.
Tao Li, Pan Zhou, Zhengbao He, Xinwen Cheng, and Xiaolin Huang. Friendly sharpness-awann aware minimization. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 5631-5640, 2024.
Yong Liu, Siqi Mai, Xiangning Chen, Cho-Jui Hsieh, and Yang You. Towards efficient and scalable sharpness-aware minimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12360-12 12370, 2022a.
Yong Liu, Siqi Mai, Minhao Cheng, Xiangning Chen, Cho-Jui Hsieh, and Yang You. Random sharpness-aware minimization. Advances in Neural Information Processing Systems, 35:24543- 24556, 2022b.
Chao Ma and Lexing Ying. On linear stability of sgd and input-smoothness of neural networks. Advances in Neural Information Processing Systems, 34: 14116555-6617, 2021.
David A McAllester. Pac-bayesian model averaging. In Proceedings of the twelfth annual conference on Computational learning theory, pp. 164-170, 1999.
Peng Mi, Li Shen, Tianhe Ren, Yiyi Zhou, Xiaoshuai Sun, Rongrong Ji, and Dacheng Tao. Make sharpness-awary aware minimization stronger: A sparsified perturbation approach. Advances in Neural Information Processing Systems, 35:30950-30662, 2022.
Gonçalo Mordido, Pranshu Malviya, Aristide Baratin, and Sarath Chandar. Lookbehind-sam: steps back, 1 step forward, 2024. URL httttttt///xxxii/ www.wawwwwwww..aaoooooooooooooot abss 2307.
Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring generaliza- tion in deep learning. Advances in neural information processing systems, 30, 2017.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng. David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations, 2019.
Harsh Rangwani, Sumukh K Aithal, Mayank Mishra, Arihant Jain, and Venkatesh Babu Radhakrish- nan. closer look at smoothness in domain adversarial training. In International conference on machine learning, pp. 18378-18388 PMLR, 2022a.
Harsh Rangwani, Sumukh K Aithal, Mayank Mishra, et al. Escaping saddle points for effective generalization on imbalancdd data. Advances in Neural Information Processing Systems, 35: 22791-22805, 2022b.
12

---

Published as a conference paper at ICLR 2026
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for scage-scale image recognition. arXiv preprint arXiv:1409.1556 2014.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way prevent neural networks from overfitting. The journal of machine learning research, 15()):1929-198, 2014.
Behrooz Tahmasebi, Ashkan Soleymani, Dara Bahri, Stefanie Jegelka, and Patrick Jaillet. A universal class of sharpness-ave aware minimization algorithms. arXiv preprint arXiv:2406.0388 03682, 2024.
Chengli Tan, Jiangshe Zhang, Junmin Liu, Yicheng Wang, and Yunda Hao. Stabilizing sharpness- aware minimization through simple renormalization strategy. Journal of Machine Learning Research, 26(68):1-35, 2025.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Kaiyue Wen, Tengyu Ma, and Zhiyuan Li. How does sharpness-aware minimization minimize sharpness?, 2023. URL
Tao Wu, Tie Luo, and Donald C Wunsch II. Cr-sam: Curvature regularized sharpness- aware minimization. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 6144-6152, 2024.
Yun Yue, Jiadi Jiang, Zhiling Ye, Ning Gao, Yongchao Liu, and Ke Zhang. Sharpness-aware minimization revisited: Weighted sharpness as a regularization term. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 3185-3194, 2023.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115, 2021.
Xingxuan Zhang, Renzhe Xu, Han Yu, Hao Zou, and Peng Cui. Gradient norm aware minimization seeks first-order flatness and improves generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20247-20257, 2023.
Yang Zhao, Hao Zhang, and Xiuyuan Hu. Penalizing gradient norm for efficiently improving generalization in deep learning. In International conference on machine learning, pp. 26982- 26992. PMLR, 2022a.
Yang Zhao, Hao Zhang, and Xiuyuan Hu. Randomized sharpness-aware training for boosting computational efficiency in deep learning. arXiv preprint arXiv:2203.0998 2022b.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmenta- tion. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 13001-13008, 2020.
Yixuan Zhou, Qu, Xing Xu, and Hengtao Shen. Imbsam: A closer look at sharpness-aware minimization in class-imbalanced recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11345-11355, 2023.
Zhanpeng Zhou, Mingze Wang, Yuchen Mao, Bingrui Li, and Junchi Yan. Sharpness-awand mini- mization efficiently selects flatter minima late in training, 2025. URL httl:/tts///rxxiv.orrgg
abs/2410.10377.
Juntang Zhuang, Boqing Gong, Liangzhe Yuan, Yin Cui, Hartwig Adam, Nicha Dvoreek, Sekhar Tatikonda, James Duncan, and Ting Liu. Surrogate gap minimization improves sharpness-awer training, arXiv preprini arXiv:2203.08065 2022.
13

---

Published as a conference paper at ICLR 2026
APPENDIX
A VISUALIZATION OF LOSS SURFACE DURING TRAINING
(a) Epoch
(b) Epoch
(c) Epoch
(d) Epoch
Figure 5: Visualization of loss surface during training: VGG-11 trained CIFAR-100.
(a) Epoch 50
(b) Epoch 100
(c) Epoch 150
(d) Epoch 200
Figure 6: Visualization of loss surface during training: ResNet- 18 trained on CIFAR-100.
-33
(a) Epoch 50
(b) Epoch 100
(c) Epoch 150
(d) Epoch 200
Figure 7: Visualization of loss surface during training: ViT-Ti trained on CIFAR-100.
(a) Epoch 50
(b) Epoch 100
(c) Epoch 150
(d) Epoch
Figure 8: Visualization of loss surface during training: ResNet- 18 trained on CIFAR-10.
-22
(a) Epoch 50
(b) Epoch 100
(c) Epoch 150
(d) Epoch 200
Figure Visualization of loss surface during training: DenseNet-12 -21 trained on CIFAR-I0.
In this section, we provide more visualizations of the loss surfaces different datasets and models during SAM training. The results shown in Figure
and TO The gradient of the ascent
14

---

Published as a conference paper at ICLR 2026
22.00
9.29
6
16.09
1.0
Loss Value
0.00
0g
0.8
0.9
4.13
6:
-1.0
0.6
-.20
0.00
0.3
2.00
4.00
10.00
16.00
5r04
22.00
-2.
X
(a) Visualization of single-step SAM
(b) Simulation of multi-step SAM
Figure 10: (a) Visualization of the local loss surface of single-step SAM. The visualization procedure follows the same steps as in Figure Ta Data is collected at the first iteration of the 100th epoch in training ResNet-18 18 on CIFAR-I00. We see that gr@dd (i.e., 91 applied to do) points clearly closer to the direction from do toward the maximum within the local neighborhood than go- he targeted direction is roughly from the origin to the upper-right corner in the figure. The loss along grath (i.e., L( Pm 91/ /l/92 []) is higher than that along 9u (i.e., L(0,- + Pm 9a/ (| )), for sufficiently large Pm- (b) A simulation of multi-step SAM on test function. The gradient at the multi-step point, when applied the current parameters, may be inferior approximation of the direction toward the maximum.
point better approximates the direction toward the maximum within the neighborhood than the local gradient. However, the approximation can often be inaccurate and unstable during training.
CINA
p0-11
210
10
00. 00
00
(a) Epoch 50
(b) Epoch 100
(c) Epoch 150
(d) Epoch 200
Figure 11: Visualization of I(Uo + Pmn (oen//lllll I.(00 Pmn (ool)) during training.
(a) Epoch 50
(b) Epoch 100
(c) Epoch 150
(d) Epoch 200
Figure 12: Visualization of L(Uo + Amgg//l9 L(dn Pm gool) during training.
We compare L(Vo + Pm91//l91l) and L(Vo + Pmggo |[ool]) across different and Pm in Figure II and [2] We gradually increase Pm for each p. while keeping On and go fixed. As can be seen, Lúd P /rm.9//19911/ Il) becomes larger than L(0, + Pm90/ Pm99//l99ol] when Pm is relatively large. This provides further evidence for our claim that along one can find a higher loss than long go for Uh.
B PROOFS
Proposition 1. Let : + R be a twice continuously differentiable function that admits a second-order approximation at Og with:
15

---

Published as a conference paper at ICLR 2026
VL(0b) 9u. which does not equal to 0:
VL (Do+ + P |[soll 91. which is not parallel to 9u:
9o
Hessian H = ((Do) positive definite.
Then there exists po > 0 such that for all pm > Poi
1)
SAM better upproximates the direction toward the maximum in the vicinity than SGD
( 00 + Pm
91
90
L
do + Pm
1991
1
2) There exist better approximations than SAM there exists or E IR such that
L
(o
Pm
1
Jc
L
do + Pm
191
91 )
Ja agi (1 a)go-
B.1 PROOF OF THE FIRST CONCLUSION
Proof.
Since admits a second-order approximation at 0g:
P Hg1
L
0n
+ Pm
91
L(0o) Pm
91
+
+ o(pm))
|191
1991
11
0n
Pm
9u
L(0o) Pm 11ooll
Pnn
90 H
+ p(
|190|]
2
|
2. For sufficiently large Pm, the pm term dominates. Thus, we need to show:
9 Hg
96 Hgo
||
||ga1|2
Expand I1 as the gradient of (which admits a second-ordes approximation) at 1o + P 8o
[|
90
go +
pHl
+ o(p).
|g
4. Compute the numerator and denominator to the second order:
91 Hgn
+ pH
90
+
+o(p)
H
9o + pH
9o
+ o(p)
|
Toll
9n H go
90 H
9 1190 + 2p
+ o(p²),
11
1.
19 11 =
+ pH
90
+ d()
= |1
2 + 2p
90 Hgo
+ p²
90 H²
+ o(p²).
[90
190 2
Ignoring higher-order terms o()²), the inequality becomes:
90 Hgo + 2p
H go Isoll
0 go H go l
96 H
| 1 + 2p
9o H
+ p
Mn
00
11 1
1
16

---

Published as a conference paper at ICLR 2026
5. Multiply both sides by the positive denominators (since H is positive definite):
90
1
go
+ p² 90 H
go
|1 > 9n Hgu (1000l 2 + 2p T Hgo
+ p² 90 go
90 Hgo + 2p
|190|
1100 1
1901
900 1
Cancel common terms and divide by p 0:
2 [18oll00 91
(00 Hgn)
90 11 90 Hgogd H²90
> 0.
|190| ||
11
Term verification:
First term:
1/9011900 H² (90 H > 0.
This follows from the strict Cauchy-Schwwe inequality for the inner product, since do and Ho are not parallel by assumption.
Second term:
|1 90
H go
90 Hgogo H go 0.
Let H = -E Aoaay be the spectral decomposition with X; 0. Expressing 9o = -E Octyy
1190| 129. "9 90 H9990 1 go
Σ a.
Σ xaa)
{Σ Aaa
{Σ Xaa
The nonnegativity follows from Chebyshev's sum inequality applied to the series ():) and {{?).
Conclusion:
Since both terms are non-negative and the first is strictly positive, the inequality holds.
For sufficiently large Pms the pen dominates the Taylor expansion.
That is, 3po > 0 such that Vpm > Po:
L 0o + Pm [| )
I (No + Pm ||g0
Remark. If Pm is too small, the first-order term will dominate. The first-order term has
g [
= |lool cos( < ||gol]-
where cos
80 g1 [[oo In
< 1 since and gi do not parallel. Thus, if Pm is too small, it will have
L (06 + Pm
1
91
L
00
Pm
[[go
90
From another perspective, this must hold because go indicates the steepest ascent direction at do-
Remark. Pm needs to be large only to ensure that the difference in the second-order term outweighs the first-order term, intended be large to become impractical in real-world applications.
17

---

Published as a conference paper at ICLR 2026
B.2 PROOF OF THE SECOND CONCLUSION
Proof.
Since I admits a second-order approximation at 0o:
L
00
+ Pm
9a
= L((o) + Pm
90 9cc
Pm
9n H9o
+ o(p
1
g ||
2
9. 2
L
Pm
91
L(0u) + Pm
90 91
T
Pm
91
Hg1
P
19
1
2
99
1|
Define the quadratic ratio:
f(a) 91 Hga
11
At boundary points:
((1) 91 Hg
S(0) 90 Hgo
1
3. The derivative is:
f((a) 2(g1 go) Hg |19a1 2(9 H9a -0o) 9a
l9 1
a 1:
((11 [9 l
[[9] - _0) Hg |1 - (g) Hgi)(n g0) mn]
4. Using 01 9o pH 90 Tsoll
+ o(p):
DH |loul|
+ o(p).
Substituting into f'(1):
2p
(((1)
990 H2 91 || 2 (9] Hg) Hgn) t(p))
11
Further substituting 91
go
+ pll
Tsll o(p) in:
|| = 90 + p!l
90
+ o(p)
=11oo| 12 + 2p
90 Hg
+ p
9o H² go
+ o(p²). y0 12
|000|
|
90 H² 1/99ll9111
90
T 12 go +
9n H gu
+ +(())
[1900]2 + 2p 9n H
+
2 90 H gu
+ o(p²)
Iloll
|l9o||
1190|
9u H² gooolllool 2 p 2 90 H² 90
9o H
|1go 11 1130 go
[1 |l
+ p²
9n H2 gu
9n H² gu
+ 9
H 90
9n
Hgu
+ o(p3)
|2
|0 2
( Hgg) Hg1)
90 H2 H290 go
+ p² 90 H H go
+ o(p3) )
go H2
90 Hgo + 2p
90 H +
+ o(p)
lgoll
|1
I19ull
(o Hgn)² 2 p
3 9o
Hgo 90 2 g0
p²
90 H yo go H (g) H²
((gn)) gn)))
") o(2))
||
9o
119011
18

---

Published as a conference paper at ICLR 2026
6. Combining terms:
00 gillll (o) H9:) Hg1) (90 H 9ol19011 (90 Hgo)²)
90 go 90 H² 90
+
[1901190 H 90
1
90 H go 90 H + 0²
(9g 112 go)2
+ o(12).
|
7. Sign analysis:
Zero-order term 9 H gol190 12 -9 Hgo)² Strictly positive by Cauchy- Sehhh--S inequality since H is positive definite and and go are not parallel.
First-order term |190|1²gh 92 4 go 9u go 96 1f2go: Non-negative by Chebyshev's sum inequality for the sequences } and {*² where H Σ Nopee
Second-order term 90 H go 90 Hgo (97 H2 go) Non-negative by Chebyshev's sum inequality.
8. Conclusion:
The term is strictly positive, which means (1) > 0. So, there exists a > 1 such that f(a) > f(1). For sufficiently large Pms, where the second-order term dominates, this further implies:
L 0o + Pm 11gall
Ya
L
do Pm
91
1
Remark. The Pm threshold exists only to ensure the second-order term dominates the first-order term. practice, moderate values suffice to observe XSAM's advantage over SAM.
Remark. Practically, the loss surface may not admit second-order approximation, and the maxi- mum does not necessarily around a = So we search a relatively large range of a, e.g., in [0, 2]. make it more generally applicable. Additionally, we use spherical linear combination instead, for a more uniform distribution of searched directions and better coverage.
C COMPUTATIONAL OVERHEAD
The evaluation of each a will involve a forward pass of the neural network for calculating L(0p v(a:). Pm)) ). So, the cost of the dynamic search of a* roughly equals the number of samples of a times the cost of a forward pass. Typically, we use 20~40 samples to search for a.. If this were required at every iteration, it would incur considerable computational burden. Fortunately, frequent updates of a" are unnecessary. According to our experiments, a fairly stable and changes smoothly during training, depicted in Figure 2 and Figure 13 In experiments, we by default adopt epoch-wise update strategy: a* is updated at the first iteration of each epoch and then kept fixed for the rest. Each epoch typically contains over 400 iterations. SAM requires per iteration. So, the computational overhead of XSAM is roughly 40/ 40/(400 (400- 2 -k + 1)) S 0.025, i.e., the increased cost is typically A straightforward comparison of runtimes is presented in Table The runtime of XSAM is nearly identical that of SAM, indicating that the additional computational overhead is negligible.
1 forward and & + backward passes
than 2.5% when compared to SAM, which is negligible.
D ADDITIONAL EXPERIMENTAL DETAILS FOR RESULTS IN SECTIONS 5
D.1 DETAILS ABOUT THE 2D TEST FUNCTION
The test function used is defined by:
L(0) = L(u, o) = - logg (0.7e -K.6o))i(4)/11.8 0.3e - K2(p.o)/1.22
19

---

Published as a conference paper at ICLR 2026
where Ke(H,,) is the KL divergence between two univariate Gaussian distributions,
o) = log 0² + (1 p4)
(10)
20
2
with (1) (20, 30) and (1/2 02) (-20, 10). It features a sharp minimum at around (-16.8, 12.8) with a value of 0.28 and flat minimum at around (19.8, 29.9) with value of 0.36.
The visualized training trajectories in Figure 3a share the same start point (-6.0, 10.0) and rum for 400 steps. The learning rate is 5 (the gradient scale is small), momentum is 0.9, is 6.0, and pm. is 18.0. The points passed at each step were recorded to plot the trajectories.
D.2 EXPERIMENT SETUP
CIFAR-10, CIFAR-100, and Tint- Metteem We use RandomCrop and CutOut (Dees 2017 augmentations for CIFAR-IO and CIFAR-100 while using BandomResizedCrop and RandomErasing (Zhong et al. 2020) augmentations for LinyImmgeNet, since we believe improvements over strong augmentations can be more valuable. We use a batch size of 125 for all the datasets, such that the sample size of each dataset is divisible by the batch size, while near the typical choice of 128. We adopt the typical choice, SGD with a momentum of 0.9, as the base optimizer, which carries the true gradient descent to 0. All models are trained for 200 epochs, while the cosine annealing learning rate schedule is adopted in all settings.
We run each experiment 5 times with different random seeds and calculate the mean and standard deviation. Each experiment was conducted using a single NVIDIA Tesla V100 GPU.
ResNet50 on ImageNet. We evaluate our method on the larger dataset, ImageNet. Standard data augmentation techniques are applied, including resizing, cropping, random horizontal flipping, and normalization. We take SGD as the base optimizer with a cosine learning rate decay.
IWSLT2014. We conduct experiments on the Neural Machine Translation (NMT) task, specifically German English translation on the IWSLT2014 dataset (Cettolo al., 2014), using the Transformer architecture following the FAIRSEQ (Ott et al.) 2019). We use Adam as the base optimizer due to its better performance on the transformer.
VIT-Ti. We further use lightweight Vision Transformer (ViT-Ti) model on CIFAR- 100 to evaluate our method. Note that following (Zhao al.) [022a], we do not use Cutout augmentation for CIFAR-100 when trained by ViT-Ti. We use AdamW the base optimizer.
D.3 HYPERPARAMETER DETAILS
Table 5: Hyperparameter details for Results in Table
CIFAR-I0
CIFAR-100
Tiny-ImupeNet
XSAM
SGD
WSAM
XSAM
Epuch
200
200
200
Batch size
125
125
125
Initial learning mie
0105
0,05
0.05
Momentum
0.9
0.9
0.9
Weight decay
30--
10-a
0.15
0.20
Pm
0.0
1.0
0.75
0.0
1.0
0.0
1.0
1.0
ResNet-18 18
SGD
SAM
WSAM
XSAM
SGD
SAM
WSAM
XSAM
SGD
SAM
WSAM
XSAM
Epuch
200
200
200
Batch size
125
125
125
Inital learning
0.05
0,05
0.05
0.9
Wright decay
10-5
10--
0.15
0.15
0.15
0.15
0.15
0.15
0.20
0.20
0.20
0.25
0.30
0.25
0.0
1.0
0.5
0.0
1.0
1.25
0.0
1.0
1.0
DenseNeet122 121
SGD
SAM
WSAM
XSAM
SGD
SAM
WSAM
XSAM
SGD
SAM
WSAM
XSAM
Epach
200
200
Initial learning mie
0.05
0.05
0.05
Momentum
8.9
0.9
0.9
Weight decay
11
1xx 10
1 10--
0.05
0.05
0.05
0.30
0.10
0.10
0.20
0.20
0.30
0.20
1:25
20

---

Published as a conference paper at ICLR 2026
Table 6: Hyperparameter details for Results in Figure 3b Note that, in this experiment, a for WSAM adopts the average value of the dynamic a" in the corresponding XSAM. We see from the results that such WSAM already clearly outperforms SAM.
p=
p=0.20
p=0.30
SAM
WSAM
XSAM
SAM
WSAM
XSAM
SAM
WSAM
XSAM
Epoch Batch size
200
200
200
125
125
125
Initial learning rate
0.05
0.05
0.05
Momentum
0.9
0.9
0.9
Weight decay
10-3
10-3
1 10-3
Pm a
0.30
0.30
0.30
1.0
1.57
1.0
1.15
1.0
0.92
Table 7: Hyperparameter details for Results in Figure and 14a
Note that the basic hyperparameters are provided here, while the other hyperparameters are clearly illustrated in the respective figures.
Figure
Figure
14a
SAM
XSAM
SAM
XSAM
Epoch
200
Batch size
125
125
Initial learning rate
0.05
0.05
Momentum
0.9
0.9
Weight decay
10-3
x 10-3
0.15
0.15
Table 8: Hyperparameter details for Results in Figure 3c Note that, in this experiment, a for WSAM adopts the average value the dynamic a in the corresponding XSAM. We see from the results that such WSAM already clearly outperforms SAM.
p=0.04
p=0.08
p=0.12
SAM
WSAM
XSAM
SAM
WSAM
XSAM
SAM
WSAM
XSAM
Epoch
200
200
200
Batch size
125
125
125
Initial learning rate
0.05
0.05
0.05
Momentum
0.9
0.9
Weight decay
x 10
x 10
x 10
0.30
0.25
0.20
1.0
1.72
1.0
1.15
1.0
0.41
Table 9: Hyperparameters for SAM and XSAM on ImageNe//esNNet500,, Transformee/IWSSL22044
and ViT-T//CIFAR-I00 Table 3
ImageNet/ResNett550
Transfommer/WSS122211
DIFAR-I00/VIT-TT
SAM
XSAM
SAM
XSAM
SAM
XSAM
Epoch
90
300
300
Batch size Max Token
512
4096
learning rate
0.2
5
0.001
Momentum
0.9
(0.9.0.98)
(0.9.0.999)
Weight decay
10-4
0.3
0.3
Label smooth
0.0
0.1
0.1
0.05
0.15
0.9
Pm
0.3
0.45
0.9
1.0
1.0
21

---

Published as a conference paper at ICLR 2026
E ADDITIONAL EXPERIMENTAL RESULTS AND ANALYSIS
E. EVALUATION OF XSAM WITH OTHER SAM VARIANTS
In this section, we further evaluate the performance of SAM variants and their combinations with XSAM. As discussed in Section 4 some SAM variants, such as ASAM, FSAM, and VaSSO, target aspects of SAM that are largely orthogonal to those addressed by method, making them potentially compatible for integration. Given the large number of such orthogonal approaches. we focus here on combining XSAM with ASAM and evaluating their performance CIFAR- 1RR00 using ResNet- 18. The results in Table I0 indicate that XSAM outperforms both SAM and ASAM individually. Furthermore, integrating XSAM with ASAM leads to further improvement, demonstrating the effectiveness of XSAM in combination with other SAM variants.
Table 10: Test accuracy of SAM variants and their combinations with XSAM.
SAM
ASAM
XSAM
XSAM+ASAM
Test Accuracy
80.93 + 0.11 81.11 = 0.06 81.24 + 0.07
81.68 + 0.11
We have additionally compared XSAM with ASAM, VaSSO, and WSAM on CIFAR- using both ResNet-18 18 and DenseNet-121. As shown in the Table II XSAM achieves the highest accuracy across both architectures, further demonstrating its effectiveness.
Table 11:
Comparison on CIFAR-100 with ResNet-18 and DenseNet- 121, All baseline methods are carefully tuned for optimal performance. XSAM uses the same as SAM, as in the paper.
Method
ResNet-18
DenseNet-121 121
SAM
80.93 + 0.11
83.81 + 0.02
ASAM
81.11 + 0.06
83.99 0.25
VaSSO
80.84 + 0.15
83.78 + 0.25
WSAM
80.95 + 0.19
83,91 = 0.15
XSAM
81.24 + 0.07
83.96 + 0.10
XSAM + ASAM
81.68 + 0.11
84.06 + 0.21
E.2 ADDITIONAL EXPERIMENTS OF MULTI-STEP SAM
We additionally compare multi-step SAM variants and XSAM under varying p. As we see in Table 12, all of these variants, especially LSAM and MSAM+, which involve intermediate gradients rather than merely using the last gradient gk. managed to get consistently superior results than SAM. The performance of SAM constantly decreases as P gets large, which, from our perspective, suggests the deviations of from du are too large. Under such circumstances, the earlier 9: must have less deviation, so combining with earlier gradients would help. Besides, we see no clear trend for LSAM, LSAM+, MSAM, and MSAM+ as P gets large.
Although MSAM+ can viewed as LSAM+ with weights of gradients changed from 1/1lg:| || to simply the performance gap between them is obvious. This demonstrates that the weighting of gradients at different steps affects performance, and a more appropriate weighting scheme can lead to higher accuracy. Regardless, XSAM consistently outperforms all these methods in all cases.
We further compare XSAM with MSAM and LSAM under & = 1,2, 4 on DenseNet-12 121 using CIFAR- 100 and ResNet- 18 using CIFAR-10 10. As shown in Tables 13 and 14 XSAM consistently attains high accuracy while maintaining strong robustness. In contrast, existing multi-step SAM variants may even underperform their single-step counterparts.
22

---

Published as a conference paper at ICLR 2026
Table 12: Results on CIFAR-I00 using ResNet-18 in multi-step (k 3) setting.
Method
P 0.04
P 0.08
P 0.12
SAM
80.79.0.41
80.75 10.27
79.72 10.33
LSAM
81.00.1..21
81.20
81.16
LSAM+
80.5640.20
80.77
80.21 20.27
MSAM
81.04..0.6
81.1220.117
80.93 11
MSAM+
80.72
81.16=0.05
81.16 10.05
XSAM
81.23
81.36
81.29 20.06
Table 13: Results on DenseNet-121 with CIFAR-I00 with different p p² pk/k with p" for single- step.
Method
k =
== 2
k = 4
LSAM
83.81 + 0.02
83.82 + 0.28
83.40 + 0.17
MSAM
83.81 + 0.02
83.67 + 0.23
83.74 = 0.18
XSAM
83.96 + 0.10
84.05 + 0.04
84.02 = 0.31
Table 14: Results ResNet-18 18 with CIFAR-10 with different k. p= p²/1: with p" for single-step.
Method
k
k: 4
LSAM
96.59 + 0.06
96.66 + 0.03
96.72 + 0.07
MSAM
96.59 + 0.06
96.78 + 0.05
96.80 = 0.07
XSAM
96.74 + 0.04
96.81 + 0.06
96.81 = 0.11
E.3 ADDITIONAL EXPERIMENTS ON CORRUPTED DATASETS
We have conducted additional experiments on CIFAR-10-C and CIFAR-100-C using ResNet-18 and DenseNet-1211 Specifically, we consider 19 types of corruptions, each applied five severity levels, and group them into four categories: noise, blur, weather, and digital.
We report the mean accuracy the evaluation metric, with higher values indicating better performance. The results, as shown in the Table 15 indicate that XSAM consistently achieves high performance and demonstrates robustness across all settings,
Table 15: Performance on CIPAR-10-CICIFAR-ION with ResNet-18 18 and DenseNet-122
(a) 10FAR-100-C, -8
CIFAR-I0-C ResNet-18
Method
Noise
Blur
Weather
Digital
Overall
Method
Noise
Weather
Digital
Overall
SGD
22.36
47.47
55.44
60.03
48.07
SGD
51.14
71.85
85.23
85.85
74.76
SAM
25.58
51.14
58.82
63.30
$1.45
SAM
53.65
75.91
85.23
86.48
76.59
XSAM
25.44
52.65
59.83
63.54
52.07
XSAM
55.13
75.94
85.76
85.99
76.81
(c) CIFAR- 100---00-- C, DenseNet 121
(d) CIFAR-10-C DenseNet 121
Method
Noise
Blur
Weather
Digital
Overall
Method
Noise
Blur
Weather
Digital
Overall
SGD
26.07
$1.12
59,93
SGD
49.50
85.88
74.85
SAM
29.78
55.22
63.58
67.26
55.62
SAM
54.75
77.52
87.30
87.85
78.08
XSAM
31.02
56.73
64.15
67.25
56.37
XSAM
55.94
77.05
87.60
87.72
78.20
E.4 INNER PROPERTIES XSAM
In this section, we present investigations into the internal properties of XSAM.
23

---

Published as a conference paper at ICLR 2026
Best Alpha Start Poine End Peint
Deet Aipho
Start Poins End Point
1.00
6231
0.00
0.00
0:25
0.2
3.90
0.08
1.0
(a) CIFAR-10, 10, ResNet-18 18
(b) CIFAR-100, ResNet-18 18
(c) Tiny Mny-ImmgeNet, ResNec-18 18
Poirt
Start Point
Erd Faine
End Point
End
1/00
1.52
5005
5
225
022
1.0
2.2
(d) CIFAR-10, 10, DenseNet-121
(e) CIFAR-100, DenseNet-121
(f) Tmay-em,,eNet,, DenseNet-122 121
Figure 13: More visualizations of the dynamic estimations of a..
We first visualize the dynamic evaluations of a in a training instance in Figure and in Figure T3 where loss values normalized for better visibility. As a, there is a clear optimal a. With the epoch-wise evaluation of a, we still see that the change of a² during training is very smooth, which supports our choice of less frequently updating a* for reducing computational overhead. On the other hand, we do see that a" is changing during training, which validates our argument that a fixed a may not be optimum.
see, for every dynamic evaluation of
62.00
0.40
2.00
81.64
0.70
0.72
2.75
Teat Accuracy
0.63
[nal
81.28
86.92
3.40
0.19
0.50
3.55
Grideet
2:00
0.42
1.79
80.56
XBMM
SAN
p-0.33
p-011
0.20
0.32
1.5
1,30
86.20
0.15
0.20
0.25
0:30
0.35
0.40
0.45
30
230
0:2
302 Epoch
125
16c
Epoch
Esoch
Epoch
(a)
(b)
(c)
Figure 14: (a) Robustness analysis of XSAM with respect to Pm- (b)Training statistics of XSAM. (c) The norms of gi during training.
We further study how Pm influences the final performance. As results presented in Figure T 44a while Pm does impact performance to some extent, XSAM is able to outperform SAM in fairly large range of pm. from p to 3p. So, we consider that XSAM is not sensitive to Pm. The counterpart, as to how p influences when fixing the Pun, is actually demonstrated in Figure 3bl where have used
fixed Pm = 0.3 by intention. It seems fairly robust to p.
also see that a has decreasing tendency during training. In fact, the angle db between Un and Un has an increasing tendency during training. We visualize such changes along with the offset angle a* so from Up to the direction of the local maximum in Figure 14b] We see that
In our experiments,
24

---

Published as a conference paper at ICLR 2026
the offset angle a" + tends to increase. This may be because it converges to lower position in minima region as the learning rate decreases. Nevertheless, XSAM is able to help it away from the maximum within the local neighborhood in any case, as evident by the test accuracy results.
We show in Figure 14c an instance norm change of gi during training in multi-step settings.
E.5 THE FLATNESS/SHARPNES RESULTING MODELS
Hessian spectrum. To demonstrate that XSAM converges flatter minima (more precisely, success- fully shifts to a region where the maximum within the local neighborhood is lower), we calculate the Hessian cigenvalues of ResNet- 18 trained for 200 epochs
CIFAR-10 with SGD, SAM. and XSAM. Following (Foret et al. 2020 Jastrzebski et al. 2020 Mi et al. 2022), we adopt two metrics: the largest eigenvalue (i.e., A1 and the ratio of the largest eigenvalue to the fifth largest (i.e., X1 AA)). To avoid the expensive computation cost of exact Hessian spectrum calculation, we approximate eigenvalues using the Lanezos algorithm (Ghorbani al. 2019). The results, shown in Table 16 indicate that XSAM yields the smallest hessian spectrum, suggesting that it converges to flatter minima than SAM and SGD.
Table 16: Hessian spectrum of ResNet-18 using SGD, SAM, and XSAM on CIFAR-10.
SGD
SAM
XSAM
A
78.79
36.15
33.92
A1/A5
2.26
1.89
1.59
Visualization of loss landscape. We visualize the loss landscape of ResNet-18 trained on CIFAR-10 with SGD, SAM, and XSAM further compare the flatness of the minimum. Using the visualization procedure in (Li et al. 2018), we randomly choose orthogonal normalization directions (i.e., X axis and axis) and then sample 50 x 50 points in the range of [-1,1] from these two directions. As shown in Figure [5] XSAM has flatter loss landscape than SAM and SGD.
1000
100 Mor Post
175
(a) SGD
(b) SAM
(e) XSAM
Figure 15: Loss landscape visualizations of ResNet-18 on CIFAR-10 with SGD, SAM, and XSAM.
Average sharpness. We further visualize the average sharpness of the loss landscape at the conver- gence point. Specifically, following (Foret al. 2020). we define the sharpness as the difference between the loss of the perturbation point and the loss of the convergence point. The average sharp- ness is then computed as the mean sharpness over multiple perturbations under the same perturbation radius. Then, we sample multiple random directions (e.g.. 10, 50, 250, 1250) and continue this process until the average sharpness loss curve stabilizes, which provides more representative charac- terization of the loss behavior around the convergence point. Based on experiments, sampling 250 random directions sufficient to achieve stable results. addition, for the perturbation method, we adopt filter-wise and element-wiee perturbation following (Li al.. 2018) to between different optimizers (i.e., SGD, SAM, and XSAM), As shown in Figure 16 SAM exhibits smaller average sharpness compared SGD, while XSAM further reduces the average sharpness.
fair comparison
25

---

Published as a conference paper at ICLR 2026
0.70
SGD
CIFAR-10, ResNet-18, k=1
0.35
SGD
CIFAR-10, ResNett118 k=1
SAM
SAM
Average Sharpness
Sharpness
0.30
0.50
0.25
0.40
0.30
0.30
0.10
0.05
0.00
0.00
0.3
0.4
0.5
0.1
0.4
(a) Element-wise perturbation
(b) Filter-wisc perturbation
Figure 16: Visualization of the average sharpness of the loss landscape at the convergence point.
F STRATEGIES FOR GRADIENT SCALE
Our default gradient scale strategy using IIgx|l to match the scale with SAM. In this section, empirically study set of different ways for setting the gradient scale, which includes: typical choices
(k + 1) and max ||9.]]. Besides, we further like II and II simple extensions like Ei llalll(kk explored two slope-based strategies:
L(0x)) L(Qu)
slope;
110 - Poll
slope Liuo + v(a) pm) L(Vo)
Pm
which is the averaged slope from do to 0g and from 0g to the approximated maximum, respectively.
Note that since our direction is away from the approximated maximum, it can be an interesting combination when using the slope from dn to the approximated maximum as the gradient scale, which shares the intrinsic core stochastic gradient descent. However, it would require an extra forward pass to evaluate L(uo + v(a.) pm)
75.00
CIFAR-100, VGG-11,
CIPAR-100, ResNet-18, k=1
74.50
Accuracy
Test Accuracy
81.00
73.50
80.50
73.00
80.00
72.50
0.01
0.025
Learning Rate
Learning Rate
lool
meant]o.ll
(L08)) LO 6v1 (L68g via) :e) 46801M
10.1
meand
[4]8.] LOBWIV13,
10.1
max
max((pl)
(419g viel: Ae) LiBgMiPe.
(a)
Figure 17: Comparison of various gradient scale strategies.
The results are shown in Figure 17 As we can see, the gradient scale seems to be something that is even more mysterious than the gradient direction. It is hard to draw a direct conclusion on which might be the best choice among such reasonably large group. Nevertheless, some choices appear to be good in most circumstances, which may include | [ and slopem. These primary results included for completeness. Notably, the work TTn et al. 2025) argues that rescaling the gradient using ||gn|] more stable than using [[ |]. our experiments, however, we do not observe noticeable stability advantage. We would leave further investigation into this future work.
26

---

Published as a conference paper at ICLR 2026
G ADDITIONAL RELATED WORK
The connection between flatness/sharpness and generalization was realized early on (Hochreiter & Schmidhuber 1994) and further explored in subsequent works (Hochreiter & Schmidhuber 1997]
McAllester 1999; Neyshabur et al. 2017 Jiang et al. 2019), motivating efforts toward finding flatter solutions. While SGD believed favor flat minima implicitly (Keskar et al. 2016 Ma Ying 2021), more explicit methods are preferred and developed. Typical instances include Entropy-SGDD
(Chaudhari et al., (017)) that employs entropy regularization, SWA (Izmailov et al. 2018) that seeks
flatness by averaging model parameters, and SAM (Foret et al. 2020) that optimizes sharpness.
There are some variants that focus on improving the performance of multi-step SAM. Vanilla multi- step SAM (Foret et al. 2020) updates the model using the gradient at the last step. MSAM (Kim et al.) 2023) suggests averaging all gradients except the first gradient at the original location. Lookbehind- SAM (LSAM) (Mordido et al. 2024) suggests another way that utilizes all gradients but excludes the first. In comparison, in multi-step settings, our method leverages all gradients (19, ) in vo. and Ok
in U1) in a dynamic interpolation manner and explicitly approximates the direction of the maximum.
There are also some works that seek to reduce the computational overhead of SAM. For instance, ESAM (Du et al. 202T) achieves this via stochastic weight perturbation and sharpness-ennsttiv data selection. SSAM (Mi al. 2022) accelerates SAM with a sparse perturbation. LookSAM (Liu et al., 2022a) reduces computational overhead by computing SAM's gradient only periodically and relying on an approximate gradient for most of the training time. RST (Zhao al. (022b) and AE-SAM (Jiang et al. 2023) suggest alternating between SGD and SAM in randomized and adaptive ways, respectively.
Another important line of research on SAM focuses on understanding its underlying mechanism. For instance, (Wen et al. 2023) finds that the gradient of SAM aligns with the top eigenvector of the Hessian in the late phase of training. This phenomenon is also concurrently found by (Bartlett et al. 2023). (Andriushchenko al., (023a) argues that SAM leads to low-rank features. In addition, an interesting fact observed by (Andriushchenko & Flammarion, (022) is that training with SAM only in the late phase of training can achieve an improvement similar to that of full training with SAM. A recent work (Zhou et al., 2025) further analyzes and theoretically shows the learning dynamics of applying SAM late in training. (Tahmascbi et al., 2024) introduces a universal class of sharpness measures, in which SAM, known for its bias toward minimizing the maximum eigenvalue of the Hessian matrix, can be regarded as a special case. Our work is orthogonal to these works, providing a new perspective for understanding fundamental question of why applying the gradient from the ascent point to the current parameters is valid. At the same time, we propose XSAM as a better alternative.
In addition, SAM achieves extraordinary performance on various tasks. For instance, it has proven particularly effective in long-tail learning (Rangwani al. 20226). ImbSAM (Zhou et al. 2023) applies SAM only to the tail classes to improve their generalization. Further, CC-SAM (Gowda & Clifton 2024) generates class-specific perturbations for each class, although this comes at an increased computational cost. Focal-SAM M (Li et al. 2025) aims to achieve fine- fin-grained sharpness control for each class while maintaining efficiency. These SAM variants are specialized in long-tail learning and differ from our work.
H USE OF LARGE LANGUAGE MODELS
We used large language model (LLM) only for language polishing (grammar, wording, and clarity) of drafts written by the authors. The model did not generate research ideas, methods, analyses, results, or figures, and it did not write any sections from scratch.
27


Reviews:
[{"review_id": "Z5v5SXuUvP", "reviewer": "Reviewer_Qt3b", "summary": "This paper revisits Sharpness-Aware Minimization (SAM) and introduces eXplicit Sharpness-Aware Minimization (XSAM), which improves upon SAM by explicitly estimating the direction toward the local maximum loss within a neighborhood, rather than relying on the potentially inaccurate gradient at the ascent point. XSAM consistently outperforms SAM and its variants across various models and datasets in both single-step and multi-step settings. The authors provide both theoretical justification and empirical visualizations to explain why SAM’s approximation can be suboptimal and how XSAM addresses these limitations.", "questions": "1. Your 2-D search is restricted to the plane spanned by v_{0} and v_{1}.\nWhat theoretical or empirical evidence guarantees that the true worst-case perturbation   \\delta^{*} (i.e., the arg-max of L(\\theta+\\delta) inside the \\rho-ball) actually lies in this plane, and how might the method behave if the Hessian is highly anisotropic or the loss landscape is strongly non-quadratic?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. XSAM provides a more accurate and adaptive estimation of the direction toward the local maximum, leading to better generalization.\n\n2. The method is unified across single-step and multi-step settings and shows consistent improvements over SAM with minimal computational overhead.\n\n3. The paper offers a clear theoretical and intuitive explanation of SAM’s limitations, enhancing understanding of sharpness-aware optimization.", "weaknesses": "1. XSAM introduces additional hyperparameters (e.g., search range for α and update frequency), which may complicate hyperparameter tuning.\n\n2. Although the overhead is small, XSAM still requires extra forward passes for direction estimation, slightly increasing computational cost.\n\n3. Although the paper critiques multi-step SAM’s degradation, the comparison with existing multi-step variants (like MSAM or LSAM) could be more extensive to fully establish XSAM’s superiority in that regime.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LTjTWDuUQm", "reviewer": "Reviewer_Rjdp", "summary": "In this paper, the authors study how the sharpness-aware minimization (SAM) algorithm works and how to improve it. \nThey first show that the single-step ascent used in SAM, when applied to current parameters, is a better approximation of the maximum over a ball, compared to the naive gradient. This shows why the SAM algorithm is superior in practice, as it is indeed minimizing the maximum loss over a ball. \nIndeed, they show that the approximation of max loss with naive gradient in a single step is often inaccurate, and the quality gets worse if we consider multi-step ascent, a phenomenon observed in practice. \n\n \nIn Proposition 1, they show that while SAM provides a better approximation of the maximum loss around parameters (up to a ball), linear combinations of SAM updates and naive gradients can be even better. They thus propose XSAM: \nThey probe in the hyperplane of the naive gradient and the SAM gradient, using spherical linear interpolation (Eq. 6). It needs some maximization over a parameter alpha.\nThe proposed method proposed only a small additional computational overhead.\nThey conclude the paper with experiments supporting their proposed method.", "questions": "This is an interesting paper, and I believe it makes a good contribution to our understanding of why SAM works and how to improve it. \n\nMy only concern is that the paper lacks good writing. I appreciate that the authors provided an explanation of how they used LLMs just for polishing sentences, but the fact that there are long sentences in the paper is quite annoying (sometimes LLMs suggest them as a way to polish the paper, but they indeed make reading the paper harder).  For example, Line 19-24 of abstract, the main question of the paper, is introduced in a very, very long sentence, hard to follow. Please modify the abstract and other parts of the paper, and make sure the message is delivered clearly. The paper, in my opinion, needs major changes. \n\n\n- (minor) Line 11 in the algorithm: Where is the definition of v(alpha) within the algorithm?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The theoretical work on explaining the success of the SAM algorithm in minimizing the SAM objective, compared to the naive gradient, is of potential community interest. \n\n\n - The paper is probing between the SAM and naive gradient method, and it shows that it will strictly improve the performance, which is of potential interest to the community. The experiments also support this.", "weaknesses": "- Long sentences, hard to follow. The paper would benefit from better writing, focusing on short and clean sentences.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VTHTAmlJ8e", "reviewer": "Reviewer_oQCt", "summary": "This paper revisits Sharpness-Aware Minimization (SAM) and provides a new interpretation of its mechanism: the gradient at the ascent point more accurately approximates the direction toward the worst-case loss in a neighborhood than the local gradient. Motivated by visualization and theoretical analysis showing that this approximation is often inaccurate and deteriorates under multi-step ascent, the authors propose XSAM, which explicitly estimates the ascent direction within a two-dimensional subspace via spherical interpolation. XSAM dynamically adjusts the interpolation factor and updates parameters in the opposite direction to escape local maximum more effectively. Extensive experiments validate the effectiveness of the proposed method.", "questions": "Please see above.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper is generally well-written and easy to follow.\n- The authors perform visualization studies to show how single-step SAM gradient directions better approximate ascent directions within the neighborhood, while multi-step SAM may degrade. These visualizations ground the theoretical intuition in empirical phenomena.\n- Despite the additional probing steps, the runtime overhead remains negligible. The method is compatible with SAM , making it practical and easy to integrate into real-world training pipelines.", "weaknesses": "- The underlying motivation for using $-v(\\alpha^*)$ as the final gradient descent direction remains unclear. Following the direction of $-v(\\alpha^*)$ appears to encourage moving away from a local neighborhood maximum. However, this does not necessarily guarantee convergence toward a flatter minimum. Additional clarification and theoretical justification would strengthen the argument.\n- In the experimental section, the paper primarily compares the proposed method against standard SAM and SGD. Given that numerous SAM variants have been discussed in the related work, including stronger SAM-based baselines would provide a more rigorous and convincing empirical evaluation of the proposed method's effectiveness.\n\n- More representative SAM-based methods, such as GSAM [1], GAM [2], ImbSAM [3], CC-SAM [4], and Focal-SAM [5] should be included in the related work for a more comprehensive review.\n\n-----\n\n[1] Surrogate Gap Minimization Improves Sharpness-Aware Training, ICLR 2022\n\n[2] Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalization, CVPR 2023\n\n[3] ImbSAM: A Closer Look at Sharpness-Aware Minimization in Class-Imbalanced Recognition, ICCV 2023\n\n[4] Class-Conditional Sharpness-Aware Minimization for Deep Long-Tailed Recognition, CVPR 2023\n\n[5] Focal-SAM: Focal Sharpness-Aware Minimization for Long-Tailed Classification, ICML 2025", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mXrJ344KCU", "reviewer": "Reviewer_SZGB", "summary": "This paper proposes a new variant of SAM optimizer, namely, XSAM that provides a more faithful and effective estimation of the gradient with respect to the current parameters. The authors argue that the gradient at the single-step ascent point offers a better approximation of the direction from the current parameters towards the maximum within the local neighborhood than the local gradient. And they further justify this observation under a second-order approximation, theoretically. Extensive experiments are also presented to validate the efficacy of the proposed optimizer.", "questions": "please see Weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper is well-motivated and easy to follow. Actually, SAM optimizer has attracted a lot of attention and many works have been devoted to improve the generalization performance of SAM. In this work, the authors identified a critical issue in estimating the gradient descent direction and proposes a variant of SAM that unifies one-step and multi-step versions.", "weaknesses": "While the authors have presented their work in a good shape, I still have serveral questions:\n\n - In line 22, the authors claim that **thereby enabling a more direct escape from the maximum within the local neighborhood**. Actually, I do not catch why is this case. As Figure 1a suggests, descending along $g_0$ quickly steers the optimization process towards the low-loss regions,  then $g_1@\\theta_0$,  and $g_1$ gradually mitigates this issue. Frankly speaking, I do not know how it relates to the maximum of the local neighborhood.\n \n - In line 72, the authors states that **...eveal that the approximation by the gradient at the single-step ascent point is often inaccurate**. Possibly, here might be a mistake. Please recall that Figure 1a is visualized by spanning $g_0$ and $g_1$. And, the direction of $g_1$ is heavily dependent on the perturation radius $\\rho$.  Therefore, the loss peak might vary significantly. It would be much better to visulize the true gradient with respect to the current parameters as well. Moreover, what's the value of $\\rho$?\n \n - On the theoretical respect, my biggest concern is that the proof of Proposition 1 is too hand-wavy. In line 810,  **for sufficiently large $\\rho_m$, the $\\rho_m^2$ term dominates**. I believe here deserves further investigation because the taylor-expansion holds only for small $\\rho_m$ and the remainding terms cannot be simply removed. Moreover, the assumption that Hessian $H$ is positive definite also requires careful handling and experimental verification. Idealy, I would suggest to plot $L(\\theta_0+\\rho_m\\frac{g_1}{\\|g_1\\|})$ versus $L(\\theta_0+\\rho_m\\frac{g_0}{\\|g_0\\|})$ for different combinations of $\\rho$ and $\\rho_m$.\n\n- In Experiments section, more baselines should be included, such as ASAM, FisherSAM, etc. Particularly, line 16 of Algorithm 1 rescales the descent direction with $\\|g_k\\|$. I am wondering how it performs when rescaled with the base gradient $\\|g_0\\|$? The result of a recent study [1] seems to suggest it works. Moreover, how XSAM performs on corrupted dataset such as CIFAR-10/100-C should be reported as well.\n\n[1] Tan et al., Stabilizing sharpness-aware minimization through a simple renormalization strategy, JMLR, 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)