Towards Understanding The Calibration Benefits of Sharpness-Aware Minimization
Authors: Chengli Tan
Yubo Zhou
Haishan Ye
Guang Dai
Junmin Liu
Zengjie Song
Jiangshe Zhang
Zixiang Zhao
Yunda Hao
Yong Xu
Venue: ICLR 2026 Poster (2026)
Keywords: deep neural networks
stochastic gradient descent
sharpness-aware minimization
Primary area: optimization
TL;DR: Beyond the well-known benefits on improving generalization, SAM is also found to be surprisingly effective in model calibration.

Abstract: Deep neural networks have been increasingly used in safety-critical applications such as medical diagnosis and autonomous driving. However, many studies suggest that they are prone to being poorly calibrated and have a propensity for overconfidence, which may have disastrous consequences. In this paper, unlike standard training such as stochastic gradient descent, we show that the recently proposed sharpness-aware minimization (SAM) counteracts this tendency towards overconfidence. The theoretical analysis suggests that SAM allows us to learn models that are already well-calibrated by implicitly maximizing the entropy of the predictive distribution. Inspired by this finding, we further propose a variant of SAM, coined as CSAM, to ameliorate model calibration. Extensive experiments on various datasets, including ImageNet-1K, demonstrate the benefits of SAM in reducing calibration error. Meanwhile, CSAM performs even better than SAM and consistently achieves lower calibration error than other approaches.

Published as a conference paper at ICLR 2026
TOWARDS UNDERSTANDING THE CALIBRATION BENE- FITS OF SHARPNESS- AWARE MINIMIZATION
Chengli Tan 12, Yubo Zhou?.3 Haishan Ye Guang Dai Junmin Li
Zengjie Song?, Jiangshe Zhang?, Zixiang Zhao', Yunda Hao Yong Xu!
Northwestern Polytechnical University
SGIT AI Lab, State Grid Corporation of China Chinese University of Hong Kong
3 Xi' an Jiaotong University
ETH Zürich
ABSTRACT
Deep neural networks have been increasingly used in safety-critical applications such as medical diagnosis and autonomous driving. However, many studies suggest that they are prone being poorly calibrated and have a propensity for overcon- fidence, which may have disastrous consequences. In this paper, unlike standard training such stochastic gradient descent, show that the recently proposed sharpness-aware minimization (SAM) counteracts this tendency towards overconfi- dence. The theoretical analysis suggests that SAM allows us learn models that are already well-calibrated by implicitly maximizing the entropy of the predictive distribution. Inspired by this finding, we further propose variant of SAM, coined various
CSAM, to ameliorate model calibration. Extensive experiments datasets, including ImageNet-Iee IK, demonstrate the benefits of SAM in reducing cal- ibration error. Meanwhile, CSAM performs even better than SAM and consistently achieves lower calibration error than other approaches.
INTRODUCTION
While the relation between generalization and flatness is still in dispute (Dinh et al. 2017 Ramasinghe cct al. 2023 Andriushchenko et al., [2023] Wen et al. (0024), is empirically appreciated that under
some constraints the flatter solutions tend to generalize better (Hinton & van Camp 1993 Keskar
ct al. 2017 Chaudhari et al. 2019 Kaddour et al. 2022). From this point of view, many approaches
have been proposed to bias solutions toward flat regions of the loss landscape explicitly or implicitly
(Huang et al., 201 Izmailov et al., 2018 Chaudhari et al. 22019] Zhang et al. 2019 Wang et al.
2021b; Bisla al.) 2022), amongst which SAM (Foret al. 2021
garnered increasing attention
due to its surprising effectiveness on popular tasks such as image classification (Chen et al. 2022).
language generation (Bahri al., 2022), and even physical computation (Xu et al.) 2024).
Different from standard training like stochastic gradient descent (SGD), SAM minimizes a perturbed loss, and each iteration is composed of two consecutive steps.
V L (0:) 0, = 0 + V (0:)ll2 0 0; Lo, (0.).
where O, Rdd represents the learnable parameters of the neural network (-th iteration, 7) is the learning rate, is the perturbation radius, and Lo, (-) denotes the empirical loss on a mini-batch Q2 of the training set S. This scheme constantly penalizes the gradient norm (Zhao et al. 2022a Wen et al. 2022 Compagnoni et al. (202)) and significantly promotes generalization. On the other hand, model calibration refers to how reliable the model predictions are. Idcally, when the model is confident about its predictions, the predictions are supposed to be as accurate possible. This is particularly important for real-world applications such as autonomous driving (Chib & Singh (023) and medical diagnosis (Jiang al.) (2012). As example, consider self-driving car that uses deep neural networks to detect whether an obstruction a pedestrian or not. For an ill-calibrated model,
"Corresponding authors juumminii.ycchaishaneee www.iaanmmmml......c.....o.mm

---

Published as a conference paper at ICLR 2026
SCD
SAM
SGD
SAM
1.0
0.8
0.5
0.6-
Austrary
0.5
0.4
Proquetion
0.4
0.2-
0.2
0.0
0.00
0.25
0.50
0.75
1.00
[U00
0.25
0.00 Conddraze
0.75
1,00
0.00
0.25
0,50
0.75
1.00
0.00
0.25
0,50
0.75
1.00
Conbdeuce
Confdence
Coundrnce
Figure I: Confidence and reliability histograms for a PyramidNet (Han et al. 2017) trained on CIFAR-100 (Krizhevsky et al. (209) with different optimizers. For clarity, the term confidence here refers to the predicted probability, namely, the maximum output of the softmax layer.
when its confidence is low, it may just pass through and will not trigger emergency braking, which could cause undesired consequences. the obstruction is pedestrian or not when its confidence is low. As a result, a more cautious decision would be made by the car to avoid an accident.
contrast, for a well-calibrated model, it not certain whether
It is known that modern neural networks such as ResNets (He et al. 2016) and DenseNets (Huang al. 7b) often suffer from miscalibration problem, and this issue appears to be more serious
when the network starts to overfit the training data (Nguyen al., 2015 Guo et al. 2017 Zhu ict al. (202)). Since SAM is more effective in preventing overfitting FForet et al. 202 one could anticipate that neural networks optimized by SAM may be better calibrated than by base optimizers such as SGD and A (Loshchilov & Hutter 2019). This is illustrated
Figure [ where a
large PyramidNet (Han al., (017)) respectively trained on CIFAR- 100 (Krizhevsky al. 2009)
with SGD and SAM. One can easily observe that the average confidence of SAM closely matches its accuracy, while the average confidence of SGD is substantially higher than its accuracy. This is further confirmed with reliability diagram (Niculescu- Mizil & Caruana 2005). where we plot the accuracy as a function of the confidence. The diagram indicates that SAM is better calibrated than SGD, as the accuracy almost overlaps with the confidence along the diagonal line.
While previous studies (Zheng al. 2021 Möllenhoff & Khan 2023) have reported this phenomenon, the question of how SAM alleviates the miscalibration problem has not been formally investigated, and we attempt to fill this gap in this paper. In brief, our contributions are as follows:
We provide theoretical justification for the calibration benefits of SAM, which essentially performs an implicit regularization on the negative entropy of the predictive distribution. This is similar to focal loss (Mukhoti et al., 2020), but SAM calibrates models much better without compromising accuracy.
We investigate how SAM performs on model calibration under distribution shift and find that SAM allows models to remain well-calibrated under different types of corruption. Moreover, the trick of ensembling also useful for SAM, and compared SGD, the improvement more pronounced on ant-of-distributio data.
We develop a variant of SAM, termed CSAM, that attempts to improve model calibration further. By extensive experiments with a variety of network architectures and datasets, we observe that CSAM consistently performs better than SAM and surpasses other approaches that are focused on improving calibration.
The remainder of the paper is organized as follows. We first review the related work in Section and then introduce some backgrounds in Section After presenting the theoretical analysis of SAM and the derivation of CSAM in Section 4 we further provide the experimental results in Section
2
RELATED WORK
In this section, we present the most relevant works on SAM and the miscalibration of deep neural networks.
2

---

Published as a conference paper at ICLR 2026
Sharpness-aware minimization. Because SAM is particularly effective in improving the general-
ization performance of realistic neural networks (Foret et al. [02]] Chen et al.) 2022 Bahri et al.
2022), it has received lot of attention in recent years, and there surge of research along this
direction. For example, to reduce the computational overhead incurred by the additional backpropa-
gation, some works choose to apply SAM and standard training alternatively (Liu et al. [022b Zhao
cct al. [022b] Jiang et al. [0233 Tan et al., 2024a)), while some other works focus on perturbing
fraction of parameters (Du et al. 2022; Mi al. (022) or examples (Ni et al. 2022). Concurrently,
some researchers also attempt to further enhance the generalization performance of SAM Zhang
al. 2022; Li & Giannakis [023 Yue et al.) 2023 Zhou et al. 2023). For example, |woon et al.
(2021) propose ASAM to consolidate the correlation between sharpness and generalization, which might break up due to model reparameterization (Dinh et al., 2017). And Kim et al. (2022) further propose FMM to enforce that the optimization occurs on the statistical manifold induced by the
Fisher information matrix.
On the theoretical aspect, Wen al. 2022 Bartlett et al. (023) prove that the largest eigenvalue of the Hessian decreases along the trajectory of SAM, a result which is quite similar to that of Compagnoni et (2023) though derived from the perspective of the stochastic differential equation. Andriushchenko & Flammarion (2022) propose to study the unnormalized SAM and demonstrate the implicit bias on simple diagonal neural networks. Based on uniform stability (Bousquet & Elisseeff 2002. Hardt et al.] 20166), Tan et al. (2024b) prove that SAM generalizes better than SGD on strongly convex problems, and propose renormalization trick to mitigate the instability issue near the saddle points (Compagnoni et al. 22033 Kim et al. 2023).
Miscalibration of deep neural networks. In machine learning, calibration has been extensively studied (Platt et al.) Greiting et al. |207]] Futami & Fujisawa 2024). Since popular classification losses like squared error and cross-entropy (CE) are proper scoring rules (Gneiting al.. 2007), they are guaranteed to produce perfectly calibrated models at their global minimum. However, as first disclosed by Guo al. (2017). modern neural networks suffer from serious miscalibration due to overfitting and overparameterization (Lakshminarayanan et al. 2017, Thulasidasan et al. 2019 Wang jet al.] 10 Wang 2023). While Minderer al. (2021) argue that the most recent non-convolutional models like MLP-Mixe (Tolstikhin et al.) 2001 and vision transformers (Dosovitskiy et al.) (022]) are better calibrated, the issue of miscalibration is still prevalent in a wide spectrum of applications
like data distillation (Zhu et al., 2023) and object detection (Kuzucu et al. (025).
variety of approaches have been proposed to improve model calibration. In the training-time calibration, for example, an intuitive idea is to penalize overconfidence. either explicitly via entropy- based regularization (Pereyra et al., 2017) and label smoothing (Müller et al.) 2019) or implicitly using focal loss (FL) (Mukhoti et al. 22200] Tao al., 2023). However, pointed out by previous works Wang et al.] 1a) Singh (2021), the penalty of confident outputs may suppress the potential improvement in the post-hoc calibration phase. On the other hand, post-hoc calibration addresses the miscalibration problem by appending post-processing step to the training phase and typically requires a hold-out validation set for hyperparameter tuning. Popular post-hoc methods include non cor-paraettric calibration methods- histogram binning (Zadrozny & Elkan (001) and isotonic regression (Zadrozny& & Elkan 2002). and parametric methods like Bayesian binning (Naeini et al., 2015) and Platt scaling (Platt et al. 1999p. Out of them, Platt scaling-based approaches such as temperature scaling (Guo et al. 2017) and Dirichlet calibration (Kull et al.) (019) are more frequently used due to their low complexity and efficiency.
3 PRELIMINARIES
In this section, we first introduce one measure of model calibration that we use throughout, and then briefly recap the difference between SAM and SGD. Without loss of generality, we consider the multi-class classification problem where a categorical variable Y E {l...., K} is predicted when an input variable X is observed. And we further assume that the training set S contains examples {a, = (Iz. y that are i.i.d. sampled from an unknown data distribution D. For a deep neural network parameterized by E IRd, we naturally obtain a predictor so that maps the features X to a categorical distribution over K labels, which we denote it by So (X) )) that belongs to
(K -)-dimensional simplex A- {p E [0, 11 EK= Py -1). Then, 인 A arg max K Py is the predicted label.
3

---

Published as a conference paper at ICLR 2026
3.1 EXPECTED CALIBRATION ERROR
model well-calibrated if the confidence truthfully recovers the probability of correctness. That is, if we gather all data points for which the model predicts Py = 0.8, we expect that 80% of them should take on the label Mathematically. we refer to model as well-calibrated (Bröcker 2009) if
P(Y y (X) p) - PyΑ VpE
In practice, however, we will focus on the top-label calibration (Guo et al. 2017) that requires the above equation to hold only for the most likely label, namely,
P(Y = ø|
max Pv = p) = p. Ap [0,
SySK K
Expected calibration error (ECE) is the most commonly used metric to measure the degree of miscalibration, which quantifies the expected difference between two sides of the above equation as follows
E P(Y P(Y
max Py = = ISySK
In practice, due finite examples, it works by firstly grouping all examples, say, {z+ (Ii, yt yi) into M bins B,...., Bas based on their top confidence scores. Next, we compute in each bin B; the average confidence conf (B,) 1/|B,] E,,eB, max fo(z,) and the average accuracy acc B,)
1/|B;] Σ,,eB, Tyy = arg max fx,))), where I[.] is the indicator function. Then, we can obtain an estimator by averaging over the bins
M
Bi
ECE
-
m
3.2 SHARPNESS-AWARE MINIMIZATION
The intuitive idea of SAM FForet al. 2021 is improve generalization by constantly minimizing the solution sharpness during training. To this end, instead of minimizing the loss at the current point, it minimizes the worst-case loss within its neighborhood. Mathematically, it equivalent to solving the following optimization problem,
min max Ls(0 +),,
DE ||
where Rdd is perturbation vector whose norm is bounded by the perturbation radius p 0. It is not easy to solve this minimax problem explicitly. But, after simple Taylor approximation, we observe that
ex A arg max Lg(0 +
VLs
arg max (0) + eVVLs( = L L
|e This suggests that, as opposed to SGD, we first need to do an extra gradient backpropagation to estimate the perturbed vector e*. Therefore, SAM actually consists of two consecutive steps each iteration,
VLg, (O,)
0, -, 0, + P V Ln, (0.) 6 - - 7/ LL, (0.).
where Ω2 denotes random mini-batch of S. We note that the same S2 is used for the ascent and descent steps, and smaller 52 is preferred in practice for better generalization (Foret al. 202T Andriushchenko & Flammarion, 2022
4 METHODOLOGY
In this section, we first show that SAM is bound to prevent deep neural networks from producing overconfident predictions. As in previous studies (Guo et al. 2017 Minderer et al.) 202 Wang et al. 2021a), we focus on the most widely used cross-entropy (CE) loss in the classification problem, which for example (z, y) is defined as (0(2) straightforward, and all proofs are deferred to Appendix for clarity. Towards the end of this section, also develop variant of SAM to improve its calibration performance.
log Pg in one-ho hot encoding. The analysis is
4

---

Published as a conference paper at ICLR 2026
Youss (0)
lound
Valoo
8-
50th epich
120th goohh
-22
50
100)
130
200
100
100
200
Epock
Epoods
(a)
(b)
(c)
Figure 2: To verify whether the boundedness assumption of Emin (see Lemmas I and 2 holds for realistic neural networks, we trained a ResNet-56 CIFAR-10 using a constant p 0.05. (a) compares the value of Kmmn at the two endpoints 0 and & throughout the training process. (b) further illustrates how Kmin evolves along the path from O(t 0) to Ó = 1) 50th, 150th, and 200th epoch, respectively. (c) records how the true coefficient À and its lower bound λu (see Equation 2 vary during training. Notice that similar results for the realistic ImageNet- dataset can be found Figure S4
4.1 THEORETICAL ANALYSIS
Let Py [f)())]y and P, (x)l, denote the confidence on the true label y conditioned on the current weight O and the perturbed weight 0, respectively. When the mini-batch namely, every step sample one example only to estimate the gradient, the following lemma suggests that Py can be consistently smaller than Py during training.
Lemma 1 (1-SAM version). Assume that at each step, the gradient Vol(z) + and there always exists some p > 0 such that the smallest eigenvalue of the Hessian Kimin ( (V (2)) 2 =
holds for all 0' (1 1)0 10, |0,11
Then, given Py, Py defined as above, we have
Py Se (t))
Actually, the boundedness of Kmin O can be easily verified along the optimization trajectory (Zhou et al. 2021 Section 6.2). However, it should be noted that the inequality does not necessarily hold for all 0. But if vary P accordingly each step (p - in the worst case), the validity of the inequality can be assured because we are always ascending along the gradient direction. This lemma shows that py, the probability of the perturbed network assigned to the true label, exponentially decays with the perturbation radius and the gradient norm || Vo((=)]l-
Remark 1. A similar result for the mini-batch SAM also developed in Lemma Notice that varying p every step is quite different from the practical setting, in which we oflen use constant p instead. Luckily, as Figure (a) suggests, the boundedness assumption of Kmin can be validated for the constant p over mini-batch SAM. Surprisingly, we also find that Kymin linearly decreases along O (see Figure ())) suggesting that the boundedness assumption can be simplified to requiring Kminn (V2(g(=) -[Ve(()) only. This finding further reveals why large value of is not preferred because the boundedness assumption can be easily violated in that case.
Remark 2. It should be highlighted that the gradient norm V (2((2)] also plays a critical role in determining Py. Lemma 1 indicates that the SAM optimizer is more effective for larger gradient norm, while simultaneously allowing us choose relatively large p. This finding aligned with the observation that SAM is particularty effective in training ViT models with Adam W. which eventually improves more than 5% accuracy on ImageNet-IK 1K using a large p (Chen al. 2022).
Under Lemma I we show that minimizing the perturbed loss L2(2) has the same effect adding maximum-entropyy regularizer to (2) as focal loss (FL. (Mukhoti et al. 2020) Section 4).
Theorem (1-SAM version).
(1 - P1)/(1 - Py), the following inequality holds (0(2) 2 lo(z) >H((p)) + H(P)).
(1)
where H(p) -plogp -1 p) log(1 p) is the binary entropy function.
5

---

Published as a conference paper at ICLR 2026
According to Lemma we know that the coefficient À is larger than 1, which implies that minimizing lo implicitly puts more emphasis on maximizing H (Py) in contrast to minimizing H (Py). That is, SAM forces Py be smaller when it approaches and to be larger when it is near O. Moreover,
when replacing Py with eellVatt))l//2 Pyy. we have
Py A A0 1 epll (oll))
We note that the penalty maximizing HPy)) is stronger the terminal phase of training than at the initial phase (see Figure c)). Since model architecture is also a major determinant of model calibration (Minderer al. 2021). it suggests that SAM could calibrate better for model architectures
that are seriously overconfident.
In practice, suggested by (Foret al. 2021: Andriushchenko & Flammarion, 2022), we attempt minimize the so-called m-sharpness to achieve the largest performance increment. Different from 1-SAM, every step we determine the ascent direction using the gradient averaged over mini-batch Ω of m examples. As a result, the gradient corresponding to one example V (z) is not promised to align well with the mini-batch gradient V La - 1/m E (l)(2)). Therefore, the relation py: S Py, does necessarily hold for all 2, E Q. However, when both of them are taken into account, we do have result similar to Lemma ass follows.
Lemma (m-SAM version). Assume that at each step, the gradient (o) + and there always exists some p > O such that the smallest eigenvalue of the Hessian Kmin (VL 2 Ln holds for all 6' = t)e + tθ, t E [0, Denote P, = (II py. 1/m P3 (II." Pyy.)/" respectively. Then, we have P,, I <=lllbat0)1//2 Py.
and
The proof is straightforward, and accordingly, we have the following result. Theorem (m-SAM version). Let Py and Py defined as above. Then, it follows that Ln(0)= 2 (o) - (H H(,,),
where À = (1-p))/(1 1111111p) -py)
This theorem is similar to Theorem I albeit Py is the geometric mean of the predicted probabilities. But it is enough to make sure that m-SAM prevents models from producing overconfident predictions well.
4.2 IMPROVING SAM TOWARDS BETTER CALIBRATION
As shown in Figure c), we notice that SAM primarily starts to penalize the predictive distribution at the late stages of training where Py is high. Therefore, we propose suppress the contribution of the over-confident examples so that their predictive probability P., is virtually higher. That is, we can redefine the per-example loss function for the outer loop of SAM as follows:
1-- log P o (- (1+ log py.
if p, S 1/2, otherwise,
where 0 < S is a hyperparameter. It is trivial to recover the standard SAM when = 0. Actually, the following result suggests that the modified loss function (z(2) enforces SAM to penalize the predictive distribution of over-confident examples.
Theorem 3. Let Lemma
1 hold and X = (1 1By)/(11 Pyy), for all P., 1/2, the following
inequality holds
(g(2) (o(=) (H.)),) + (1-7 where H(p) -plog (1 p) log(1 p) is the binary entropy function.
Slightly different from Theorem I here it brings a coefficient before (P,)), which suggests that the implicit penalty on // (p,,) is stronger if (1~>/2) > 0. Meanwhile, we also require that < so that the optimization process always biased towards decreasing (2(2 as in SAM. Note that this argument is also valid for m-SAM as it increases the geometric mean as well. For notational convenience, we will refer to this variant of SAM as Calibrated SAM (CSAM) in the sequel, and its pseudocode is summarized in Algorithm [ (see Appendix A).

---

Published as a conference paper at ICLR 2026
(a) CIFAR-I0
(b) CIFAR- 100
(c) ECE evolution
(d) TCE evolution
Figure 3: (a)-(b) display the calibration performance of SAM and SGD (after various post-hoc processing) on CIFAR- R0/000000 datasets. (c)-(d) report the variation of ECE and TCE (namely, ECE after temperature scaling) during training.
5 EXPERIMENTS
In this section, we present the experimental results. We begin with the standard benchmarks showing that SAM significantly calibrates better than SGD, We further demonstrate on datasets including ImageNet- KK (Deng et al. 2009) that this calibration benefit is not limited to the in in-distribution (ID) data, but also translates to the out-of-distribution (OOD) data. last, we compare the proposed CSAM and SAM against a variety of baselines that attempt to reduce miscalibration. The results suggest that SAM is competitive and even superior to these approaches in many cases. More surprisingly, our proposed CSAM consistently outperforms SAM and achieves the lowest calibration error out of all baselines without deteriorating the generalization performance.
5.1 SAM ATTAINS A LOWER CALIBRATION ERROR THAN SGD
As a starting point, we first evaluate how SAM differs from SGD on the classical benchmarks for classification. The loss function defaults to be the standard cross-entropy (CE) loss, and train several neural networks, including ResNets (He et al. 2016). Wide ResNets (Zagoruyko & Komodakis 2016), and PyramidNets (Han et al. (017) to classify CIFAR-10/101 (Krizhevsky et al. 2009). As in common practice, we split the data into the train, validation, and test subsets so that the same validation subset used for hyperparameter tuning and post-hoc calibration. Without further specification, the optimizer is SGD with momentum 0.9, and the learning rate is scheduled in a cosine decay (Loshchilov & Hutter 2017). To conduct a fair comparison, we first make grid search of learning rate and weight decay coefficient
the model trained with SGD, and then apply them to SAM. The perturbation radius p is 0.05 for CIFAR- 10 and 0.2 for CIFAR- (see Appendix for more discussion on the effect of calibration).
As illustrated in Figure 3 (ba)-b), the ECE of SGD (red bar) is always much higher than the ECE of SAM (purple bar). This is more pronounced for ResNet-56 of SGD is approximately six times larger than the ECE of SAM. More surprisingly, we further observe that the uncalibrated ECE of SAM is generally smaller than the calibrated ECE of SGD by calibration methods such as temperature scaling (Guo et al.) 2017) and isotonic regression (Zadrozny)
CIFAR-I0/11 where the ECE
Blkan 2002). This indicates that SAM by itself tends generate accurate and reliable predictions.
Furthermore, as shown in Figure 3 c)-(), the superiority of SAM is persistent across the full training process. And the reduction of ECE more pronounced than TCE since SAM has already suppressed the over-confident outputs during training, and temperature scaling is thus not as effective as in SGD.
5.2 MODEL CALIBRATION UNDER DISTRIBUTION SHIFT
It is important for sufety-critice applications that the model not only produces reliable predictions for the in-distribution data but also is robust enough when there exists distribution shift between the training data and the test data. For this purpose, we first train ResNet- 18 on CIFAR- 10 using vanilla SGD and SAM, and then evaluate its performance on other datasets, including SVHN (Netzer) et al. 2) CIFAR-10/100-4 (Hendrycks & Dietterich, 2019). To enhance model uncertainty, we further encapsulate them with MC-Dropout (Gal & Ghahramani 2016) and Ensemble (Ovadia et al. 2019). Table shows that model ensembling and MC-Dropout both reduce ECE for SGD, SAM, and CSAM, but their gap is still significant- ECE of SGD approximately remains two times larger than ECE of SAM. This is different from their behavior on test accuracy, for example, SGD almost.
7

---

Published as a conference paper at ICLR 2026
Table Model performance on OOD data. The base model is ResNet-18 trained on CIFAR-IO. The size of MC-Dropout and Ensemble is 5.
Metrics
OOD AUROC
Test Ace 1
ECE
SVHN
CIFARIO-C
CIFAR100-C
Vanilla
89.18 + 0.26
5.76 0.43
83.94 + 0.96
62.26 + 4.46
83.26 0.71
SGD
MC-Dropout
89.13 + 0.18
4.39 0.27
84.11 = 0.69
57.09 + 2.81
82.11 0.82
Ensemble
90.88 + 0.11
1.84 + 0.22
86.41 + 0.36
63.39 + 4.72
85.81 0.17
Vanilla
90.01 0.23
3.24 0.39
86.38 0.39
63.32 + 4.77
84.83 0.83
SAM
MC-Dropout
89.49 = 0.33
2.21 0.41
83.02 = 0.35
56.11 2.50
80.96 0.35
Ensemble
91.16 + 0.14
1.09 0.22
88.05 = 0.21
64.03 = 4.95
86.84 0.75
Vanilla
89.95 + 0.16
2.55 +: 0.24
85.98 = 0.42
63.49 4.74
84.87 0.86
CSAM
MC-Dropout
89.57 + 0.21
1.52 + 0.21
82.82 = 0.31
56.05 + 2.48
80.70 0.55
Ensemble
91.22 0.17
0.86 0.17
88.21 0.12
64.17 + 4.91
86.92 0.70
Table 2: Results on the ImageNet- IK dataset. Slightly different from the custom setting, we reserve
20% of the ImageNet- 1K validation set as a new validation for early stopping and temperature scaling, and the remaining images therefore constitute test set. Both metries (TCE is short for ECE calibrated by temperature scaling, and AdaECE adaptive ECE) are evaluated on the test set.
ID Metries
OOD Metries
Ace 1
ECE |
TCE
AdaECE
AUROC 1
(1/2/3)
(1/2/3)
SGD
76.97
3.39
1.80
3.31
94,01
36.89
35.81
24.99
7.97
4.23
17.29
ResNet-50
SAM
77.32
1.52
1.54
1.44
94.35
37.45
36.35
27.85
4.91
3.74
6.92
CSAM
77.95
1.18
1.09
1.19
94.67
38.29
37.11
28.69
3.28
3.02
5.47
AdamW
65.03
9.11
2.63
9.11
88.63
33.53
32,87
26.48
14.73
12.28
19.57
VIT-S/32
SAM
69.21
3.04
3.05
91.01
37.95
36.09
33,36
3.35
6.27
7.89
CSAM
70.01
2.88
0.92
2.78
91.54
38.88
36.94
34.16
3,01
5.76
5.41
Adam W
71.35
9.72
3.66
9.72
90.61
37.40
35.54
24.26
14.14
12.27
18.63
VIT-S/16
SAM
75.42
93.27
2.92
3.58
5.02
CSAM
75.91
1,58
1.34
1.54
93.66
44.01
39.82
29.57
2,81
3.24
4.75
generalizes as well as SAM with Ensemble. On the other hand, it should be highlighted that SAM generalizes much better than SGD OOD data. And Ensemble also works well under this scenario. unexpected finding that MC-Dropout hurts both optimizers' performance on OOD data and more evident for SAM. One possible explanation is that the fusion of Dropout and SAM adversely increases model uncertainty, which, as result, impedes generalization.
Next, we train models on the clean ImageNet-1K dataset and then assess the calibration performance of SAM on the ImageNet-C (Hendrycks & Dietterich 2019) dataset, which consists of images that have been modified with several synthetic corruptions at five different severities. Following Minderer et al. (202T). we reserve 20% of the ImageNet-Ie 1K validation set for early stopping and temperature scaling. Moreover, also exclude the corresponding corrupted images ImageNet-C that are created from ImageNet- 1 K at the evaluation phase. We train one ResNet and two vision transformers (ViTs) (Dosovitskiy et al., [222]] on ImageNet-IK IK for 100 epochs and 300 epochs. The base optimizers are SGD and AdamW, and a cosine learning rate scheduler is used in all runs, As in previous studies (Foret al. 2021 Chen al. 2022), the perturbation radius p for ResNet and ViT is 0.05 and 0.2.
As shown in Table SAM and CSAM consistently improve the test accuracy on ImageNet- IK validation set, though being more pronounced for ViTs (~~ 4%). Meanwhile, ViTs are generally less calibrated than ResNet, which is somewhat inconsistent with the findings of (Minderer et al.) 2021). One explanation might be that their comparison is based on the pretrained neural networks rather than training them from scratch. But when models are trained by SAM, both of them achieve much lower calibration error, and their gap becomes negligible. For ImageNet-C, we consider three kinds of corruption: 1-motion blur, 2-defocus blur, and 3-impulse noise. For each kind of corruption, further average the accuracy and ECE across the five different severities. Consistent with previous findings, Table 2 also indicates that SAM generalizes better than SGD and that ViTs trained by AdamW also tend to be less calibrated on ImageNet-C. Interestingly, however, we observe
8

---

Published as a conference paper at ICLR 2026
Table 3: Performance comparison between different methods on CIFAR-10. The results are averaged
over 3 random seeds, with standard deviation displayed as well.
Test 1
ECE
ClasswiseECE
AdaECE
TCE 4
AUROC f
CE
95.83 + 0.21
2.36 = 0.11
0.52 + 0.01
2.04 = 0.11
1.06 0.19
98.68 + 0.04
Focal Loss (FL.)
95.91 + 0.02
1.16 0.13
0,38+ 0.01
1.42 = 0.09
1.01 + 0.28
99.04 + 0.01
DualFoccll
95.73 0.10
1.74 0.09
0.48 0.02
=
1.00
99.26 0.02
AdaFocal
95.78
0.91 0.14
0.01
0.65
0.97
99.10 0.04
Mixup
9.13
2.21 1.11
0.21
1.3
1.23
99.12 0.02
MIT-L
0.16
1.5 0.02
0.01
1
0.11
99.12 1 0.03
MMCE
95.94
2.47 0.04
0.04
2.02
1.15
98.65 0.05
BatchEnsemble
0.11
1.91 0.06
0.05
185
0.11
98.96 0.01
Rank1-BNN
95.50
1.92 0.29
0.06
1.94 0.29
0.51
98.81 +: 0.12
VI
0.10
3.14 0.12
0.09
3.06 = 0.15
0.76 1.08
98.28 1: 0.07
MIMO
2..9
0.88 0.06
0.01
0.73 0.08
0.74
99.16 0.01
ACLS
95.91 0.08
2.48 0.07
0.01
2 0.07
1.09 + 0.08
98.62 0.01
BalCAL
96.23 0.09
1.89 0.07
0.42 0.01
1.93 = 0.01
0.79 + 0.05
98.77 0.44
bSAM
96.45 + 0.03
1.82 0.10
0.43
1.78 0.10
0.70 = 0.23
98.95 + 0.06
SAM
96.91 + 0.14
0.86 0.13
0.26 0.02
0.84 0.14
0.52 = 0.09
99.30 + 0.02
CSAM
96,97 + 0.05
0.50 0.03
0.23 0.01
0.48 0.03
0.47 = 0.05
99.53 + 0.02
that while ViT-S/16-SAM generalizes and calibrates worse than ResNet-50-SAN it performs much better than the latter. This might arise from the different implicit biases of SGD and AdamW. W.
5.3 CSAM EVEN CALIBRATES BETTER THAN SAM
In this section, attempt compare CSAM and SAM against other popular baselines, including: focal loss (Mukhoti et al. 2020) that implicitly penalizes the gradient norms of confident examples and its two variants- variants-DuallFoce Focal (Tao et al. 2023) and AdaFocal (Ghosh al. 2022), mixup (Zhang) 2018) that implicitly performs label smoothing (Carratino et al. 2022) to avoid the overconfidence issue, MMCE (Kumar al., (018)) that acts as continuous and differentiable calibration error
regulariser, MIT-L (Wang et al. 20033 that involves mixup inference in training, BatchEnsemble (Wen al.) 2020), ACLS (Park al. 2023 BalCAL (Ni al. 2025), ) and several probabilistic (Dusenherry et al. 2020). VI (Ovadia et al. 2019), MIMO (Havasi et al.) 202T), and bSAM (Möllenhoff & Khan (023). The backbone WideResNet-22 10 (Zagoruyko Komodakis 2016), and generally follow the recommended setting to reproduce the results of each baseline. The perturbation radius p of SAM and CSAM is 0.2 for CIFAR- 1010000 and we vary the hyper-parameter of CSAM in (0.5, 1.0, 2.0).
approaches
we can observe that while focal loss generally hurts generalization, does reduce the calibration error. This observation also applies to the probabilistic approaches, such as RankI-BNN and MIMO. As a comparison, SAM significantly reduces the calibration error and is competitive,
From Tables and S5
superior other baselines in many cases. Note that the Bayesian variant, bSAM, does not perform better than SAM. The reason might be that it additionally introduces several hyperparameters, making it more difffcuult to tune and apply. In contrast, the proposed CSAM further decreases the calibration error while simultaneously achieving competitive generalization performance to SAM. And when compared other baselines, CSAM always achieves the lowest error, showing its versatility in generalization and calibration. While our current study is limited to cross-entropy loss, preliminary studies (see Table ST5) indicate that SAM/CSAM can further integrated with other training losses. More results, such as sensitivity analysis of the hyperparameters and comparison to other variants of SAM, can be found in Appendices and C
6 CONCLUSION
Besides its well-known generalization benefits, showed that SAM also excels at calibrating deep neural networks. We proved that SAM achieves this goal by imposing an implicit regularization on the negative entropy of the predictive distribution during training (see Equation (I]. which is similar to focal loss (Mukhoti et al. (020). We further proposed a variant SAM to improve calibration and validated its superiority across number of networks and datasets.

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported in part by the National Natural Science Foundation of China under Grants 12501710, 12301656, 62276208, 12326607 and 12371512, in part by the Natural Science Basic Research Program of Shaanxi Province under Grant 2024JC-JCQN-02 and in part by Fundamental Research Funds for the Central Universities.
DISCLOSURE OF THE USE OF LARGE LANGUAGE MODELS
The authors use large language models to assist with copyediting and to improve the overall readability of the text. The authors review and edit the manuscript and assume full responsibility for its final content.
REFERENCES
Maksym Andriushchenko and Nicolas Flammarion. Towards understanding sharpness-aware mini- mization. In ICML, pp. 639-668, 2022.
Maksym Andriushchenko, Francesco Croce, Maximilian Müller, Matthias Hein, and Nicolas Flam- marion. A modern look at the relationship between sharpness and generalization. In ICML, pp. 840-902, 2023.
Dara Bahri, Hossein Mobahi, and Yi Tay. Sharpness- aware minimization improves language model generalization. In ACL, pp. 7360-7371, 2022.
Peter L Bartlett, Philip M Long, and Olivier Bousquet. The dynamics of sharpness- aware minimiza- tion: Bouncing across ravines and drifting towards wide minima. Journal of Machine Learning Research, 24(316):1-36, 2023.
Devansh Bisla, Jing Wang, and Anna Choromanska. Low-pass filtering SGD for recovering flat optima the deep learning optimization landscape. In AISTATS, pp. 8299-8339, 2022.
Olivier Bousquet and André Elisseeff. Stability and generalization. Journal of Machine Learning Research, 2:499-526, 2002.
Jochen Bröcker. Reliability, sufficiency, and the decomposition of proper scores. Quarterly Journal of the Royal Meteorological Society, 135(643);1512--511)) 2009.
Luigi Carratino, Moustapha Cissé, Rodolphe Jenatton, and Jean-Philippe Vert. On mixup regulariza- tion. Journal of Machine Learning Research, 23(325): 2022.
Pratik Chaudhari. Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-SGD: Biasing gradient descent into wide valleys. Journal of Statistical Mechanics: Theory and Experiment, 2019(12); 2019.
Xiangning Chen, Cho-Jui Hsieh, and Boqing Gong. When vision Transformers outperform Resnets without pretraining or strong data augmentations. ICLR, pp. 1-20, 2022.
Jiacheng Cheng and Nuno Vasconcelos. Calibrating deep neural networks by pairwise constraints. In CVPR, pp. 13709-13718, 2022.
Pranav Singh Chib and Pravendra Singh. Recent advancements in end-to-end autonomous driving using deep learning: A survey, IEEE Transactions Intelligent Vehicles, pp. 103-118, 2023.
Enea Monzio Compagnoni, Luca Biggio, Antonio Orvieto, Frank Norbert Proske, Hans Kersting, and Aurelien Lucchi. An SDE for modeling SAM: Theory and insights. In ICML, pp. 25209-25253 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, pp. 248-255. 2009.
10

---

Published as a conference paper at ICLR 2026
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In ICML, pp. 1019-1028, 2017.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, pp. 1-21, 2021.
Jiawei Du, Hanshu Yan, Jiashi Feng, Joey Tianyi Zhou, Liangli Zhen, Rick Siow Mong Goh, and Vincent Tan. Efficient sharpness-aware minimization for improved training of neural networks. In ICLR, pp. 1-18, 2022.
Michael Dusenberry, Ghassen Jerfel, Yeming Wen, Yian Ma, Jasper Snoek, Katherine Heller, Balaji Lakshminarayanan, and Dustin Tran. Efficient and scalable Bayesian neural nets with rank-1 factors. In ICMI., pp- 2782-2792, 2020.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. In ICLR, pp. 1-20, 2021.
Futoshi Futami and Masahiro Fujisawa. information-theorettie generalization analysis for expected calibration error. In NeurlPS, pp. 1-52. 2024.
Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In ICML, pp. 1050-1059, 2016.
Arindam Ghosh, Thomas Schaaf, and Matthew Gormley, Adafocal: Calibration-awar adaptive focal loss. In NeurlPS, pp. 1583-1595, 2022.
Tilmann Gneiting, Fadoua Balabdaoui, and Adrian Raftery. Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society Series B: Statistical Methodology, 69(2): 243-268, 2007.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In ICML, pp. 1321-1330, 2017.
Dongyoon Han, Jiwhan Kim, and Junmo Kim. Deep pyramidal residual networks. CVPR, pp. 5927-5935, 2017.
Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In ICML, pp. 1225-1221 1234, 2016.
Marton Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snoek, Balaji Lakshmi- narayanan, Andrew Mingbo Dai, and Dustin Tran. Training independent subnetworks for robust prediction, In ICLR, pp. 1-13, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pp. 770-778, 2016.
Ramya Hobbalaguppe, Jatin Prakash, Neclabh Madan, and Chetan Arora. A stitch time saves nine: A train-tima regularizing loss for improved neural network calibration. In CVPR, pp. 16081-16090, 2022.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In ICLR, pp. 1-16, 2019.
Geoffrey E Hinton and Drew van Camp. Keeping neural networks simple. In IJCNN, pp. 11-18, 1993.
Gao Huang, Yixuan Li. Geoff Pleiss, Zhuang Liu, John E Hoperoft, and Kilian Q Weinberger. Snapshot ensembles: Train get for free. In ICLR, pp. 1-14, 2017a.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks, In CVPR, pp. 4700-4708, 2017b.

---

Published as a conference paper at ICLR 2026
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. In UAI, pp. 1-10, 2018.
Weisen Jiang, Hansi Yang, Yu Zhang, and James Kwok. An adaptive policy to employ sharpness- aware minimization. ICLR, pp. 1-19, 2023.
Xiaoqian Jiang, Melanie Osl, Jihoon Kim, and Lucila Ohno-Machado. Calibrating predictive model estimates support personalized medicine. Journal of the American Medical Informaties Association, 19(2):263-21 2012.
Jean Kaddour, Linging Liu, Ricardo Silva, and Matt J Kusner. When do flat minima optimizers work? In NeurlPS, pp. 16577-16595, 2022.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In ICLR, pp. 1-16, 2017.
Hoki Kim. Jinseong Park, Yujin Choi, and Jaewook Lee. Stability analysis of sharpness-aware minimization. arXiv preprint arXIv:2301.06308 2023.
Minyoung Kim, Da Li, Shell X Hu, and Timothy Hospedales. Fisher SAM: Information geometry and sharpness aware minimisation. In ICML, pp. 11148-11161, 2022.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Meelis Kull, Miquel Perello Nieto, Markus Kängsepp, Telmo Silva Filho, Hao Song, and Peter Flach. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration. In NeurlPS. pp. 1-11, 2019.
Aviral Kumar, Sunita Sarawagi. and Ujjwal Jain. Trainable calibration measures for neural networks from kernel mean embeddings. In ICML, pp. 2805-28144 14, 2018.
Selim Kuzucu, Kemal Oksuz, Jonathan Sadeghi, and Puneet K Dokania. On calibration of object detectors: Pitfalls, evaluation and baselines. ECCV, pp. 185-204, 2025.
Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and Kwon Choi. ASAM: Adaptive sharpness- aware minimization for scale-invariant learning of deep neural networks. In ICML, pp. 5905-5914, 2021.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In NeurIPS, pp. 1-12, 2017.
Bingcong Li and Georgios B Giannakis. Enhancing sharpness-aware optimization through variance suppression. In NeurlPS, pp. 1-19, 2023.
Bingyuan Liu, Ismail Ben Ayed, Adrian Galdran, and Jose Dolz. The devil is in the margin: Margin- based label smoothing for network calibration. In CVPR, pp. 80-88, 2022a.
Yong Liu, Siqi Mai, Xiangning Chen, Cho-Jui Hsieh, and Yang You. Towards efficient and scalable sharpness-aware minimization. In CVPR, pp. 12360-12370, 2022b.
Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In ICLR, pp. 1-16, 2017.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, pp. 1-18, 2019.
Peng Mi, Li Shen, Tianhe Ren. Yiyi Zhou, Xiaoshuai Sun, Rongrong Ji, and Dacheng Tao. Make sharpness-aweer aware minimization stronger: sparsified perturbation approach. In NeurIPS, pp. 30950-30962, 2022.
Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neural networks. In NeurIPS, pp. 15682-15694, 2021.
12

---

Published as a conference paper at ICLR 2026
Thomas Möllenhoff and Mohammad Emtiyaz Khan. SAM optimal relaxation of Bayes. In ICLR, pp. 1-25, 2023.
Jooyoung Moon, Jihyo Kim, Younghak Shin, and Sangheum Hwang. Confidence-aware learning for deep neural networks. In ICML, pp. 7034-7044, 2020.
Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr, and Puneet Dokania. Calibrating deep neural networks using focal loss. In NewrlPS, pp. 15288-15299, 2020.
Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? In NeurlPS, pp. 1-10, 2019.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated proba- bilities using Bayesian binning. In AAAI, pp. 1-7, 2015.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In NeurIPS Workshop, pp. 1-9, 2011.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In CVPR, pp. 427-436, 2015.
Jiani Ni, He Zhao, Jintong Gao, Dandan Guo, and Hongyuan Zha. Balancing two classifiers via simplex structure for model calibration. In CVPR, pp. 30712-30721, 2025.
Renkun Ni, Ping-yeh Chiang, Jonas Geiping, Micah Goldblum, Andrew Gordon Wilson, and Tom Goldstein. K-SAM: Sharpness-aware minimization at the speed of SGD. arXiv preprint 1xXiv:2210.128644 2022.
Alexandru Niculescu-Mizil and Rich Caruana. Predicting good probabilitics with supervised learning. In ICML, pp. 625-632, 2005.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanann and Jasper Snock. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In NeurlPS, pp. 1-12, 2019.
Hyekang Park, Jongyoun Noh, Youngmin Oh, Donghyeon Back, and Bumsub Ham. Acls: Adaptive and conditional label smoothing for network calibration. In ICCV, pp. 3936-3945, 2023.
Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. In ICLR Workshop, pp. 1-11, 2017.
John Platt al. Probabilistic outputs for support vector machines and comparisons regularized likelihood methods. Advances in Large Margin Classifiers, 10(3):61-74, 1999.
Sameera Ramasinghe, Lachlan Ewen MacDonald, Moshiur Farazi, Hemanth Saratchandran, and Simon Lucey. How much does initialization affect generalization? In ICML, pp. 28637-28655. 2023.
Aditya Singh. On deep neural network calibration by regularization and its impact on refinement. In ICLR, pp. 1-15, 2021.
Chengli Tan, Jiangshe Zhang, Junmin Liu, and Yihong Gong. Sharpness-aware Lookahead for accelerating convergence and improving generalization. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1-14, 2024a.
Chengli Tan, Jiangshe Zhang, Junmin Liu, Yicheng Wang, and Yunda Hao. Stabilizing sharpness- aware minimization through simple renormalization strategy. arXiv preprint arXiv:2401.07250. 2024b.
Linwei Tao, Minjing Dong, and Chang Xu. Dual focal loss for calibration. In JCML, pp. 33833-33849, 2023.
Sunil Thulasidasan, Gopinath Chemnupati, Jeff A Bilmes, Tanmoy Bhattacharya, and Sarah Michalak. On mixup training: Improved calibration and predictive uncertainty for deep neural networks. In NeurIPS, pp. 1-12, 2019.
13

---

Published as a conference paper at ICLR 2026
Iya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mlp architecture for vision. In NeurIPS, pp. 24261-24272, 2021.
Cheng Wang. Calibration in deep learning: A survey of the state-of-the-art. arXiv preprint arXiv:2308.0122,, 2023.
Deng-Ban Bang-Bao Wang, Lei Feng, and Min-Ling Zhang. Rethinking calibration of deep neural networks: Do not be afraid of overconfidence. In NeurIPS, pp. 11109-11820, 2021a.
Deng-Bao Wang, Lanqing Li, Peilin Zhao, Pheng-Ann Heng, and Min-Ling Zhang. On the pitfall of mixup for uncertainty calibration. CVPR, pp. 7609-7618, 2023.
Xingyu Wang. Sewoong Oh, and Chang Han Rhee. Eliminating sharp minima from SGD with truncated heavy-tailed noise. In ICLR, pp. 1-157, 2021b.
Kaiyue Wen, Tengyu Ma, and Zhiyuan Li. How does sharpness-awant minimization minimizes sharpness? In NeurlPS Workshop, pp. 1-94, 2022.
Kaiyue Wen, Zhiyuan Li, and Tengyu Ma. Sharpness minimization algorithms do not only minimize sharpness to achieve better generalization. In NeurlPS, pp. 1-12, 2024.
Yeming Wen, Dustin Tran, and Jimmy Ba. BatchEnsemble: an alternative approach to efficient ensemble and lifelong learning. In ICLR, pp. 1-19, 2020.
Tengji Xu, Zeyu Luo, Shaojie Liu, Li Fan, Qiarong Xiao, Benshan Wang, Dongliang Wang, and Chaoran Huang. Perfecting imperfect physical neural networks with transferable robustness using sharpness-aware training. arXiv preprint arXiv:2411.12333 2024.
Yun Yue, Jiadi Jiang, Zhiling Ye, Ning Gao. Yongchao Liu, and Ke Zhang. Sharpness-aware minimization revisited: Weighted sharpness as a regularization term. In KDD, pp. 1-10, 2023.
Bianca Zadrozny and Charles Elkan. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In ICML, pp. 609-616, 616, 2001.
Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass probabil- ity estimates. In KDD, pp. 694-699, 2002.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv: 2016.
Hongyi Zhang. Mixup: Beyond empirical risk minimization. In ICLR, pp. 1-13, 2018.
Michael Zhang, James Lucas, Jimmy Ba, and Geoffrey E Hinton. Lookahead optimizer: k steps forward, step back. In NeurlPS, pp. 1-19, 2019.
Zhiyuan Zhang, Ruixuan Luo, Qi Su, and Xu Sun. GA-SAM: Gradient-strength based adaptive sharpness aware minimization for improved generalization. In EMNLP, pp. 3888-3903, 2022.
Yang Zhao Hao Zhang, and Xiuyuan Hu. Penalizing gradient norm for efficiently improving generalization in deep learning. In ICML, pp. 26982-26992, 2022a.
Yang Zhao, Hao Zhang, and Xiuyuan Hu. Randomized sharpness-aware training for boosting computational efficiency in deep learning. arXiv preprint arXiv:2203.09962. 2022b.
Yaowei Zheng, Richong Zhang, and Yongyi Mao. Regularizing neural networks via adversarial
model perturbation. In CVPR. pp. 8156-816 2021.
Pan Zhou, Hanshu Yan, Xiaotong Yuan, Jiashi Feng, and Shuicheng Yan. Towards understanding why Lookahead generalizes better than SGD and beyond. In NeurlPS, pp. 27290-27304, 2021.
Yixuan Zhou, Yi Qu, Xing Xu, and Hengtao Shen. ImbSAM: closer look at sharpness-aware minimization in class lass-imbalanced recognition. In ICCV, pp. 11345-11355 2023.
Dongyao Zhu, Bowen Lei, Jie Zhang, Yanbo Fang, Yiqun Xie, Ruqi Zhang, and Dongkuan Xu. Rethinking data distillation: Do not overlook calibration. ICCV, pp. 4935-4945, 2023.
14

---

Published as a conference paper at ICLR 2026
Algorithm 1 CSAM Optimizer
Input: Training set = (z; (z,, 1/ 1= objective function (0) initial weight Og E Rdd learning rate 7 > 0, perturbation radius p > 0, training iterations T, regularization coefficient
y> 0, and base optimizer A (e.g. SGD)
Output: 0r
1: for 0,,,
2:
Sample mini-batch 22 {zf. 2m ]:
3: Compute cross-entropy loss Ln, (0t) 2,, En, L
4: Compute perturbed weight 0 - 0, + VV, Lo, (0+)ll
Vo, La. (0.)
5:
Compute perturbed loss Ln, (0.)
Ex
(z(z)) per Equation 4
6:
Compute gradient 9, = VoLn, (0,) o= of the loss over the same 94;
7: Update weight with base optimizer A, e.g. Oc+1 0. 19
8: end for
-50
egooh
1000
Jo0th eguch
1.1
150
Raue (0)
-50
175
1.0
0.2
0.00
0.8
100
Epods
Epock.
(a)
(b)
(c)
Figure S4: To verify whether the boundedness assumption of Kimnn (see Lemmas 1 and 2 holds for realistic neural networks, we further trained ViT on the realistic ImageNet- IK dataset. (a) compares the value of Kmin at the two endpoints O and i throughout the training process. (b) further illustrates how Kmin evolves along the path from O(t 0) to O(t 1) 50th, 200th, and 300th epoch, respectively. (c) records how the true coefficient À and its lower bound Ån (see Equation vary during training.
A THEORETICAL PROOFS
In this section, we first present the pseudocode of CSAM (Algorithm) D and then the missing proof's in Section We also validate the boundedness assumption on the ImageNet-IK dataset, shown in Figure S4
Proof of Lemmal
According to the Taylor theorem, there always exists some o' such that
log Py (0(2) (=(=) (Ō- 0)² V(o(=) マ/ (0 ( 0)-
Since 0 0 1V ( and Kminn (V2lo( > -V it follows that
log p. (=() > (0(2) + ((o + 2 Kmin (F) - log P, + [
thus concluding the proof.
Proof of Theorem
T It follows from Lemma [ that Py I Py- Recall that
(2)2) -log P, (z(22 + log
Py p
(0(2) + P, log Ps
+ (1 (l,))) log
1 P
D
1 P,
1 Dy H (py) H())):
lo(z)
Py
thus concluding the proof.
15

---

Published as a conference paper at ICLR 2026
Proof of Lemma
2
There always exists some O' E Rd such that
1/m
m log p.
- -log II py.
m7
(n)
1 Ln(() + (0 0) T (L) + (o (0-0) 0) (2
2 (6-
A similar argument as Lemma concludes the proof.
Proof of Theorem 2 yields that
The proof is straightforward. According to the definition of Py and py
(0) log Py (z) + log
Py Py
L0) + Py log
Py
+ (1-Py) log Py 1 Py
Py
1 Pv HP,)) ) + (by),
2 (0x)))
1 Pv
thus completing the proof.
Proof of Theorem Recall that
(2) -(1 -1+ + p,) 7 log Pu 2 -(1 Ppn) log P.y
= =b((2) + TD, log Py,
1 Py (P,)) + (P.,) + +,p,
(z)2)
1 Py
1
Py H(,,)) + (D,)) + 3o log P.y + -1,,,, log(1 (.)))
(0(=)
1
Py
2
2
1
Py H(,,) + (1 - 7 HP,))
= (x(2)
1 P
2
thus concluding the proof.
B EFFECTS OF PERTURBATION RADIUS AND COEFFICIENT y
The perturbation radius p is an important factor in determining the generalization performance (Foret et al., (02T), but its effect model calibration remains unknown. To answer this question, we conduct another set experiments while varying the perturbation radius p from 0.02 to 0.2, an interval in which the optimal value of p is often found. Figure SS5 shows that the entropy of the predictive distribution (P,,) continues to increase for both models and datasets as expected. However, we also observe that for both models the test accuracy CIFAR-10 first increases and then decreases with the perturbation radius p. though the test accuracy on CIFAR- 100 keeps increasing in this interval. This implies that larger values of p do not assure better generalization. On the other hand, the ECE on CIFAR-I0 first decreases and then increases with the perturbation radius. Moreover, the ECE of ResNet-56 higher than that of ResNet-20 in the descending regime, which is aligned with the previous finding that increasing capacity by width or depth may hurt model calibration (uuo et al. 2017). Meanwhile, when the perturbation radius exceeds the changing point, the ECE of ResNet-20 undergoes sudden rise and becomes higher than that of ResNet-56, phenomenon that more pronounced for CIFAR- 10 in this interval. One explanation for this observation might be that models with low capacity more amenable to the implicit regularization imposed by SAM. The key point is that the perturbation radius p should be relatively small to simultancously achieve a lower ECE and higher test accuracy than SGD.
And below we present how the additional hyperparameter y of CSAM affects the final generalization and calibration. The base network is ResNet-56 trained on CIFAR-10, 10, and the perturbation radius p is 0.05. We sweep y over {0, 0.5, 1.0, 1.5, 2.0, 2.5, 3.0) and when 7 = 0, CSAM degenerates to the standard SAM. As shown in Figure S6
observe that when 7 0.5, CSAM improves
16

---

Published as a conference paper at ICLR 2026
accuracy (%)
ECE (%)
Entropy (P.)
CIFAR-10
95
20
2
10
0.1
0.2
0.1
0.2
0.1
0.2
RecNee.5e 56
CIFAR-1O1
RecNNe no
30
5
20
0.1
0.2
0.1
0.2
0.1
0.2
Perturbation radius
Perturbation radius
Perturbation radius
Figure S5: Variation of different metrics for models trained under monotonically increasing pertur- bation radius p. Note that Py indicates the predicted probability associated with the true label in one-hot encoding, and H (py) is the corresponding entropy.
both the generalization and calibration. And the lowest value of ECE attained when 1, but the test accuracy slightly decreases. In contrast, increasing y up significantly deteriorates the performance. Therefore, relatively smaller value of 7 is preferred.
Note that the perturbation radius p has important impact À that controls the weight of the entropy term - H (- To investigate the interaction between p and the hyper-parameter 7/s we trained number of ResNet-56 models on CIFAR-10/100 using differentt choices of p and 7. Namely, p from [0.05, 0 } and 7 from (0.0, 0.5, 1.0, 1.5, 2.0 Note that when y = 0.0, CSAM reduces to the standard SAM optimizer. From Table S4 we observe that CSAM can always generalize and calibrate better than SAM, when and are carefully tuned. Moreover, when training with a small value of is suggested to combine with a relatively large value of y, and vice versa. This is because when p is large, the penalty coefficient À in Theorem is also very large. Using large in this case will over-penalize the examples, which, as a result, adversely affects the generalization and calibration.
2.5
94.85
accuracy
2.0
94.80
94.75
1.0
94.70
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Regularization coefficient
Regularization coefficient
Figure S6: Effects CSAM hyperparameter y on test accuracy and ECE,
C MORE EXPERIMENTAL RESULTS ON CSAM
In this section, we include several networks like ResNets (He et al., 2016), Wide ResNets Zagoruyko Komodakis 2016), and PyramidNets (Han et al.) 2017) to classify CIFAR-10/100. Further- more, the classical ResNet-18 18 for ImageNet-IK is further adapted to classify Tmay-Im The initial learning rate and the weight decay coefficient are swept over (0.01, 0.05, 0.1) and (1.0e- mini-batch size of 128. The optimizer SGD with momentum 0.9, and the learning rate is scheduled in a cosine decay (Loshchilov GPU cluster with 2 cards, and it
4. 5.0e-4, 1.0e-3), respectively. By default,
Hutter 2017). Note that all experiments are
17

---

Published as a conference paper at ICLR 2026
Table S4: number ResNet-56 were trained CIFAR-10/100 datasets with varying p and 7.
yy = 0.0
7= 0.5
==
y= 1.5
? 2.0
0.05
94.67 + 0.06
94.61 + 0.09
94.62 0.14
94.67 0.14
94.74 + 0.06
Test Acc
0.1
94.74 = 0.05
94.71 + 0.07
94.90 = 0.17
94.66 0.09
94.79 = 0.05
CIFAR-I0
0.05
1.77 1 0.04
1.46 1 0.14
1.09 0.07
0.66 1 0.06
0.65 0.14
BCE
0.1
0.82 + 0.12
0.54 + 0.11
0.69 + 0.14
1.34 + 0.33
2.04 + 0.17
0.05
73.52 0.42
73.48 0.31
73.55
0.18
73.57 0.46
73.67
0.18
Test Ace
0.1
73.94 0.09
74.16 + 0.17
73,96 0.22
74.12 0.09
73.86 0.39
CIFAR-I00
p = 0.05
8.83 + 0.15
7.52+ 0.14
6.24 0.28
4.86 + 0.37
3.67 0.12
ECE
0.1
6.17 = 0.18
+ 0.40
1.84 0.14
1.42
quire approximately 1500 GPU hours in total. As shown from Table S6 to Table STO CSAM consistently performs better than SAM, and it surpasses other baselines as well. The code to reproduce these results is available at httpt::t:///// /ttttps///riee.ggggggg google www. TuMfEhsP Zo- Ho
com./arive/Eolders While other variants of SAM potentially altered the loss landscape, their effects on calibration are supposed be similar to SAM. This could be seen from Tables STI and S12 which summarize the results on CIFAR-10/ 10/100 Both
(06upa077scceEErGGnee Iww.AAEErreeeetttt.......e
SAM variants, including ASAM (Kwon et al. 202T) and VASSO (Li Giannakis 2023
can reduce
the calibration error, though ASAM generalizes much worse than SAM.
To further demonstrate the efficacy SAM on calibration, we also compare the result against several calibration-oriented training losses such as Label Smoothing (LS) and its variants, c.g., MBLS (Liu cct al. (022a) and ACLS (Park et al. 2023). Besides them, popular approaches such as CPC (Cheng & Vasconcelos 2022), MDCA (Hebbalaguppe al. 2022). and CRL (Moon et al. 2020) included. As shown in STT3 both Label Smoothing and its variants can reduce the ECE. But, unfortunately, they also hurt the generalization performance. In contrast, SAM and CSAM not only reduce the ECE but also significantly improve the test accuracy. Moreover, we evaluate these methods ImageNet-11 K well. The base model is ViT-S-32 and all runs are trained only once due to limited time. Table S14 suggests that SAM/CSAM perform much better than other methods, both in generalization accuracy and calibration ECE, ID and OOD settings.
also
We are also interested in whether CSAM results in a flatter minima. For this purpose, a number of ResNet-56 models are trained on CIFAR-100 with different optimizers. Table T indicates that CSAM can reach a flatter minima without compromising the calibration performance. Moreover, we also examine with Focal Loss and Label Smoothing, both of which can significantly reduce the ECE value. It is interesting to find that they both converge to flatter minima as well. This indicates a positive correlation between calibration and sharpness.
Table S5: Performance comparison between different methods CIFAR-100. The results averaged over 3 random seeds, with standard deviation displayed as well.
Test Acc
ECE
ClasswiseECE
AdaECE
TCE
AUROC1
81.01 0.11
3.95 : 0.28
0,21 0.01
3,86 0.22
0.22
0.41
93.93
0.05
Focal Loss (FL)
80.55 0.17
2.84 0.36
0.09
2.79=0.45
0.36
94.43 0.01
DualFocal
80.74
2.68 0.51
0.08
2.66
2.24
94.81 0.17
AdaFocal
0.11
2.58 0.31
0.01
0.37
2.29
93.75 0.05
Mixup
0.26
4.28 0.27
0.02
4.24=0.31
0.20
94.35 0.08
MIT-L
0.
3.26 0.118
0.18
3.24 0.19
0.49
94,76 0.12
MMCE
0.05
4.02 0.29
0.18
3.96
93.83 = 0.07 0.39
BatchEnsemble
0.111
6,86 0.21
0.21
6.77
2.11
94.15 1 0.02
BaNN1-BNN
3.59 0.06
0.19
=.08
0.11
94,29 = 0.06
VI
76.30
10.29
0.03
0.11
2.08
92.62 1 0.08 0.35
MIMO
2.38 0.133
0.17
2.01
2.04
95.14 + 0.04
ACLS
6.38 0.119
0.1910011
6.1
1.47
93.16 1 0.14
BaICAL
81.34 0.02
5.69 + 0.18
0.88+0011
5.66 0.24
2.64 0.21
93.49 0.09
6SAM
80.59 0.07
0.07
8.27 1 0.13
0.20
8.27 0.14
2.59 0.17
94,01 1 0.11
SAM
82.93 + 0.15
2.11 + 0.17
0.00
2.17 = 0.21
1.89 + 0.11
94.15 + 0.06
CSAM
83,07 1: 0.19
1.93 + 0.15
0.15
1.99 = 0.05
1.54 1 0.30
96.07 1 0.03
18

---

Published as a conference paper at ICLR 2026
Finally, apart from the cross-entropy loss, are also wondering whether the calibration benefit of SAM persistent across other training losses. As shown Table we can observe that when integrated with Focal Loss (FL) and ACLS (Park al. 2023). SAM still calibrates better than the baseline, whereas CSAM achieves the lowest ECH/TC..
Table S6: Results (mean-tstd) test accuracy (%) over 3 random runs. Text marked as bold indicates the best result.
CR
FL
DualFocal
AdaFical
Misup
MMCE
MIT-L
SAM
CSAM
PenamilNet
72.06
0.28
004
72:00
0:88
14.15
72.17
0.43
1:33
74.95
CIBAR-ICO
WRN-28-10
$1.04
0.11
80.55
2.117
50.74 0.34
a0.70
0.11
82.00 = 0.26
51.02 0.85
81.29 0.18
$2.93 2:0.15
$3.05
0.19
PynmoNNa-111 10
61.31
0.52
4
61766:0007
81.81
0.38
82.94
0.29
$1.30 0.3.1
K7.41 1 000
$4.0% 0:29
84.16
0.15
Thy tmogeeNe
ResNer-18
51.96
0:35
5261 0.59
5500 0.85
30.36
0.06
51.45
0.70
51.31 0.7.
51.97 0:24
36.81 0:33
57.13 0.96
Table S7: Results (tt) of ECE (%) with M = 15 over 3 random runs. Text marked as bold indicates the best result.
CF:
F1.
DualFors
AdaFacil
Misup
MMCE
MIT-I
SAM
CSAM
ResNea-t6
1:11
0.12
0.03
0.89
3.6 0.17
0.54
0.67
CIRAR-10
WRN-22--10
0.09
0.91
1y11niaNes-110
0.19
0.05
0.12
2:23
13.29 0.15
1.71
2:11
PyrsmiNNe-1110
1esNer-18
4.35
06:30:00.5
10.81
9.34
2.75
Table S8: Results (tea of Classwise ECE (%) with M 15 over random runs. Text marked as bold indicates the best result.
Misup
MMCE
WRN-228-10
PyramiNNes110
0.34:::000
0:00
CIFAR-100
WRN
PyramiNNet
0.01
0.03
Thyy ImageNee
0.01
0.01
0.01
19

---

Published as a conference paper at ICLR 2026
Table S9: Results (tean=std) of Adaptive ECE (%) with M = 15 over 3 random runs. Text marked bold indicates the best result.
CE
FL
DualFocol
AdaFacal
Mixup
MMCC
MIT-L
SAM
CSAM
ResNea-5n
321 + 0.06
2.1-- 0.19
2.19 0.18
1003 +03.4
3.97 0:08
3.55 +.1.5
1.11+-0.14
0:50 +
0.51 0.8
CIFAR-10
WRN-25510
2356 + 0.11
1.2 0.09
4.64 0.7
0.6 + 0.004
4.63 1.04
2.42 +:04
+.50+0000
0.54 = 0.14
0.48 0.84
PynmichNet
2.5 0.19
1.78 1 0.00
4,54 a 0.02
0.8 a 0.00
2.09 0.22
2.49 1..1..011
1.19
0.70 0.05
0.19 0.00
ReaNes-t
0:06
188
2.48
0000
0.6
CIFAR-100
WRN-22-110
2.61 +:337
4.67
111
10.58
1.76
1.01
9.18
0.0
ImageNel
Table S10: Results (mean of AUROC (%) over 3 random runs. Text marked as bold indicates the best result.
F1.
DualFrcca
AdoiPaca
Misup
MMCE
MIT-1
SAM
CSAM
ReiNra-Sh
V7.88 0035
98:47 07 105
96 75 0109
98.23
0.01
us.
0.00
M6.00 0.00
96.72 D00
99117
0.07
EUI2 99.80
99.19
CIFAR 10
WRN 28-10 PymmithNa--110
98.68 0.00
99.04 0.00
99.26 0.02
90.10 0.04
99:12 = 0.00
98,65 0.05
99.12.2:000 0.0
99.30 0.02
0.01
0.04
90:0- 0.04
90000
0002
RoNN1136
01.06
92.32
0.00
92.57
PenaiidNett
Thyy Taa--EaageNa
RoNNa-188
K2.62 0.37
83.31
51.05
0.40
31.32
0.65
79.32 =: 0.53
82.57 a: 0.06
62.38 0.13
85.63 a
$5.09
0.15
Table S11: Calibration performance of different SAM variants on CIFAR-10.
Test Acc
ECE
ClasswiseECE
AdaECE
TCE
AUROC
SGD
94.26 0.06
3.52 0.12
0.78 = 0.01
3.52 = 0.11
1.03 + 0.01
98.09 = 0.08
SAM
94.50 0.20
1.83
1 0.08
0.01
1.78 0.03
0.70 0.12
98.78 0.08
ASAM
94.82 0.11
2.03
0.11
0.48
0.01
2.01 0.14
0.61 + 0.05
98.71 + 0.02
VASSO
94.70 0.10
1.75
- 0.04
0.42 + 0.01
1.69 - 0.05
0.71 + 0.09
98.87 - 0.06
CSAM
94.58 0.14
1.47
0.17
0.41 0.02
1.41 0.19
0.72 + 0.20
98.87 + 0.06
Table S12: Calibration performance of different SAM variants CIFAR-100.
Test Acc
ECE
ClasswiseECE
AdaECE
TCE
AUROC
SGD
72.00 : 0.15
13.14 +: 0.25
0.33 0.01
13.13 + 0.25
1.64 1 0.09
91.12 1 0.11
SAM
74.87 = 0.21
1.59 + 0.10
0.17 1 0.01
1.39 1 0.16
1.38 0.11
94.40 1 0.01
ASAM
74.11 0.11
6.48 + 0.26
0.22 +
6.43 0.25
1.31 0.31
93.13 0.09
VASSO
74.94 = 0.53
1.55 0.25
0.17 0.01
1.55 + 0.24
1.43 0.23
94.31 + 0.07
CSAM
74.85 = 0.20
1.31 0.40
0.17 + 0.01
1.22 + 0.23
1.36 = 0.44
94.55 + 0.06
Table $13: Comparison against other calibration-orrented training losses.
Test Ace
72.
0.12
72 +018
72.49 + 0.08
72.36 + 0.18
72.45 + 0.47
72.42 0.24
72 38 0.48
74.63 + 0.24
74.98
0.16
ECE
13.03
0.26
4.38 + 0.14
2 + 0.40
3.43 + 0.25
2.62 = 0.54
2.33 + 0.42
3,16 + 0:17
1.87 0.25
1.25 + 0.15
Table S14: Results on the ImageNet- IK dataset. Slightly different from the custom setting, we reserve 20% of the ImageNet-1K validation as new validation for early stopping and temperature scaling, and the remaining images therefore constitute test set. Both metrics are evaluated on the test (1/2/3 indicate different types of corruption).
ID Metries
OOD Metries
Test Acc f
ECE 1
Test Ace (1/2/3) 1
ECE (1/2/3) J
SGD
65.91
11.56
33.54
30.21
34.18
23.70
24.65
18.00
MBLS
67.19
2.21
34.54
31.58
35.58
11.83
12.61
8.58
CPC
65,68
9.34
32.34
31.01
35.71
20.46
23.19
15.04
ACLS
66.68
3.97
34.67
31.88
36.00
15.72
16.17
10.25
MDCA
65.23
8.38
33.11
29.91
33.71
23.31
24.28
17.15
SAM
69.48
2.47
38.25
35.27
39.54
7.33
7.77
3.15
CSAM
69.78
1.79
38.58
35.46
39.85
7.07
7.70
2.76
20

---

Published as a conference paper at ICLR 2026
Table S15: Performance of CSAM/SAM when combined with other training losses (e.g. FL. and ACLS).
ResNet-56
WRN-28-10
Test Ace
ECE
TCE
Test Acc f
ECE 4
TCE
FL
71.96 0.28
8.25 + 0.23
3.27
0.16
80.55 = 0.17
2.84 + 0.36
2.75 - 0.36
FL+SAM
73.11 0.01
6.90 0.38
1.90
0.13
81.33 0.23
2.41 + 0.06
1.76
FL+CSAM
73.51 = 0.10
0.17
1.58 + 0.13
82.08 = 0.08
1.88 - 0.11
1.42 0.16
ACLS
72.55 0.08
3.29
0.25
8.149
6.38 0.29
2.40
ACLS+SAM
74.53 = 0.09
1.37 0.05
1.27
83.04
+.10
1.122
ACLS+CSAM
74.86 0.07
1.01 0.12
0.85 + 0.05
83.13 0.08
1.37 + 0.21
1.36 + 0.06
Table $16: Comparison of sharpness (measured by the largest eigenvalue Kmaz of the Hessian) of different optimizers.
SGD
Focal Loss
Label Smoothing
SAM
CSAM
ECE
13.03 + 0.26
1.85 + 0.09
2.31 + 0.15
1.27 + 0.25
1.08 + 0.12
Kyar
631.89 86.81
496.99 + 22.02
583.74 52.08
177.99 + 13.25
148.16 16 10.09
21


Reviews:
[{"review_id": "RQGiJrKLVj", "reviewer": "Reviewer_pqTz", "summary": "This paper studies why Sharpness-Aware Minimization (SAM), beyond its known generalization benefits, tends to produce better-calibrated models. The authors show theoretically that SAM implicitly maximizes the entropy of the predictive distribution, discouraging overconfident outputs. Thus, they propose Calibrated SAM (CSAM), a simple variant that further suppresses overconfident examples through a lightweight modification to the loss. Experiments on CIFAR-10/100 and ImageNet-1K confirm that SAM already improves calibration compared to SGD or AdamW, and CSAM achieves the lowest Expected Calibration Error (ECE) without sacrificing accuracy.", "questions": "See weaknesses", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "## Strengths\n\n* This paper provides a simple and practical variant. CSAM requires only a minor change and no additional computation, yet consistently improves ECE.  \n* The experiments are broad, covering both in- and out-of-distribution evaluations across convolutional and transformer-based models.  \n* The finding that SAM alone can outperform post-hoc calibration methods like temperature scaling is practically valuable for reliability-sensitive applications.  \n* The paper gives a clear mathematical explanation of why SAM tends to yield better-calibrated predictions, connecting it to implicit entropy regularization.", "weaknesses": "## Weaknesses\n\n* The empirical effect has been reported before. The observation that SAM improves calibration is not new—Zheng et al. (2021) showed similar improvements in long-tailed recognition, and the original SAM paper (Foret et al., 2021) also mentioned better reliability qualitatively.\n* The theoretical assumptions are strong. The derivations rely on smoothness and bounded-Hessian assumptions, which may not fully hold for complex networks, limiting the generality of the proofs.\n* While CSAM consistently reduces calibration error, its advantage over SAM is small.\n\n**References**\n\n* Foret, P., Kleiner, A., Mobahi, H., & Neyshabur, B. Sharpness-Aware Minimization for Efficiently Improving Generalization. ICLR, 2021.  \n* Zheng, Z., Yang, X., Wang, Y., Li, Z., Liu, Y., & Zhang, T. Improving Calibration for Long-Tailed Recognition. CVPR, 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kE2HQZfNWl", "reviewer": "Reviewer_nCsi", "summary": "This paper investigates the calibration benefits of Sharpness-Aware Minimization (SAM) in deep neural networks. The authors provide a theoretical analysis showing that SAM implicitly regularizes the predictive distribution by encouraging higher entropy, thereby mitigating overconfidence. Specifically, they derive a lower bound and building on this insight they propose Calibrated SAM (CSAM), a variant that further enhances calibration by adaptively reweighting overconfident predictions. Extensive experiments on CIFAR, ImageNet-1K, and corrupted datasets demonstrate that both SAM and CSAM achieve significantly lower Expected Calibration Error (ECE) than standard training and existing calibration methods, without sacrificing accuracy.", "questions": "(1) The assumption on the Hessian’s smallest eigenvalue (Lemma 1) is only validated for ResNet-56 on CIFAR-10 (Figure 2). Given that the theoretical conclusions rely on this assumption, do the authors have evidence—e.g., from preliminary experiments on ImageNet or ViTs—that this behavior (linear decay of  $k_{min}$ along $\\theta$ to $\\tilde{\\theta}$) holds more broadly? If not, could they discuss potential failure modes where the assumption breaks down and how that might affect SAM’s calibration benefits?\n\n(2) In the ECE estimator (Section 3.1, lines 177–178), the expression for $\\text{acc}(B_i)$ appears to miss a summation over samples in the bin. Is this a typographical error? If yes, could it be corrected for formal precision?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "(1) The paper’s key strength lies in its originality and significance: it provides the first formal theoretical explanation for why Sharpness-Aware Minimization (SAM) improves calibration—linking it to implicit entropy maximization—offering a principled understanding beyond empirical observation. This insight bridges optimization geometry and uncertainty quantification, a valuable contribution to both communities. \n\n(2) The proposed CSAM variant is a high-quality and practical extension that consistently improves calibration across diverse architectures and datasets, including ImageNet-1K and distribution-shifted settings, without sacrificing accuracy. \n\n(3) The paper is also clearly written, with well-structured theory, intuitive figures (e.g., reliability diagrams, Hessian analysis), and thorough experiments that effectively support its core message.", "weaknesses": "(1) Missing summation symbol in the ECE estimator (Lines 177–178). The definition of bin accuracy in the Expected Calibration Error (ECE) computation omits the summation over samples in bin. I think the correct expression should be  $\\text{acc}(B_i) = \\frac{1}{|B_i|} \\sum_{z_j \\in B_i} \\mathbb{I}[y_j = \\arg \\max f_\\theta(x_j)]$.\n\n(2) The theoretical analysis (Lemma 1 and Theorem 1) relies on an assumption about the lower bound of the smallest Hessian eigenvalue along the linear interpolation between $\\theta$ and $\\theta'$. However, this assumption is only verified on a single model (ResNet-56) and a single dataset (CIFAR-10) in Figure 2. The generality of this assumption—and thus the applicability of the theoretical conclusions—remains unclear for other architectures (e.g., ViTs) or more complex datasets (e.g., ImageNet).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jCQCFfgFhE", "reviewer": "Reviewer_AfCQ", "summary": "This study primarily explores the relationship between the probability-based loss output at the point of maximum loss within the perturbation radius 𝜌 and the loss-to-probability mapping under the current model parameters. This analysis establishes a theoretical connection between model calibration and Sharpness-Aware Minimization (SAM), elucidating their underlying mechanisms. Building upon these findings, the authors propose an improved method aimed at enhancing model calibration.", "questions": "1. Table S4 shows improved calibration results when 𝛾=0. Does this suggest that the proposed enhancement does not significantly improve calibration performance? At  𝛾=0, the method reduces to the standard SAM framework.\n2. The conclusion states: “We proved that SAM achieves this goal by imposing an implicit regularization on the negative entropy of the predictive distribution during training, which is similar to focal loss.” Could the authors clarify which mathematical expression supports this statement? No corresponding equation appears in the theoretical derivations or references. A more detailed explanation would be appreciated.\n3. Since the proposed method is derived from the theoretical analysis, can the assumptions established in the theory be satisfied in practical applications? In other words, do the conclusions derived under these assumptions hold in real-world optimization scenarios?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Both SAM and model calibration represent important research directions, and exploring their mutual influence holds significant scientific value.\n2. The study demonstrates strong completeness by beginning with theoretical foundations to explain their respective roles and subsequently extending the analysis to methodological applications.", "weaknesses": "1. Although Theorems 1, 2, and 3 are presented, they largely reflect variations of similar concepts. Consequently, the theoretical framework appears somewhat repetitive and lacks sufficient depth.\n2. The methodological contribution is not particularly innovative, as it mainly combines the existing SAM framework with focal loss. Thus, the degree of novelty—especially from a methodological perspective—may be limited in its ability to inspire readers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OMCqIQ4B2T", "reviewer": "Reviewer_7xpc", "summary": "This paper investigates why Sharpness-Aware Minimization (SAM) tends to produce better-calibrated neural networks than standard optimizers such as SGD and AdamW.\nThe authors provide theoretical analysis showing that SAM implicitly regularizes the negative entropy of the predictive distribution, similar to focal loss, and thus discourages over-confident predictions.\nBuilding on this, they propose a simple variant Calibrated SAM (CSAM), which re-weights over-confident examples by modifying the SAM outer loss.\nExperiments on CIFAR-10/100 and ImageNet-1K demonstrate that (1) SAM improves Expected Calibration Error (ECE) compared with SGD and common post-hoc methods, and (2) CSAM further improves calibration without harming accuracy.", "questions": "Would the effect persist if temperature scaling or focal loss were combined with SAM?", "limitations": "", "rating": 2, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Provides a clean theoretical link between sharpness minimization and confidence calibration.\n2. Solid empirical confirmation that SAM reduces calibration error across multiple architectures.\n3. The CSAM variant is simple to implement and empirically effective.\n4. Clear exposition and well-structured proofs.", "weaknesses": "1. The core theoretical insight—that SAM implicitly maximizes predictive entropy—is conceptually close to prior entropy-regularization or focal-loss analyses; novelty is limited.\n2. The proposed CSAM is a minimal heuristic extension rather than a principled optimization method.\n3. Missing systematic comparisons against other calibration-oriented training losses such as label smoothing. Although table3 is given but the setting is very limited. Author should also test if the CSAM be compatible with other training loss.\n4. Analysis of how calibration evolves during training (claimed in Appendix D) is not deeply discussed in the main text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)