Rényi Sharpness: A Novel Sharpness that Strongly Correlates with Generalization
Authors: Qiaozhe Zhang
Jun Sun
Yingzhuang Liu
Venue: ICLR 2026 Poster (2026)
Keywords: generalization
sharpness
sharpness aware minimization
Primary area: learning theory

Abstract: Sharpness (of the loss minima) is widely believed to be a good indicator of generalization of neural networks. Unfortunately, the correlation between existing sharpness measures and the generalization is not that strong as expected, sometimes even contradiction occurs. To address this problem, a key observation in this paper is: what really matters for the generalization is the *average spread* (or unevenness) of the spectrum of loss Hessian $\mathbf{H}$. For this reason, the conventional sharpness measures, such as the trace sharpness $\operatorname{tr}(\mathbf{H})$, which cares about the *average value* of the spectrum,  or the max-eigenvalue sharpness $\lambda_{\max}(\mathbf{H})$), which concerns the  *maximum spread* of the spectrum, are not sufficient to well predict the generalization. To finely characterize the average spread of the Hessian spectrum, we leverage the notion of *Rényi entropy* in information theory, which is capable of capturing the unevenness of a probability vector and thus can be extended to describe the unevenness for a general non-negative vector (which is the case for the Hessian spectrum at the loss minima). In specific, in this paper we propose the *Rényi sharpness*, which is defined as the negative of the Rényi entropy of loss Hessian $\mathbf{H}$. 
Extensive experiments demonstrate that Rényi sharpness exhibit *strong* and *consistent* correlation with generalization in various scenarios. Moreover, on the theoretical side, two generalization bounds with respect to the Rényi sharpness are  established, by exploiting the desirable reparametrization invariance property of Rényi sharpness. Finally, as an initial attempt to take advantage of the  Rényi sharpness for regularization, Rényi Sharpness Aware Minimization (RSAM) algorithm is proposed where a variant of Rényi Sharpness is used as the regularizer. It turns out this RSAM is competitive with the state-of-the-art SAM algorithms, and far better than the conventional SAM algorithm based on the max-eigenvalue sharpness.

Published as a conference paper at ICLR 2026
RÉNYI SHARPNESS: A NOVEL SHARPNESS THAT STRONGLY CORRELATES WITH GENERALIZATION
Qiaozhe Zhang , Jun Sun', Yingzhuang Liu School of Electronic Information and Communications Huazhong University of Science and Technology (qlaozhezhang, juns, liuyz] @hust. edu. en
ABSTRACT
Sharpness (of the loss minima) widely believed to be a good indicator of gen- eralization of neural networks. Unfortunately, the correlation between existing sharpness measures and the generalization not that strong as expected, some- times even contradiction occurs. To address this problem, key observation in this paper is: what really matters for the generalization the average spread (or unevenness) of the spectrum of loss Hessian H. For this reason, the conventional sharpness measures, such as the trace sharpness tr(H), which cares about the average value of the spectrum, the max-eigenvalue sharpness Amax (H), which concerns the maximum spread of the spectrum, are not sufficient to well predict the generalization. To finely characterize the average spread the Hessian spectrum, we leverage the notion of Rényi entropy in information theory, which is capable of capturing the unevenness of a probability vector and thus can be extended describe the unevenness for a general non-negative vector (which is the case for the Hessian spectrum at the loss minima). specific, in this paper we propose the Rényi sharpness, which is defined as the negative of the Rényi entropy of loss Hes- sian H. Extensive experiments demonstrate that Rényi sharpness exhibit strong and consistent correlation with generalization in various scenarios. Morcover, on the theoretical side, two generalization bounds with respect to the Rényi sharpness are established, by exploiting the desirable reparametrization invariance property of Rényi sharpness. Finally, as an initial attempt to take advantage of the Rényi sharpness for regularization, Rényi Sharpness Aware Minimization (RSAM) al- gorithm is proposed where a variant of Rényi Sharpness is used as the regularizer. It turns out this RSAM is competitive with the ofate-of-the-art art SAM algorithms. and far better than the conventional SAM algorithm based on the max-eigenvallue sharpness.
INTRODUCTION
Understanding why stochastic optimization methods, such as stochastic gradient descent (SGD) can achieve strong generalization performance
when the neural networks overparameterized
remains a fundamental yet open challenge in deep learning (Zhang et al., 2016) Gunasekar et al., 2017 Li al., 2018 (Soudry et al. 2018 Woodworth et al. 2020). Many empirical and theoretical
studies have observed that the generalization of neural networks is closely related to the flatness of
the loss landscape (Keskar al. 22066; Neyshabur al. 2017 Jiang al. 2019 Petzka al.) 2019
Kaddour et al.
2022; suzuuu et al. 2020, Jang et al., 2022 Dziugaite & Roy 20177jjasstzzbbs
al. 201
Wu al. 2018 Blanc et al. 2020; Wei & Ma 2019 Foret et al. 2020 Damian al.
2021
Li et al.
2021 Ma & Ying 2021 Ding et al. 2024 Nacson et al. 2022
Lyu et al. 2022 Wu
2023
K Kwon et al. 202 1 Zhou et al. (024).
Intuitively speaking, local minima with flat (with low sharpness) neighborhood in the landscape are expected ineur small loss change (Hochreiter & Schmidhuber 1994 Keskar et al. 2016).
"Jun Sun (juns @ www.tuuuuu.... is the corresponding author.
The source code publicly available at link

---

Published as a conference paper at ICLR 2026
The core question is therefore: how should measure the flatness in proper way? The flat- ness or sharpness is normally quantified by functionals of the loss Hessian e... tr(H) and (m) or by loss increase with constrained weight perturbations, while the latter is normally closely related to the former. Despite the above intuition, recent empirical evidences indicate that existing sharpness measures usually correlate weakly with generalization (Andriushchenko et al.) 2023), sometimes even contradicting phenomenon occurs (Dinh et al. 2017; Wen et al. (023). To close the gap between the intuition and the reality, is of crucial importance to develop better sharpness measure.
To address this problem, a key observation of ours is: what really matters for characterizing the generalization lies in the unevenness or average spread of the spectrum of the Hessian. Intuitively speaking, an even spectrum (with almost identical eigenvalue) is very much desirable to ensure good generalization, since if there exists no particularly large eigen-direction, a small perturbation of data (which can be translated to weight perturbation) would just incur small loss change. More concretely, when characterizing the loss change resulting from the train-test data discrepancy, the unevenness or average spread of the the spectrum can reflect the influences from all categories of eigenvalues of loss Hessian (Sankar et al. 2021): 1) the top eigenvalues, which very important for the loss change but are of quite small quantity; 2) the middle eigenvalues, which are less im- portant for the loss change individually but of very big quantity; 3) the tail eigenvalues, which are normally located near zero and thus play a minor role regarding the loss change. In contrast, the conventional sharpness measures, such as the trace or maximum eigenvalue of the loss Hessian, they care about only part of the cigenvalues. For example, the trace sharpness tr(H) actually cares about only middle eigenvalues, while the max-eigenvalue sharpness Amar (H) concerns only top eigenvalues. Therefore, both of them might experience significant information loss when predicting the generalization performance.
To finely characterize the unevenness or average spread of the Hessian spectrum, we propose a novel sharpness measure, Rényi sharpness by leveraging the notion of Rényi entropy (Rényi 1961) in information theory, which can well describe the unevenness of probability vector p by exploiting an appealing property, i.c. concavity in p. Naturally, Rényi entropy can be employed to describe the unevenness of any general non-negative vector by normalization, i.c. by transforming the original vector to a virtual probability vector. Moreover, Rényi entropy enjoys extra advantages of flexibility (with one free parameter) compared against the classical Shannon entropy (c.f. Section 2 In addition, it is worth noting that to describe the average spread of a vector, the sample variance an alternative which easy enter the mind. Unfortunately it is improper for characterizing the generalization, because the tail eigenvalues (near-zero) of the spectrum contribute a lot to the variance, while they play a very minor role for the generalization gap.
To rigorously establish the relationship between generalization and Rényi sharpness, we develop several generalization bounds in terms of Rényi sharpness, by taking advantage of the reparametriza- tion invariance property of Rényi sharpness, and the technique of translating data discrepancy to the multiplicative weight perturbation. Moreover, to verify the correlation between the Rényi sharpness and generalization, we provide a fast algorithm, which is based on the Stochastic Lanezos Quadra- ture (SLQ) method (Yao et al.) 2020), to estimate the Rényi sharpness. Finally, we introduce Rényi Sharpness-Aware Minimization (RSAM) for network training, which basically employs the Rényi sharpness as a regularizer.
In summary, contributions are stated as follows:
We introduce novel notion of sharpness Rényi sharpness, which is motivated by the observation that generalization highly depends on the average spread of the spectrum of the loss Hessian, which can be captured by the Rényi entropy, an important functional in information theory.
We present two generalization bounds with respect to the Rényi sharpness, thus establish- ing the link between them in rigorous way. In developing these generalization bounds, it is important to leverage the reparametrization invariance of the Rényi sharpness and the technique of translating data perturbation to (multiplicative) weight perturbation.
We demonstrate that there exists strong correlation between Rényi sharpness and general- ization. Meanwhile, a fast algorithm to estimate the Rényi sharpness, which leverages the Stochastie Lanczos Quadratur (SLQ) method, is proposed.
2

---

Published as a conference paper at ICLR 2026
A preliminary version of Rényi Sharpness-Awwn Minimization (RSAM) is proposed, where a variant of Rényi Sharpness is employed as a regularizer during training. turns out to be competitive with the state-of-the- art SAM algorithms and significantly outperform the conventional SAM method, such as that using max-eigenvalue sharpness.
1.1 RELATED WORKS
Sharpness vs. Generalization: The exploration of relationship between sharpness and generaliza- tion dates back to Hochreiter & Schmidhuber (1994), which proposes an algorithm to achieve high generalization capability by searching flat minima. Keskar et (2016) shows that the generalization performance of large batch SGD is correlated with the sharpness of the minima. [Nyshabur et al. (2017) studies various generalization measures and highlights the promising correlation between sharpness and generalization. Jiang et al. (019) performs large-scale empirical study and finds that soiness-based measure is higher correlated with generalization than the concepts like weight norms, margin-, and optimization- based measures. Petzka et al. (202I) studies relative flatness of a layer through a multiplicative perturbation setting and shows the correlation with generalization. However, many recent studies point out that sharpness does not correlate well with generalization. Dinh et al. (2017) focuses on deep networks with rectifier units and builds equivalent models whose sharpness can be significantly changed. (Anrriuhchenkoo et al. (2023) find that sharpness may not have a strong correlation with generalization for a collection of modern architectures and settings. Wen et al. (2023) shows that flatness provably implies generalization but there exist non-geneen non-generalizing flattest models. Kaur et al. (2023) shows that the maximum eigenvalue of the Hessian can not always predict generalization even for models obtained via standard training methods. central reason why these works consider sharpness to be unreliable is that there exist sharp models with good general-
ization.
Sharpness-Aware Minimization (SAM): As early as 1994, Hochreiter & Schmidhuber (1994) sought to achieve stronger generalization by identifying flat minima, many recent researches find that sharpness is correlated with generalization. This investigation inspires multiple methods that optimize for more flat minima. These algorithms impose penalties based on different criteria, such as the trace in average case (Jia & Su (020)) or the worst-case perturbation such SAM (Foret et al. 2020) and its variations (Kwon et al. 2021 Zhuang et al.) 2022 Du et al. 2022 Kim et al.) 2022 Mi et 2022 Li & Giannakis 2023] Li al. (024a)). To enhance the generalization, Eigen-SAM is proposed (uoo et al. (2024) which periodically estimates the top eigenvalue of the Hessian matrix and incorporates its orthogonal component the gradient into the perturbation, thereby achiev- ing more effective top eigenvalue regularization effect. To obtain
sharpness measures, a universal class of sharpness is proposed in Tahmasebi et al. (2024).
2 PROBLEM FORMULATION, KEY NOTIONS AND PROPERTIES
Model. Let f(0,x) be a model with L layers, where 0 = (W1, W WL-1, Wt), and W is the weights of the 1-th layer, the vectorization of 0 and W, is f and w, = vec( W, corre- spondingly. For given training dataset S y,) and twice differentiable loss function ((f(0,x) x),y) y), the empirical loss is given by L(S, 0) 2 i-1 ((f(0,x,) y:). The training and testing dataset sampled from the real data distribution D. and the population loss is given by L(D,0) 0) = Ex,y)>=((( (0, y)]. The generalization gap is defined as the difference between the population loss T(DD, A) and the empirical loss L(S, 0)
Having observed only S, the model utilizes L(S,0) as estimation of L(D, 0), and solves mingL((,, 0) using an optimization procedure such as SGD or Adam.
Rényi Entropy. Rényi entropy is generalization of the classical Shannon entropy, which enjoys the advantage of increased flexibility by adding one parameter and reduced computational complexity. The Rényi entropy of probability vector p- |,,,P,,-.. .......... ,P] is defined
(p)
log
for O < a < oo and a # 1. The Shannon entropy can be seen as a special example when the order
- 1.
3

---

Published as a conference paper at ICLR 2026
Two notable properties of Rényi entropy are follows: 1) Concavity p Rényi entropy a concave function of the distribution p. A direct implication of this property is that Rényi entropy takes its maximum when p is uniformly distributed. 2) Monotonic decrease in a When increases, the penalty over the non-uniformity (or unevenness) gets more strict, thus more emphasis would be the high probability mass, and vice versa.
The Rényi entropy can be generalized to the matrix setting. In specific, for a positive definite matrix H. we can define its Rényi entropy as the normal Rényi entropy of its normalized eigenvalues, i.e.,
Ha (H)
log
(2)
In theory, typically amalyze the Hessian a (local) minima, and therefore assume the Hessian to be positive definite. In practice, however, due to imperfect convergence or numerical errors in the algorithm, some negative eigenvalues may appear. Since these eigenvalues usually have very small magnitudes, we commonly take their absolute values before performing subsequent computations.
Definition 2.1 (Rényi Sharpness) For a neural network, consider an arbitrary layer within the model, denote the Hessian matrix of the loss function w.rt. the layer's 's weight as H. The Rényi sharpness is defined as the negutive Rényi entropy of the normalized spectrum of H, i.e., -H,
Rényi Sharpness has a valuable property, i.e., the reparamettizzaioon invariance when the activation functions are homogeneous or nearly homogeneous. This property turns out to play an important role in developing the generalization bounds in terms of Rényi Sharpness. A formal statement regarding this property is as follows:
Proposition 2.2 (Reparameterizaitoon (Scaling) Invariance of Rényi Sharpness) Consider a L-- layer feedforward neural network with positively homogeneous activation function (i.e., o(ex) = (x for all с > 0), and parameters (WI..... W J. Let the network output be f(x) WT, o 1-11 -......... o (Wiz)), (z)), and let (()) denote the loss function, where O denotes the weights of arbitrary layer, i.e., W. Define the loss Hessian as Hg = (2)( Consider a layer-wise scaling transformation defined by W, = WW,, a <>> 0, with I G 1. Let â = Wt be the scaled parameters, and define Hg as the corresponding Hessian. Then the spectrum-normaiiza Rényi entropy of H is invariant:
H. (Hg) H. (Ho), Va 0, 10
(3)
The detailed description about reparameterization invariance and the proof of Proposition 2.2
provided in Appendix B This invariance is valid for the positive homogeneity of the activation function. In Transformer architectures (e.g.. ViTs), although GELU is not strietly homogeneous, one has GELU(az)/a ze GELU(x) (Andriushchenko et al. 2023), and thus the Rényi sharpness is approximately invariant in this setting. Note that when the order - the Rényi entropy reduces to the Shannon entropy, which is also invariant under the settings in Proposition 2.2 We also remark that this invariance only holds for the layerwise sharpness, the connection between global sharpness and the layerwise one can be found in Appendix F
3 GENERALIZATIONS BOUNDS WITH RESPECT TO RÉNYI SHARPNESS
In this section, we will provide several generalization bounds in terms of Rényi sharpness, by taking advantage of the trick of translating the data discrepancy to multiplicative weight perturbation and the reparameterization invariance Rényi sharpness.
First of all, we'll argue that the data perturbation can be translated to the multiplicative weight perturbation when characterizing the generalization.
Proposition 3.1 (informally) For any p 0, and training S draw from the distribution DD with high probability,
L(D,0) < EA(L(S(A,,), C
(4)
where S(A,p) = {(x + pAx, (x) E S} and A is a orthogonal matrix sampled under Haar measure, iee,, uniform on (d).
4

---

Published as a conference paper at ICLR 2026
The more detailed description and proof of Proposition 3.1 can be found in Appendix B Intuitively, Theorem 3.1 uses ((A, p) to approximate D. treating the discrepancy between D and S as the perturbation to S. This assumption is essentially akin to the data-separation assumption: data from different classes are spatially separated with no inter-class overlap. Under this premise, one can perturb sample within its class (i.e., move along the within-class manifold) without affecting other classes. Note that D and S can also be feature distributions, thus we can also bound the population loss using the perturbation in the feature space.
The key idea of the perturbation translation is that a multiplicative perturbation in input (feature) space can be transferred into parameter space. A neural network can be written as composite function/ f o (Wh(z)) where W is the weight at given layer, h(x) is the function consisting of the layers front of W all the way to the input, and g is the function behind the W all the way to the output. Let f = o( (Wh())), if h(x) = then W = W1, which is the weights of the first layer, and the perturbation to h(x) happens input space, other-wisely happens in feature space. Consequently,
9( (W()x)) + (AI(x))) 3( p((())AA)(()) s(( (WA)((x))
(5)
i.e., the perturbation to the h(x) is fully transferred to the parameter W. Thus, the generalization gap is closely related the sharpness of single layer, therefore we can examine the generalization by studying the sharpness of only a single layer.
Based on the above translation result and motivated by the work of (Jia & Su 2020), we have the first generalization bound based on Rényí sharpness as follows (informally stated):
Theorem 3.2 (informally) Let 0" Rd be the parameter of one layer and be an isolated local minimum of a bounded loss function L(s-)) E [0, 11, and define a posterior Q concentrated near 0 via local loss deviations, ie., Q has a density q(0) xx LLLo)-(10), where L(0) is the loss function and To is the minima loss obtained by the optimization algorithm. Then, for any 5 E (0, and > 0, a + 1, with probability at least 1 - 8 over a training set S of size N, we have:
2L. + CV2/n exp( n Ha (H) A]) + log 2N
ED]L(D,0) S BolL(S, +
8
(6)
--1
where V is the volume of the neighborhood M(0)), and A, C are positive constants, H V2L(S,0) 0*) is the Hessian at 6 and (H) is the Rényi entropy of order a of the normalized
eigenvalues of H.
To exhibit a more direct relationship between the population risk and the empirical risk, we provide another generalization bound as follows:
Theorem 3.3 (informally) Given a loss function L(.,-) and layer-wise local minimum 0 R".
Let H denote the Hessian of the loss w.r.t 8" Take a prior uniform in a ball that contains the ellipsoid (p((p)
(:: (0- )²) H(O 0*) p²} where p is sufficienlly small and satisfy p >> O. Take a posterior uniform in (p) For any 8 E (0, and a > 0, # 1, we have with probability
least ô over a training set S of size N, we have:
3 H. (H) + log 2VN + +C
L(D.0") S L(S,0') 2
2² +0(s)+)
+1
2(N 1)
where C is a positive constant, H = VIL(S,0* 0*) is the Hessian at 0- and Ho (H) is the Rényi entropy of order a of the normalized eigenvalues of H.
The detailed version and proof of Theorem 3.2 and Theorem 3.3 can be found in Appendices Cland D respectively. Both Theorem 3.2 and Theorem 3.3 indicate that the generalization is bounded by the Rényi entropy of the Hessian matrix of the loss with respect to the weights.
RÉNYI SHARPNESS: ORDER SELECTION & FUNCTIONAL ESTIMATION
In this section, we will discuss the choice of the order parameter a in Rényi sharpness. Furthermore, we will provide a fast algorithm for estimating the Rényi sharpness.
5

---

Published as a conference paper at ICLR 2026
4.1 ORDER SELECTION IN RÉNYI SHARPNESS
The heavy-tailed spectrum of the Hessian matrix is a ubiquitous feature in deep networks. In this section, we compute the Hessian spectrum of each layer by Hesssii (Yao al. 2020). and find that although all the spectra are heavy-tailed, the shapes of the spectrum can be divided into two categories, which correspond to different choices of a.
18
Deniiy 0.2g Scale)
156
Dersily Staain
10a
1250
1500
14:1
1000
Densty
750
10-1
500
10-4
250
0.5
1.0
0.0
0.2
0.0
1.0
Match
Nismatch
Eigendauw
Eigeneelace
Statistics
Figure 1: Hessian spectra [a,b,c]. Two zero-dominant profiles are observed: (a) multi-cluster and (b.c) uniform. Optimal vs. Hessian spectral type [d]. Statistics summarizing whether the empirically optimal a matches the predicted choice under each Hessian spectral type.
(d)
We summarize the shape the spectrum into the following two categories: 1) Zero-dominan multi-cluster spectrum and 2) Zero-dominant, uniform spectrum. We selected representative plots from CAANee18-CIFAR 10 to illustrate these two categories, as shown in Fig. [ The zero-dominant, multi-cluster spectrum (Fig.
(a)) consists of large number of near zeros (Part 1) and some large eigenvalues (Part 3), and between these two eigenvalues, there are some eigenvalues (Part2) that cannot be ignored but are significantly smaller than the large eigenvalues. The zero-dominant, uniform spectrum (Fig. [ (b.c)), on the other hand, contains only large number of near zeros and some large eigenvalues. The detailed spectrum of each layer across different tasks is pushed to Appendix K. and similar spectrum can also be found in Sankar et al. (2021).
To capture the multi-cluster nature (Fig. D (a)), we note that eigenvalues near zero (Part contribute less to sharpness and generalization. Therefore, it is important to choose a suitable that embodies the differences among the dominant (Part 3) eigenvalues and those small but non-negligible eigen- values (Part 2). When > 1, the measure disproportionately amplifies large eigenvalues while ignoring smaller ones. To better capture the spectrum's subtle variations, especially on Part 2, preferable to use order (0, 1), which balances sensitivity across both large and small eigen- values. In practice, we observe that setting a = 0.5 typically yields the most stable and significant correlation between Rényi sharpness and generalization.
In the case of uniform spectrum (Fig. (b,c)), one part of Part 2 and Part 3 vanish, leaving only a few dominant ones. Therefore, it becomes crucial to capture the differences among these dominant eigenvalues. When a E (0, 1), the order tends to suppress these differences, which is undesirable in this context. Thus, choosing a > is more appropriate, it captures the contribution of every cigenvalue and highlights their differences. However, as approaches 1, practical numerical com- putation becomes unstable. Balancing theory and practice, > will better, and we find that = 1.5 performs well and exhibits strong and robust correlation.
Overall, the key to choosing is whether the eigenvalues that influence generalization form clusters whose inter-cluster separation exceeds the clusters' enlargement. there is single cluster, selecting > suffices to examine inter-eigenvalue differences. When clusters are widely separated, we should choose a < 1 to avoid over-emphasizing the larger eigenvalues when > 1. In practice, 0.5 and a = 1.5 tend to provide robust and consistent results across different datasets and models. The summary statistics of the average correlations for different values of can be found in the Appendix K.4
We conducted a statistical analysis of the experiments in Section 5 examining whether the value of a that yields the highest correlation between the layer-wise Rényi sharpness and generalization is consistent with our prior analysis. We then recorded the number of successful and unsuccessful matches in 60 models, with a total of 1630 cases: 1451 matches and 179 mismatches, as shown in Fig. (d). Overall, the empirical findings agree well with preceding intuitive analysis.

---

Published as a conference paper at ICLR 2026
4.2 ESTIMATION OF RÉNYI SHARPNESS
To estimate the Rényi entropy of the Hessian matrix, it would be of prohibitive complexity if we directly calculate the spectrum of the Hessian matrix, due to the huge size of the matrix. To cir- cumvent this difficulty, we will reformulate the Rényí entropy as functional of the trace of matrix functions and then leverage the stochastic trace estimator (also known the Hutchinson method) and stochastic Lanczos quadrature method to greatly reduce the complexity.
Firstly, the Rényi entropy is reformulated as follows:
71
A a
EE X 1
Tr(
Ha
log
log Tr(H)"
log
(8)
1 (r(
Thus the estimation task boils down to calculating the trace of matrix functions Tr(H) and Tr(H").
To estimate the trace of matrix functions f(H), the stochastic trace estimator can be leveraged to greatly reduce the complexity:
(H)) = (H))) = Tr T) = (H) =] T f(H)v),
(9)
where f is analytic inside a closed interval function. is the identity matrix, and v is sampled from a Rademacher distribution.
To economically calculate the expectation of the quadratic form v f(H)v. the Gaussian quadrature rule can be employed transform the expectation to an integral. Further, the integral can be com- puted with the nodes and the weights of the quadrature rule given by the Lanczos algorithm, (Golub) Strakoš 1994, Golub & Meurant 2009; Bai & Golub 1996 Bai et al. 1996; Golub & Van Loan Ubaru et al. (017) which basically generates orthonormal basis for the Krylov subspace such that the matrix can be reduced to tri-diagonal one, hence greatly lower the computational bur- den. Combined all the above, it constitutes the framework of the stochastic Lanczos quadrature
Ubaru
(SLQ) algorithm (Ubaru et al., 2017), which is exactly the basis of Algorithm 1
The details for the estimation of Rényi entropy shown in Algorithm
Algorithm Rényi Entropy Estimation via Stochastic Lanczos Quadrature
Input: Positive definite matrix H of size X n, Lanezos iterations m, computation iterations
order a > and a + 1.
Output: Estimation of (H).
for k = 1,... do
Draw two random vector vı and ga of size n x 1 from (o0, and normalize it, wi = Hy w vi,
wi V1v1;
for = 2,..., m
1). 8, w
2). stop if 3, = else v, = w1-1/3,
3). w; Hv,
w; T vi W, = W. a.V; - 3yv--1;
end for
4). Te ( i)
0;, i i= 1
= 1= 1,...,m 1.
5). A e Te. e1, B₁ = g: Hg
end for
log Σi A
Return: HH I-a [ BL)
5 CORRELATION BETWEEN RÉNYI SHARPNESS AND GENERALIZATION
In this section, we estimate the Rényi entropy via Algorithm and validate that Rényi entropy is strongly correlated with generalization.
5.1 TASK
We evaluate the correlation between Rényi sharpness and generalization on: ResNet18/34 (He et al, 2016), and Simple Vision Transformer architecture from the vit vit -pytorch library CIFARTO

---

Published as a conference paper at ICLR 2026
(Krizhessky & Hinton 2009), ResNet18/34 CIFARIO0, and ResNet18 TinylmageNet (Le & Yang, (015). We vary the learning rate, optimization algorithm, and the weight decay strength to generate different local minima, and then estimate the layer- wise and global Rényi sharpness. More details can be found in Appendix J We compare with the classical Hessian-based flatness mea- sures using the trace of the loss-Hessian, the Fisher-Rao norm(Liang flatness measure that performed best in the extensive study of Jajan et al. (2019), the Frobenius norm of the weights, and the sharpness defined in SAM (Foret et al. (020) and ASAM woon et al. 2021). Notably, the sharpness defined in ASAM (Kwon et al. 2021 has been empirically shown by (nnriushchenkoo et al. (2023), on larger-scale datasets and models, to have little or no correlation with generalization performance. The definition and detailed implementation of those measures can be found in Appendix [ and the hyperparameter p in SAM and ASAM is searched over (10-9, 3x 10-6, 10-5, 3x 10-50 TO 10-1, 3x 10- 10-3, 3x 3010- 10 3 10- 10-1 0.3, 1).
al. 20T9). the PAC-Bayes
To detect correlation, we follow the previous works by Dziugaite et al. (020); Jiang et al. (2019);
Kwon et al. (202) Andriushchenko al. (2023) and use the Kendall rank correlation coefficient:
y(x,y) = N(N - 1) K
sign(a: Tzy (signi - y₁)
(10)
where y, E RN are vectors of generalization gap and sharpness values for N different models. We follow the approach of offAnndrussheenoo et al. al.al (2023) by comparing sharpness and generalization within the same model architecture. This contrasts with prior works such as Dziugaite et al. (2020) and et al, (2019), which focus on comparisons across models with varying width or depth. We always evaluate sharpness the same training points taken without any data augmentations, while the data augmentation tools are allowed in training.
5.2 CORRELATION BETWEEN RÉNYI SHARPNESS AND GENERALIZATION
After training with range of hyperparameters, we estimate Rényí sharpness and compute the Kendall rank correlation between Rényi entropy and the generalization gap (defined as the difference between training and test loss). We vary and plot the sharpness that attains the highest correla- tion coefficient. Fig. 2 reports these correlations on CIFAR-10 with ResNet-18. The "layer 1" through "all layer" subplots correspond to Rényi sharpness; the remaining subplots show alternative metrics. As evident in Fig. 2 Rényi sharpness aligns closely with generalization performance and outperforms the other measures in capturing the generalization gap.
THE 2A1
82473
1000
Spwit
2200
EVE24
Figure 2: ResNet18 on CIFAR1O, The layer 1 all layer subplots correspond to the Rényi sharpness measure. Rényi sharpness is strongly correlated with generalization than the other measures.
Owing page limits, we present the remaining tasks in compact format that aggregates all statis- tics into a single panel (Fig. 3). As shown in Fig. 3 Rényi sharpness is strongly correlated with generalization. Full per-task figures in the style of Fig. are provided in the Appendix K.2
8

---

Published as a conference paper at ICLR 2026
CRUGEE Bodkett
VT
Figure 3: Kendall correlations on various tasks. Signed coefficients are mapped to 0-1 (blue- positive, green = negative). Rényi sharpness shows the strongest correlation with generalization than other sharpness measures.
6 REGULARIZATION BY RÉNYI SHARPNESS
In this section, we propose to Rényi sharpness as regularizer during training, i.c. the Rényi Sharpness Aware Minimization algorithm. To reduce the complexity, in practice we will employ an approximation of the Rényi sharpness.
6.1 RÉNYT REGULARIZATION AND RÉNYI SHARPNESS AWARE MINIMIZATION (RSAM)
If the original form Rényi sharpness was used for regularizer, it would require multiple cycles of gradient descent, thus increasing the computational complexity by dozens of times, as compared with the traditional training method. To reduce the computational burden, we will resort to the approximations of Rényi sharpness. specific, following the work by Khan et al.] (2018))))) Kim al. 2022), we will employ the gradient magnitude as an approximation of the Hessian matrix:
H GM [Diag(
Vol(O, x.. x.)]
(11)
Consequently, the Rényi sharpness can be approximated by
E, 19, 22x
H. (H) ~ - H. (GM) 1 log (Σ, 93) )a
(12)
where g is the gradient vector computed by the optimization algorithms, and Øj Σ, 10,1 the Rényi regularizer. To avoid the memory usage and compute cost caused by explicitly computing the gra- dient with computational graph preserved (e.g., Secraate_graphPPpe minimizing the following objective instead:
N E i=1 Vo, x(, is the element in g- Thus we can -sign(1 a) (Σ, of)
in PyTorch), we consider
E, los 2a
L(e + 6) = I.(0 p p. sign(1 aa)
EE 0f of)at188)
(13)
Eq. can be expanded as follows:
19 (2, 0 == L(0) sign a)
E, |9, 2a C, g (14)
(0++)) =(0+e)= L(0) - sign - a)
Thus, optimizing Eq. 13 is approximately optimizing the original loss with Rényi regularizer, namely, Rényi sharpness-aware minimization (RSAM, see Algorithm ( We observe that penal- izing single layer (e.g.. the final layer) typically requires extending training for more epochs to achieve strong generalization, unless multiple layers are optimized concurrently. Given the combi- natorial cost of tuning layer-specific regularization strengths, adopt a single global Rényi regu- larizer applied across all layers. Appendix F Flestablishes that optimizing this global objective implies optimizing the layer-wise objectives as well.
Moreover, it is observed that incorporating the approximate Hessian matrix and penalizing Rényi sharpness at the early stages of training introduces substantial instability. To mitigate this effect,

---

Published as a conference paper at ICLR 2026
first train with plain SGD and adapt the warm-up length based on validation accuracy. For easy tasks, five epochs suffice to attain high accuracy, so the SGD warm-up is capped at five epochs. For defer switching to RSAM until the validation Top-1 I exceeds 30%, which typically occurs around epoch 20. The discussion and comparison with other related SAM variants can be found in the Appendix H
harder tasks such TinylmageNet,
6.2 COMPARISON BETWEEN RSAM AND OTHER SAM ALGORITHMS
We now apply our sharpness measure as a regularizer to train neural networks. We consider the image classification tasks involving the CIFAR 10/100 and LinylmageNet datasets. Various convolu- tional neural networks such as ResNet, and WideResNet (Zagoruyko & Komodakis, 2016) are used for CIFARTON 10/100 experiments. We also evaluated performance by fine-tuning a ViT-B-16 model pre-trained on ImageNet for CIFAR-10 and CIFAR-100. We used the checkpoint provided by Py- Torch's 's official repository| For comparison, we consider the sharpness-aware minimization (SAM) method, the adaptive SAM (ASAM) method, an extension of SAM to involve the scale-invariance, the Eigen-SAl SAM (Luo et al. 2024) method, which regularizes the top Hessian eigenvalue, the Fisher SAM (Kim et al. (2022) method which minimize sharpness under the Riemannian metric, and the Sparse SAM Mi et al. (2022) which mask the sharpness to speed up SAM algorithm. More details
provided in Appendix J.3.2
Table 1: Test accuracies (avg. + standard error) for SCDSSAM/SSAMEEiien-SAAAISS
Dutaset
Model
SGD())
SAM )
ASAM(S)
Eigen-SAM(S)
ESAM(S)
SSAM(3)
OURN(S)
ResNer20
92.68 22.0
93.44
93.6240.16
03.2440.20
93.54-9.32
03.4454024
93.69=011
CIFARIO
ResNet56
94. 04.24
94.96-9914
95.1240os
94. 40.10-00.0
95. 95.17-005
95.15-10322
95.26 0.
WideResNet.28.10
96.95=0.06
96.79
96. 08.78ss
96.96-900
96.0699004
0..0
ResNet20
1240
70.53 70.53-0.30 0.300
70.73
70.51 J0.30
70.57-00.
70.144.0.1
70.91 00.95
CIFARIO0
ResNet56
72 72.60
74.86
75.20
74.91 =m
75.42
75.71 90
WideResNen-28-10
81. 18
83.55=0.14
83.36
$2.81
83.48 48..4
83, 83.4700000
83.67 00.05
TinTeNegeNet
ResNe150
60.70-10.70
62.50
63.33 00.07
Table 2: Test accuracy for fine-tuning ViT-B-16 pretrained on ImageNet-IK on CIFAR-10 and CIFAR-100.
Dataset
Model
SGD(%)
SAM(%)
ASAM(S)
FSAM(S))
OURS(S)
CIFARIO
VIT-B-16
98. 08:00.00
98. 98.50=0..0
98 08.39
98. 08.111
98.59
20.00-
CIFAR100
VIT-B-16
88. 28.
89.38=0.04
88.78#0.33
$9.11
89,58 t0.07
We provide the averages and standard errors of the test accuracies obtained from five runs of each method in Table and Table As can be seen from the table. one can confirm that the generalization performance of SGD is significantly improved with our regularizer. Furthermore, our method out- performs the SAM, ASAM, and Eigen-SAM methods. Although our method outperforms ASAM overall, the margin is modest on certain tasks. We hypothesize this gap arises because we currently employ an approximate surrogate of the Rényi sharpness, introduced for computational efficiency. We expect further improvements if the exact Rényi sharpness can be used as the regularizer (or a tighter estimator becomes feasible), and we leave this as a promising direction for future work. Since we first warm up with plain SGD before switching to RSAM, we did not adjust RSAM's 's epoch budget to equalize total compute across methods; instead, we fixed the total number of epochs. Con- sequently, given fixed compute budget, RSAM would be allowed to more epochs and thus expected to improve further the performance.
7 CONCLUSION
In this work, we propose a novel measure of sharpness - Rényi sharpness, which defined as the negative Rényi entropy of the loss Hessian. By leveraging the reparameterization invariance of Rényi sharpness and the fact that data perturbations can be absorbed into the weight perturbations, we develop several generalization bounds based on the Rényi sharpness. Extensive experiments demonstrate a strong correlation between the Rényi sharpness and generalization. Furthermore, propose the Rényi Sharpness-Awarr Minimization (RSAM) algorithm, which penalizes Rényi sharpness during training. Experimental results demonstrate that RSAM outperforms all existing sharpness-awar minimization methods, across multiple tasks.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Maksym Andriushchenko, Francesco Croce, Maximilian Müller, Matthias Hein, and Nicolas Flam- marion. A modern look the relationship between sharpness and generalization. arXiv preprint arXiv:2302.0701, 2023.
Zhaojun Bai and Gene H Golub. Bounds for the trace of the inverse and the determinant of symmet- ric positive definite matrices. Annals of Numerical Mathematics, 4:29-38, 1996.
Zhaojun Bai, Gark Fahey, and Gene Golub. Some large-scale matrix computation problems. Journal of Computational and Applied Mathematics, 74(1-2):71-89, 1996.
Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexitie: Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463 482. 2002.
Peter L Bartlett, Dylan J Foster, and Matus Telgarsky. opectrally-normalizee margin bounds for neural networks. In Advances in Neural Information Processing Systems, pp.
2017.
Guy Blanc, Neha Gupta, Gregory Valiant, and Paul Valiant. Implicit regularization for deep neural networks driven by an orntein-uhlenneer like process. In Conference on learning theory, pp- 483-513. PMLR, 2020.
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoc Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition workshops, pp. 702-703, 2020.
Alex Damian, Tengyu Ma, and Jason D Lee. Label noise sgd provably prefers flat global minimizers. Advances in Neural Information Processing Systems, 34:27449-27461, 2021.
Lijun Ding, Dmitriy Drusvyatskiy, Maryam Fazel, and Zaid Harchaoui. Flat minima generalize for low-rank matrix recovery. Information and Inference: A Journal of the IMA, 13(2)iiaac099, 2024.
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In International Conference on Machine Learning, pp. 1019-1028. PMLR, 2017.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 100.2999..119,, 2020.
Jiawei Du, Daquan Zhou, Jiashi Feng, Vincent Tan, and Joey Tianyi Zhou. Sharpness-aware training for free, Advances Neural Information Processing Systems, 35:23439-23451, 2022.
Gintare Karolina Dziugaite and Daniel M Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv preprint arXiv: 1008, 2017.
Gintare Karolina Dziugaite, Alexandre Drouin, Brady Neal, Nitarshan Rajkumar, Ethan Caballero, Linbo Wang, Ioannis Mitliagkas, and Daniel M Roy. In search of robust measures of generaliza- tion. Advances in Neural Information Processing Systems, 33:1 1723-11733, 2020.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimiza-
tion for efficiently improving generalization. arXiv preprini arXiv:2010.01412. 2020.
Noah Golowich, Alexander Rakhlin, and Ohad Shamir. Size-independent sample complexity of neural networks. arXiv preprint arXiv: 1614.0611, 2017.
Gene H Golub and Gérard Meurant. Matrices, moments and quadrature with applications Princeton University Press, 2009.
Gene Golub and Zdeněk Strakoš. Estimates in quadratic formulas. Numerical Algorithms, 8(2): 241-268, 1994.
Gene H Golub and Charles F Van Loan. Matrix computations. JHU press, 20133

---

Published as a conference paper at ICLR 2026
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv: 026777 2017.
Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Sre- bro. Implicit regularization in matrix factorization. Advances in neural information processing systems, 30, 2017.
Ryuichiro Hataya. homura. httpps///ittubb..oommmsss/tttoo/
2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Simplifying neural nets by discovering flat minima. Advances in neural information processing systems, 1994.
Cheongjae Jang, Sungyoon Lee. Frank Park, and Yung-Kyun Kyun Noh. A reparametrization invariant sharpness measure based on information geometry. Advances in neural information processing systems, 35:27893-27995 2022.
Stanistaw Jastrzæbski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in sgd. arXiv preprint arXiv:1711.04623, 2017.
Zhiwei Jia and Hao Su. Information-theored local minima characterization and regularization. In International Conference on Machine Learning, pp. 4773-4783. PMLR, 2020.
Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization measures and where to find them. arXiv preprint arXiv: 1912.02178. 2019.
Jean Kaddour, Linqing Liu, Ricardo Silva, and Matt Kusner. When do flat minima optimizers work? Advances in Neural Information Processing Systems, 35:16577-16695, 2022.
Simran Kaur, Jeremy Cohen, and Zachary Chase Lipton. On the maximum hessian eigenvalue and generalization. In Proceedings on, pp. 51-65. PMLR, 2023.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima, arXiv preprint arXiv: 04836,, 2016
Mohammad Khan, Didrik Nielsen, Voot Tangkaratt, Wu Lin, Yarin Gal, and Akash Srivastava. Fast and scalable bayesian deep learning by weight-perturbation in adam. International conference PMLR, 2018.
machine learning. pp.
Minyoung Kim, Da Li, Shell X Hu, and Timothy Hospedales. Fisher sam: Information geometry and sharpness aware minimisation. In International Conference on Machine Learning, pp. 11 11148- 11161. PMLR, 2022.
A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.
Jungmin Kwon, Jeongscop Kim, Hyunseo Park, and In Kwon Choi. Asam: Adaptive sharpness-
aware minimization for scale-invariant learning of deep neural networks. In International confer- machine learning, pp. 9905-5914. PMLR, 2021.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS CS 7(7):3, 2015.
Bingcong and Georgios Giannakis. Enhancing sharpness-aware optimization through variance suppression. Advances in Neural Information Processing Systems, 36:70861-70879, 2023.
Tao Li, Pan Zhou, Zhengbao He, Xinwen Cheng, and Xiaolin Huang. Friendly sharpness-awe aware minimization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition, pp. 5631-5640, 2024a.
12

---

Published as a conference paper at ICLR 2026
Tian Li, Tianyi Zhou, and Jeffrey A Bilmes. Tilted sharpness-aware minimization. arXiv preprint arXiv:2410.22656, 2024b.
Yuanzhi Li, Tengyu Ma, and Hongyang Zhang. Algorithmic regularization in over-parameterized matrix sensing and neural networks with quadratic activations. In Conference On Learning The- ory, pp. 2-47. PMLR, 2018.
Zhiyuan Li, Tianhao Wang, and Sanjeev Arora. What happens after sgd reaches zero loss?-a math- ematical framework. arXiv preprint arXiv:21100069/ 2021.
Tengyuan Liang, Tomaso Poggio, Alexander Rakhlin, and James Stokes. Fisher-rao metric, geome- try, and complexity of neural networks. arXiv preprint arXiv: 2017.
Tengyuan Liang, Tomaso Poggio, Alexander Rakhlin, and James Stokes. Fisher-rao metric, ge- ometry, and complexity of neural networks. In The 22nd international conference artificial intelligence and statistics, pp. 888-896. PMLR, 2019.
Haocheng Luo, Tuan Truong, Tung Pham, Mehrtash Harandi, Dinh Phung, and Trung Le. Explicit eigenvalue regularization improves sharpness- aware minimization. Advances in Neural Informa- tion Processing Systems, 37:4424 4453, 2024.
Kaifeng Lyu, Zhiyuan Li, and Sanjeev Arora. Understanding the generalization benefit of normal- ization layers: Sharpness reduction. Advances in Neural Information Processing Systems, 35: 14689-34708, 2022.
Chao Ma and Lexing Ying. On linear stability of sgd and input smottesmoo of neural networks. Advances in Neural Information Processing Systems, 34: 16805-16817 2021.
David A McAllester. Pac-bayesian model averaging. In Proceedings of the twelfth annual confer- ence on Computational learning theory, pp. 164-170, 1999.
David A McAllester. Pac-bayesian stochastic model selection. Machine Learning, 51(1):5-21, 2003.
Peng Mi, Li Shen, Tianhe Ren, Yiyi Zhou, Xiaoshuai Sun, Rongrong Ji, and Dacheng Tao. Make sharpness-awer aware minimization stronger: A sparsified perturbation approach. Advances in Neural Information Processing Systems, 0..30900-09622 2022.
Mor Shpigel Nacson, Kavya Ravichandran, Nathan Srebro, and Daniel Soudry. Implicit bias of the step size in linear diagonal neural networks. In International Conference on Machine Learning, pp. 16270-16295. PMLR, 2022.
Vaishnavh Nagarajan and Zico Kolter. Generalization in deep networks: The role of distance from initialization. Xrviv preprint arXi::1901.0167. 2019.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. In Conference on Learning Theory, pp. 1376-1401, 2015.
Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring general- ization in deep learning. Advances in neural information processing systems, 30, 2017.
Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. A pac-bayesian approach to eeectrally-normaliiee margin bounds for neural networks. International Conference on Learning Representations, 2018a
Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. To- wards understanding the role of vveeparrametrization in generalization of neural networks. arXiv preprint arXiv: 12076 2018b.
Henning Petzka, Linara Adilova, Michael Kamp, and Cristian Sminchisescu. A reparamcterization invariant flatness measure for deep neural networks, arXiv preprint arXiv: 19/2.0005. 2019.
Henning Petzka, Michael Kamp, Linara Adilova, Cristian Sminchiseseu, and Mario Boley. Relative flatness and generalization. Advances in neural information processing systems, 24:18420-18432 2021.
13

---

Published as a conference paper at ICLR 2026
Konstantinos Pitas, Mike Davies, and Pierre Vandergheynst. Pac- bayesian margin bounds for volutional neural networks. arXiv preprint arXiv:/801.00177 2017.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PmLR, 2021.
Alfréd Rényi On measures of entropy and information. In Proceedings of the fourth Berkeley symposium on mathematical statistics and probability, volume 1: contributions to the theory of statistics, volume 4, pp. 547-562. University of California Press, 1961.
Adepu Ravi Sankar, Yash Khasbage, Rahul Vigneswaran, and Vineeth N Balasubramanian. A deeper look at the hessian eigenspectrum of deep neural networks and its applications to regularization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 9481-9488, 2021.
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The im- plicit bias gradient descent on separable data, Journal of Machine Learning Research, 19(70):
Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. arXiv preprint arXiv:2/06. 10270, 2021.
Behrooz Tahmasebi, Ashkan Soleymani, Dara Bahri, Stefanie Jegelka, and Patrick Jaillet. A univer- sal class of sharpness-aware minimization algorithms. In Proceedings of the 41st International Conference on Machine Learning. pp. 47418-474444 2024.
Yusuke Tsuzuku, Issei Sato, and Masashi Sugiyama. Normalized flat minima: Exploring scale in- variant definition of flat minima for neural networks using pac-bay baccbayesian analysis. In International Conference on Machine Learning, pp. 9636-9647. PMLR, 2020.
Shashanka Ubaru, Jie Chen, and Yousef Saad. Fast estimation of tr(f(a)) via stochastic lanezos quadrature. SIAM Journal on Matrix Analysis and Applications, 38(4): 2017.
Colin Wei and Tengyu Ma. Data-dependent sample complexity of deep neural networks via lipschitz augmentation. Advances neural information processing systems, 32, 2019.
Kaiyue Wen, Zhiyuan Li, and Tengyu Ma. Sharpness minimization algorithms do not only minimize sharpness achieve better generalization. Advances in Neural Information Processing Systems, 36:1024-1035 2023.
Blake Woodworth, Suriya Gunasekar, Jason D Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. In Conference on Learning Theory, pp. 3635-3673. PMLR, 2020.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference on machine learning, pp. 23965-23998. PMLR, 2022.
Lei Wu and Weijie J Su. The implicit regularization of dynamical stability in stochastic gradient descent. In International Conference on Machine Leurning, pp. 37656-37684, PMLR, 2023.
Lei Wu, Chao Ma. et al. How sgd selects the global minima in over-parameterized learning: A dynamical stability perspective. Advances in Neural Information Processing Systems, 31, 2018
Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W Mahoney. Pyhessian: Neural networks through the lens of the hessian. In 2020 IEEE international conference on big data (Big data), pp. 581-590. IEEE, 2020.
Sergey Zagoruyko and Nikos Komodakis. 105. 2016.
Wide residual networks.
arXiv preprint
14

---

Published as a conference paper at ICLR 2026
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:6611003500 2016.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint 1rXiv: 09.00...9 2017.
Zhanpeng Zhou, Mingze Wang. Yuchen Mao, Bingrui Li, and Junchi Yan. Sharpness-awar min- imization efficiently selects flatter minima late in training. arXiv preprint 100:::22100.00777 2024.
Juntang Zhuang, Boqing Gong, Liangzhe Yuan, Yin Cui, Hartwig Adam, Nicha Dvorekk, Sekhar Tatikonda, James Duncan, and Ting Liu. Surrogate gap minimization improves sharpness-aware training. arXiv preprint arXiv:2203.0806. 2022.
A ORGANIZATION OF APPENDIX
The appendix is organized follows:
Sec.
A
an overview of the organization of the appendix.
Sec.
B detailed proof of the PAC Bayesian generalization bound under multiplicative
perturbation (Theorem 3.1)
Sec. C
detailed proof of the PAC Bayesian generalization bound for Rényi entropy moti-
vated by (Jia & Su 2020) (Theorem 3.2).
Sec. D detailed proof of the PAC Bayesian generalization bound for Rényi entropy (The-
orem
Sec. detailed proof of the reparameterization invaricance of Rényi entropy (Proposition
2.2).
Sec.
detailed proof of optimizing global Rényi regularization implies optimizing layer-
wise Rényi regularization.
Sec.
a proof of arbitrary trace rescaling under fixed normalized spectrum.
Sec. H detailed discussion and comparison with Rényi sharpness-aware minimization and
some related sharpness-aware minimization variants.
Sec.
detailed description and definition of the baseline sharpness measures.
Sec.
M detailed descriptions of the datasets, models,
choices used in our
experiments, including correlation experiments and the sharpness-aware minimization ex- periments.
This section presents the Hessian spectrum which determine the Rényi order choice and the correlation coefficient under different Rényi order The correlation comparison between the Rényi sharpness and other sharpness measures across multiple tasks is also included.
Sec. K
Sec.
limitations of our assumptions and theoretical results.
Sec.
M
broader impacts statement of this research.
15

---

Published as a conference paper at ICLR 2026
B PAC BAYESIAN GENERALIZATION BOUND UNDER MULTIPLICATIVE PERTURBATION
Below, state generalization bound based multiplicative perturbation.
Theorem B.1 For any > 0, and training set S draw from the distribution DD assumed that L(D, 0) < L((D, + 8), where is the pertubation the weights, S(A,p) {(x + pAx, y) yx. E }} and A is orthogonal matrix sampled under Haar measure, i.e., uniform on (d). With probability E,
log !
L(D,) 0) S (A) +OV
2n
The condition I.(D,0) 0) S L(D, 0++) means that adding perturbation to weights should not decrease the test error. This is expected to hold in practice for the final solution but does not necessarily hold for any
Proof. Based on the Hoeffding's ss inequaliy, which is stated as follows:
Theorem B.2 (Hoeffding's inequaliy) Let U1,...,U, Un beindependent random variables taking values in an interval (a, 6]. Then, for any E R.
S ~
E
(15)
Let U, = (A + wi)] thus EU, = [A(A(D(A, and 1 - U n
EA EA(L(S(A,P)), where p(A,p) {(x + (x,,) E D}. S(A, p) {(x + pAx. y) yx. E S) and A is orthogonal matrix sampled under Haar measure, i.e., uniform on (d). Consequently, we have
Es In (AAAAAAAAAAAN)-ALLLAAAAA) S Se 5
pt2c2
(16)
For any 8,
Ps EA EA p))1 - (AI
(17)
Ps nh (HE[EAAL(IAA -AA > nts
(18)
Markov's inequality
(19)
n < e -nt
(20)
Consequently,
Ps EAI((D(A,))) > (AI Se nt2c² nts
(21)
when 4s/02, C² /8 nts is minimized, thus,
Ps EA PALLIAAAA,, SE -2n2?
(22)
22n C2
let
we have
log!
Pss EA [A(L(D(A,))) (A(T(S(A,))) + Ct
Se
(23)
2n
16

---

Published as a conference paper at ICLR 2026
consequently,
log 2n
Is [A(L(D(A, EA(L(S(A, p))] + CV
1 -
(24)
For any multiplicative perturbation, the perturbation in the input space can be fully transformed into weight space, which means (A(L(D(A, p))] L(D, + ō), where obeys some unknown distribution. Consider the assumption that L(D, 0) S L(D, 0 8), we have
log 2n
Ps = (AAL(((A, P)J] + +CV
S
2 21-6
(25)
The idea about multiplicative perturbation under haar measure is also reported in Pet- zka et al. (202T). whose sharpness is define by the Hessian matrix of the loss function w.r.t a full connect layer's weights, but their follow-up results need to split the Hessian matrix into multiple blocks and compute the corresponding traces individually, which proposes a huge computation bur- den when dealing with a big layer, thus they only compute the sharpness of last layer in small model. Contrary to deriving a bound via multiplicative perturbations like Petzka et al. (202I), this section aims to show that the dependency between the real and empirical data distributions formed to a weight perturbation of an individual layer, enabling the application of Theorem 3.2 and t.3 study the corresponding layer-wise spectrum. Unlike the global spectrum, the layer-wise spec- trum is more likely to be invariant under reparameterization In Section 4 we prove the invariance of the Rényi entropy in Theorem 3.2 and 3.3 Since the invariance conditions for the normalized global spectrum are much more restrictive, Theorem and 3.3 only apply to the layer-wise Rényi entropy. Nevertheless, in Section we empirically observe that the Rényi entropy of the global spectrum is still correlated with generalization. We attribute this phenomenon to the fact that the global spectrum composed of the layer-wise spectra; hence, when the layer-wise spectra exhibit strong correlations, the global spectrum also demonstrates significant correlations.
Discussion:
be trans-
Corollary B.3 For any p > O, and training set S draw from the distribution D, we sumed that L(D, 0) < L(D, 0 + 8). where is the pertubation to the weights, S(A.P) = {(x + pAx,y y) (x,,) uniform on (d). With probability c, we have
S} and A is orthogonal matrix sampled under Haar measure, i.e.,
log 2n
L(D,0) 0) < EA ((S(A,)) P), +C
17

---

Published as a conference paper at ICLR 2026
C PAC BAYESIAN GENERALIZATION BOUND FOR RÉNYI ENTROPY MOTIVATED BY (JIA & SU. 2020)
In this section, will propose generalization bound based on the Rényi entropy of the Hessian spectrum of the loss function with respect to the weights.
Proposition C.1 Given training set S with size N draw from the data distribution D and loss function L(.-)) E [0,1], a layer-wise local minimum 0 E R" is isolated and unique in its neigh- borhood M(0*) whose volume V is sufficiently small, pick uniform prior P over MM and pick the posterior Q of density q(f) e-1Lo-11.00) with Lo L(S,0". For any 8 (0. 1| and a >
O. a + 1, we have with probability at least 1 that:
2 2A + log 2N 8
(D) & Eo ()] +2
(26)
N-1
where A= I VV n exp( (H) ), and A > 0 is the constant item. H is the Hessian matrix
A of loss function w.r.t. 0"
Proof. Using PAC-Bayesian generalization bound proved by (Jia & Su 2020
Theorem C.2 Given a training sei S with size draw from the data distribution DD and loss function I( E [0, 1}, a local minimum 0" E T is isolated and unique in its neighborhood M(8*) whose volume V is sufficiently small, pick a uniform prior P over E -M(0) and pick the posterior of density q0)) x e - 1-1----- L(S,0) with Lo = L(S,0). For any 8 E (0, 1]. we have with
probability at least 8 that:
2Lo + 2A + log2
1og 2N
E [o(L(D,)) s Eo([(S, 0) 2
(27)
N
where
Txe nV *
expl
log H and H is the Hessian matrix of loss function w.r.t. 0"
Next, we will utilize the Rényi entropy to bound the log H term.
log H
logA,
(28)
A
= Σ log Tr
(29)
Tr(H)
A log Tr(H)
+
(30)
let p; Tr( we have for >>> 1
A
logpi <
p,logp:
(31)
==1
= (p)
(32)
< (p)
monotonicity of Rényi entropy
(33)
consequently,
log Tr(H)
< (H)
(34)
Thus for a: > 1, < 0, larger entropy means smaller Σn_ log TH(
When 0 <<a a < 1, considering Jensen's 's inequality, we have
p
<
Σ p:
1
(35)
18

---

Published as a conference paper at ICLR 2026
Thus,
Pi
1-a
(36)
Using the AM-GM inequality, we will get
12
1/n
1
72
Pc
Pi
(37)
11
71
consequently,
P S n n
(38)
Combining equation
36 and equation 38
we have
II p.
a
I n (n (n
1/1-))
=n 1-n < 1.
(39)
i=1
1=1
Thus we have
logp:
+
log
0 0 =
logps S - -H((p).
(40)
1=1
1=1
1=1
consequently,
X
log
- (H)
(41)
(r(H)
Combine Eq
41
Eq
34
we have for all a 0, a + 1,
1
A log Tr(H)
S (a(H)
(42)
Now we apply Eq
42
Eq
30 and Eq
2 2.4 log 2N
Eg(L(D,0) S Eo(L(S,)) 0)] + 22
(43)
ine nV π exp - ), and H the Hessian matrix of loss function w.rt.
where A
0"
We decompose the bound as Gen(Jo) & (((0)) - + C): where A(0) is paramecerizaaion-deeenede
B(0)+
A(0) = TH
(44)
while B(0) is intarramterizzaiinnnnnaaaai and C' is the constant. Let (0) (S0 S g} denote the reparameterization equivalence class that leaves the predictor fe unchanged (e.g., reparameterizztion induced by homogeneous activation function). Since A()) not invariant and can be arbitrarily altered within [0], thus it is not an identifiable property of fe.
To remove this ambiguity, we define a canonical projection II: [0] + +0] that selects, for every 0, a representative 6* II(0) e [0] satisfying
A( Ao,
(45)
where Ao is constant independent of the underlying function f. Because B is invariant under reparamcterization, we have B(0*) = B(0) =: B(f). Therefore, for every function f.
Gen(f) Gen (fo-) < (A((') + B(0")) g(Ao + B(f)).
(46)
Hence, to an additive constant Ao determined by the canonical projection, generalization is term B. Accordingly, we absorb the trace term into
governed by the reparameterizztion-nnnaa
the constant A, and obtain AA = J nV 2 # eexp! (H)+ J. The reparameterization invariance of the Rényi entropy is proved in Appendix
19

---

Published as a conference paper at ICLR 2026
Corollary C.3 Civen training set S with size N draw from the data distribution 'D and loss function L(.-) E [0. 1l, a layer-wise local minimum 0" R"" is isolated and unique in its neigh- borhood M()*) whose volume V is sufficiently small. pick uniform prior P over 0 E MM ) and pick the posterior Q of density q(o) e-ILo-LAS.90 L(S,) with Lo L(S,0", )* For any 8 (0. 1] and a > O, a + 1, we have with probability at least 1 that:
20 2.A log 2N
Eg[ S Eo|L(S, 0)] + VV
(47)
N -
where A= - V _ expl (H)H+ }, and A is the constant item. H the Hessian matrix of loss function w.E.t. 0"
Txe
20

---

Published as a conference paper at ICLR 2026
D PAC BAYESIAN GENERALIZATION BOUND FOR RÉNYI ENTROPY
Theorem D.1 Given a training set S with N samples draw from the data distribution DD and a loss function L( ) a layer-wise local minimum 6" "R.. assumed that L(D, L(D.8") S L(D, + c), where e is the pertubation to the weights. Consider prior uniform in a ball which contains the ellipsoid that satisfy { 0 (0 0*) H - 0*) S p² }. Take the posterior uniform on this ellipsoid.
For any 5 E (0, 1] and a > 0, a 1, we have with probability at least - 5 that:
1 Ha (H) + log 2/N +C
I(D,0") < I(8,0") 2 +0(e) +1
2
(48)
2(N
Where > the constant term. The condition I((,0*) < I((D, <) means that adding perturbation to weights should not decrease the test error. This is expected to hold in practice for the final solution but does not necessarily hold for any
Proof.
We recall the standard PAC-Bayes bound (e.g. McAllester (2003)): for any prior P independent of the data, with probability at least 1 5 over the draw of the sample S of size N, for any posterior Q we have
DKL (QII) + log N
((0) s Eo-qlis(0)) +
(49)
2(N - 1)
Suppose H is local minimum and in sufficiently small neighborhood we have the quadratic approximation
Ls( Lo + (0 0*) H( 0*) + (0),
[R) S, (50)
with Hessian H 0. We now consider two different posterior distributions Q, both paired with uniform prior P.
Fix independent of H. Define the ellipsoid
En(p) = (0: (0 0-) H(O - 0-) H(0-0')sp') "²}.
We take Q Unif (p)) and the prior Unif( the uniform distribution over a large Euclidean ball BR containing all such ellipsoids.
Step 1. Empirical risk under Q.
With the change of variables y H 0*), Q becomes
uniform on the ball (p). Then
0-) H(O 0*)] = E ( f(r) l- pr
Dr dr
Thus
Ba-oLLs(0) Lo + 2 - 0(s),
which is constant independent of H.
Step 2. KL divergence.
The KL between uniform distributions is log-volume ratio:
Vol(Bn)
Dkl(Q[[P) log Vol (p)()(()))
The ellipsoid volume is
Vol(EH(p) = Vol(B, p" (det -1/2
Hence
DkL(Q[[P) log Vol(BR) log Bo log plooppp0 + log det H.
constant
21

---

Published as a conference paper at ICLR 2026
Step 3. Bound.
Plugging into equation 49 gives
1 log det H + log N + constant
Ee-o|L(0)] < Lo 2n=-2 + +(E)
2
2(N 1)
Thus the only dependence through log det H.
The PAC-Bayes upper bound under quadratic approximation has the form
E [L())] S constant + f(2 log det H)
where f(-) is the complexity term of the chosen PAC-Bayes bound. Thus the only dependence on the curvature И comes from log det H: trace-type terms are absorbed into constants. Take
Taylor expansion at 0* we assume that L(D, 0) I L(D, 0 8), which means adding perturbation weights should not decrease the test error, thus we have
I(0) & (0) + constant + f(} let det H)
Recall Eq 30
Eq. and that Rényi entropy reparameterizatioo invariant, follow the poof in
Appendix C we have
L(0) < (0) + constant
+ / (constant 2 H,(H)
Corollary D.2 Given a training set S with N samples draw from the data distribution 'D and a loss function L( a layer-wise local minimum 0* IR.". We assumed that L(D, 0*) S L(D, 0* + +), where is the pertubation to the weights. Consider prior uniform in ball which contains the ellipsoid that satisfy { (0: (0 H(O 0*) S p² }. Take the posterior uniform on this ellipsoid. For any å (0, 1] and 0, a 1, we have with probability at least - 8 that:
1 (H) + +log N +C
(D) & L(S,0") + 2(+ 8² + +0(s)+
2
(51)
2(N 1)
22

---

Published as a conference paper at ICLR 2026
E REPARAMETERIZATION (SCALING) INVARIANCE OF RÉNYI ENTROPY
Neural networks that use activation functions like ReLU or leaky ReLU exhibit reparametrization- invariant properties. Specifically, when scaling each layer's weights by positive constant, the overall function computed by the network remains unchanged as long as the product of all scaling factors equals one.
For example, consider a network defined as
f(x; (WI. I...., WL)) Wt ReLU(W L- ReLU((122)
where W/ R4xxd-1, If each weight matrix W, is scaled by a positive constant > O, and the scaling factors satisfy I 81 1, then the output of the network remains unchanged for any input x. The sharpness defined by Rényi entropy is invariant under this scaling trick:
Proposition E.1 Consider a feedforward neural network with positively homogeneous ac- tivation function fi.e., o (cx) (α( for all e > 0), and parameters {W1, 1,,..., Wi). J. Let the network output be fxx W o 0.......[VV...... ( ()))). and let [(0) denote the loss function, where denotes the weights of arbitrary layer, i... W, Define the loss Hessian as Hg V2C(0). Con-
sider layer-wise scaling transformation defined by Wi a Wi, a > O, with II q 1.
Let 0 Wi be the scaled parameters, and define H8 the corresponding Hessian. Then the spectrun Rényi entropy of H is invariant:
H. (H))) (HH), Va 0, a#1
(52)
Proof.
The network function f(x) remains unchanged under the layer-wise scaling due the positive homogeneity of the activation since II = 1. Consequently, the loss L()) is invariant:
(()) = L()).
(53)
Thus, the spectrum of H()) will undergo scaling transformation:
H o .o.
(54)
This implies that the eigenvalues {A.) of H satisfy:
A di
(55)
of
Then the normalized spectrum satisfies:
X
d
X Σn
p:
Σ
1
Σ,
Pi.
(56)
so the Rényi entropy remains unchanged:
Ha (H)) 1 log Σ pr0 =
a log ΣΦp pp
(Ho).
(57)
Corollary F.2 Consider a T.-layer feedforward neural network with positively homogeneous ac- tivation function (i.e., ((xx) cx(xx for all e > 0), and parameters (W..... W w Let the o-
network output be f(x)
and let C(0) denote the loss function, where denotes the weights of arbitrary layer, i... W, Define the loss Hessian as Hg = C2)( Con-
sider a layer-wise scaling transformation defined by Wi = a W1, at > 0, with II q = 1.
Let Ô Wi be the scaled parameters, and define Hğ the corresponding Hessian. Then the spectrum-normalizee Rényi entropy of is invariant:
H HHg)) (Ho). Va > 0, a
(58)
23

---

Published as a conference paper at ICLR 2026
Discussion The reparameterization invariance is indeed a scale invariance, the Rényi entropy of the Hessian matrix is not invariant under non-linear reparameterization. We regard reparame- terization invariance necessary, but not sufficient, requirement for studying correlations with generalization. For given minimum, there typically exists a large family of functionally equivalent parameterizations (obtained via reparameterizationn) and optimization may converge any element of this family. To obtain a stable and comparable metric, it is therefore natural to seek quantities that are invariant within this equivalence class, which motivates the necessity of reparameterization invariance.
However, reparameterization invariance by itself does not guarantee strong correlation with gen- eralization. There are many possible invariant candidates, and they differ substantially in how sen- sitively they capture spectral structure. As result, their empirical association with generalization can vary, even though they all satisfy the same invariance requirement.
24

---

Published as a conference paper at ICLR 2026
F CONNECTION BETWEEN GLOBAL AND LOCAL RÉNYI SHARPNESS REGULARIZATION
Proposition F.1 Minimizing the global negative Rényi entropy with order 1 is equivalent, the block-diagonal case, to making each layer's spectrum uniform and balancing trace per dimension across layers. This configuration simaltaneously minimizes the layerwise negative Rényi entropy for all orders a O, including a < 1. With small cross-layer couplings, the same conclusion holds up to a perturbation of order | /T, where is the trace of the global Hessian matrix, and E the difference between the Hessian matrix and the diagonal Hessian matrix. Considering that layer- wise trace can be adjusted without performance degradation, thus balancing trace per dimension across layers doesn't I change the loss. Consequenlly, optimizing the global negative Rényi entropy indeed optimizing the layer-wise negative Rényi entropy, i.e. layer-wise Rényi sharpness.
Proof.
Setup. Let H Rdxd be the (symmetric) Hessian at candidate minimizer; first treat H and discuss standard relaxations in Remark F.7 Denote the eigenvalues by
(H) 2 ...
Aa H 2 O, T= Tr(H) >
Define the normalized spectrum p; (H)
X (H) /T that Σ Pr (H) 1. For define
R. ( (p. HH (H) ==
a - log Ra (H).
(59)
Since x + log x strictly increasing, minimizing - H. (H) equivalent minimizing (H) for any fixed a * (monotone transform).
Assume the network parameters are partitioned into T. layers with dimensions di,.... dr (so E, di = d). Let Her E Iedex be the principal block associated with layer e, with eigenvalues (H 2 ... Ad. (He)) 2 0 and trace T/ == Tr(H) > O. Write
de
= Te (0.1),
A (e
W 1
(H)) =
Σ
T
Te
t=1
EXACT FACTORIZATION UNDER BOCKK-IAAGNAAITY
Lemma F2 (Exact decomposition) If H is block diagonal with blocks HiI,..., HLt then for any a >
Ra (H) Σ wi oa (He).
(60)
proof. The spectrum of a block- block-diagonal matrix the disjoint union of the spectra of blocks. Since pa (H) = A,(H) / and T = -,
2,Th Te, we compute
Σ
de = ΣΣ
(He))) a
de
(Her)
T
Lemma F33 (Power-sum bounds within a layer) Fix e. and set = A (HH)) Te so that 2
and Ddz 34 1. Then:
1. If 1 (convex power), da (HHe)
Σ x 2 di-a. with equality if x, 1/de
(uniform spectrum inside the block).
2. 1f0 < 8 < (concave power), Σ
x S d, 11-8 with equality iff x, = 1/dc
25

---

Published as a conference paper at ICLR 2026
Both from Jensen's inequality (or Karamata's inequality) under the linear constraint
Σ,
Theorem E.4 (Global optimum under block-diagonalii for Assume blk diag(H......... HLL) and a > 1. Then
H
L
R₂ (H) = Σ wt aa (He)
Σ t=1
wi di-a 2 dl-a
(61)
and the following are equivalent:
1. Ra (H) attains its global minimum d d'aaa
2. (Layerwise uniformity) For each & the normalized spectrum inside He is uniform: (HHx))T
Ta is constant across 3. (Trace-per-dimenssonn balancing) The layer traces satisfy wy 4, i.e. dg
layers (equal average curvature per parameter).
proof. The first inequality in equation Hence
61 follows from Lemma
F33 applied to each da (Ha))
R. (H)
Og w
Og == di-a
For fixed positive coefficients as and 1, the function f(w) := Eg arwo is strictly convex on the simplex (w 2 0, Σ; wi = }} and a unique minimizer characterized by the KKT conditions:
w 0-1
we & 0g -1/(a-1) (d/-9)-1/(a-) = de-
Normalizing gives we de/d. Substituting this and the layerwise lower bounds da (He)) 2 de into equation 60 yields
R₂ H > Σ
Σ de d
Equality throughout holds iff (i) each aa (H attains its lower bound, i.e. the layer spectra are uniform, and (ii) We defd. This proves both necessity and sufficiency and the equivalences claimed.
Corollary F.S (Simultaneous layerwise optimality for all orders 3 0, * 1) Under the con- ditions of Theorem F4| the global minimum is attained (equivalently: each block has uniform normalized spectrum and we = de/d), then for every order O,
1
(H B T. ((
the quantity
HH
3 log Σ
is minimized (for all E).
In particular, the same configuration minimizes the layerwise negative Rényi entropy for B > 1 and for 0 < <1 1.
proof. For 3 Lemma F.3 1) shows that the uniform layer spectrum uniquely minimizes E, 2 subject to Σ. 22 1; since the logarithm and the factor (3 1) O are monotone, it also minimizes Hg. For < B < 1, Lemma F3
2) shows that the uniform layer spectrum uniquely
maximizes E. x/; because (3 1)-1 < 0. this again minimizes H3. The claim holds for each layer
STABILITY UNDER CROSS-LAYER COUPLINGS
Real Hessians may not be exactly block diagonal. Write
B blk diag(Hi,............. H,.,)
E = H B.
Note that Tr(E) (off-diagonal blocks contribute trace), hence Tr(H) Tr(B) T.
26

---

Published as a conference paper at ICLR 2026
Proposition F.6 (Perturbation bound for 1) Let
and
set
Ax
max(Amss Am)x (B)). Then
A. a-1 Vd
E
RR (H) R. (B) S a
(62)
T
T
Consequently, BlleTT is small, minimizing R. (H) is appimizzaiinnequivaale up to B[[ [ minimizing R,,,),, which by Theorem F.4 drives each layer toward its uniform spectrum (and hence decreases all layerwise - H3. B > O, simultaneously).
proof. Let {1i} and {ui} be the eigenvalues of H and B sorted in nonincreasing order. By the Hoffiman-Wichee inequality, Ei=(A,, (A, pz)² < | For a 1, the function d(x) r has derivative bounded on [O, by aA.-! Hence by the mean value theorem and Sauchy-Schwarz
Σ X Σnp
o aA
Σ A 1ul-
Vd]
Since Tr(H) Tr(B) = T, dividing both sides by 7 yields equation 62
REMARK BORDER-ROBUSTVE FOR 0 < a 1).
Recall the decomposition (H) = E' wi oa (Ha)). Passing from a > 1 to < a only changes the curvalure of R. (H) and a (Hęz) (from convex to concave) and flips the outer optimization direction (since -- changes sign), but it does not change the location of the optimizer.
<
Consequently, in the block diagonal setting, minimizing the global negative Rényi entropy - Ha (H) for any order ancing trace per dimension across layers; this configuration simultaneously minimizes the layer- wise negative Rényi entropy for all / 0 (including < 1). With small cross-layer couplings HEL) + E. the same conclusion holds up to perturbation of order O E |F (H)) by continuity of Ha in total variation.
0, # 1 is equivalent to making each layer's spectrum uniform and bal-
H blk
Remark F.7 (PSD reduction and alternatives) If indefinite, one may work with (absolute value via spectral decomposition), with a Gauss-Newton/Fisher approximation, or with shifted PSD proxy (e.g. H + 7I with 7 > O), apply the above results verbatim to the PSD object, and then the chosen proxy. The normalized formulation equation 59 is unchanged
truck the dependence as long as the trace >> 0.
27

---

Published as a conference paper at ICLR 2026
G ARBITRARY TRACE RESCALING UNDER FIXED NORMALIZED SPECTRUM
In this appendix, we study how the Hessian trace behaves under linear reparameterizations, and in particular under those that preserve the spectral shape (normalized eigenvalue distribution) of the Hessian. We show that, for each individual model, such reparameterizations give continuous family of possible scalings of the Hessian trace. For a finite collection of models, this leads to an explicit infinite feasibility condition under which all Hessian traces can be aligned to common value while preserving spectral shape.
Let w E R" denote the parameter vector of given (layer of a) model, and let H E R be the corresponding Hessian at a local minimum. Throughout this subsection, we assume that H is symmetric positive definite.
We consider linear reparameterizations of the form
w A0,
A E R invertible,
(63)
and define the reparameterized loss by L(O) = L(A0). By the chain rule, the Hessian 0- coordinates is
Hg = V2T(()= = A HA.
(64)
The corresponding parameter vector is
A w..
(65)
are particularly interested in reparameterizations that preserve the spectral shape of the Hessian, ie. that only rescale all cigenvalues by common positive factor.
Lemma G.1 ( reparameterizaation)) Let H O. For any scalar d and any orthogonal matrix Q E =x (Q Q = 1), define
A(d, Q) H -/2 (dQ) 1//
(66)
Then the corresponding reparameterized Hessian Ha A(d, Q) HAdd, Q) satisfies
Hg d H.
(67)
In particular, the eigenvalues of Hg are {d))(H)), so the normalized spectrum (T((H))TT(HH)) coincides with that of H.
Proof. A direct computation yields
Ha A(d, Q) H Q)
(1 dh (12) и (H=1)2dQHH1))
de 11 --122H->/2@H"1?
= da 2²22Q -==2 12
Thus all eigenvalues scaled by d,, and the normalized eigenvalue distribution is unchanged.
We next study the effect of equation 66 on the parameter norm. Let
H 1/2 w
||u|22 > 0.
(68)
For A A(d, Q) as in Lemma G.1 we have
1 " -1/2 1/2
A-tw = H 1/2 (dQ) (12))
H 11.Quu.
(69)
Let Tr(H) denote the original trace. Under the reparameterization with factor d² we have
Tr(Ha) dTr(H).
(70)
28

---

Published as a conference paper at ICLR 2026
As d E I. can adjust the trace of the Hessian matrix to arbitrarily prescribed value while keeping the normalized eigenvalue spectrum completely unchanged.
Since sharpness measure is defined in terms of the normalized spectrum (c.g. via the Rényi tropy of (H)))T(HHH))) the global scale of the trace is factored out by design. Combining this observation with the reparameterization freedom described above, conclude that scale-dependent quantities such as the raw trace (HH do not carry reparameterivaaion-rrous geometric informa- tion. What remains intrinsic is precisely the shape of the Hessian spectrum, which we quantify via its Rényi entropy.
29

---

Published as a conference paper at ICLR 2026
H ON THE DISCUSSION OF SAM VARIANTS
In this section, we discuss several sharpness-aware minimization variants and compare them with Rényi sharpness-aware minimization (RSAM). We focus on closely related methods, including the original SAM (Foret et al. 2020), Sparse SAM (Mi et al., 2022). Eigen SAM (Luo et al. 2024).
Tilted SAM (Li al. 2024b), Frobenius SAM (Tahmasebi al. 2024p. and Fisher SAM (Kim
et al. 2022).
Vanilla SAM has been shown implicitly minimize the largest eigenvalue of the training loss Hes- sian (Wen et al.) 2023), and Sparse SAM, which accelerates SAM by explicitly masking part of the updates, essentially targets the same quantity. Eigen SAM directly penalizes the largest eigenvalue in its minimization step. Tilted SAM samples noise in multiple directions to perturb the weights and penalizes the sum of the exponentiated perturbed losses over these noise samples. Intuitively, the exponential transform amplifies the sharpest directions of the loss landscape, so it imposes stronger penalty along these directions. From this perspective, Tilted SAM can be viewed as ef- fectively penalizing the largest (or relatively large) eigenvalues of the Hessian. Frobenius SAM penalizes the Frobenius norm of the Hessian matrix; if we normalize this norm by the squared trace, the resulting quantity becomes essentially monotone function of the order-2 Rényi entropy. Fisher SAM minimizes the same type of robust objective as SAM, but with the neighborhood defined by Riemannian metric induced by the Fisher information; this is equivalent penalizing the largest cigenvalue of the Hessian with respect to the Fisher metric.
Overall, these methods regularize some spectral function of the Hessian eigenvalues. Whether one penalizes the largest eigenvalue or minimizes the Frobenius norm of the Hessian, the implicit goal to encourage the eigenvalues move closer each other; for example, reducing the largest eigenvalue typically decreases the overall spread of the spectrum.
In contrast, Rényi sharpness explicitly focuses on the dispersion of the normalized eigenvalues. Modern deep models usually enjoy certain reparameterization invariances, so we can rescale the overall magnitude of the Hessian without changing the model's behavior. Consequently, the regu- larizer depends only on the unnormalized eigenvalues (such as the spectral norm or generic spec- tral function), then shrinking the global scale of the Hessian will always reduce the regularization term, even when the model performance and the relative shape of the spectrum remain unchanged. Therefore, minimizing such penalties alone does not guarantee that the eigenvalues become more uniformly distributed.
30

---

Published as a conference paper at ICLR 2026
I SHARPNESS MEASURES
In this section, we give detailed introduction the sharpness measure use. The content of this section refers to Jiang et al. 2019) and the original works corresponding to these measures.
1.1 NORM BASED MEASURES
Several generalization bounds have been proved for neural networks using margin and norm no- tions. In this section, we go over several such measures. For fully connected networks, Bartlett & Mendelson 2002) have shown a bound based on product of É100 norm of the layer weights times a 2" factor where (,o is the maximum over hidden units of the &2 norm of the incoming weights the hidden unit. Neyshabur et al. (2015) proved bound based on product of Frobenius norms of the layer weights times 2d factor and Golowich al. (2117) was able improve the factor vd. Bartlett et al. (2017) proved bound based on product of spectral norm of the layer weights times sum over layers ratio Frobenius norm spectral norm of the layer weights Neyshaabr al.
(018a) showed similar bound can be achieved in simpler way using PAC-bayesian framework.
Spectral Norm Unfortunately, none the above founds are directly applicable to convolutional networks, Pitas et al. (207)) built on Neyshabur et al. 200888) and extended the bound on the spectral norm to convolutional networks. The bound is very similar to the one for fully connected networks by Bartlett et al. (2017). We next restate their generalization bound for convolutional networks including the constants.
Theorem L.1 (Pitas et al. (2017) Let B upper bound the E norm of any point in the input domain. For any B, 7, >> 0, the following bound holds with probability 1 ö over the training set:
(ABEEL k/G+ +enett/innn2 Vin(An2d)
IT I. 1. E
In in("))
|W
S \/
(71)
2m
Parameter Norm
Given recent evidence the importance of distance to initialization (Dziu-
gaite & Roy 2017
Nagarajan & Kolter 2019: Neyshabur et al. (018b)) we calculate the following
measures:
Prisheninsddistance (fw)
I. w
(72)
In the case when the reference matrix p for weights, Eq (22 the Frobenius norm of the parameters, which also corresponds to the distance from the origin:
Mparam norm (Sw)
(73)
Fisher-Rao Norm Fisher-Rao metric was introduced in Liang et al. (017 as a complexity mea- sure for neural networks. Liang al. (2017) showed that Fisher-Rao norm is lower bound on the path-norm and it correlates in some cases. We define a measure based on the Fisher-Rao metric of the network:
J4iisherRRoo (fw) (fw)) (d 1)²
Σ w NoVel(fwe (f.())
(74)
m
where é is the cross-entropy loss.
Trace Trace measure is defined as the trace of the Hessian matrix of the loss function on the training dataset with respect to the weights, i.e., Tr(H), where H V2 I(S, w).
31

---

Published as a conference paper at ICLR 2026
1.2 FLATNESS- BASSED MEASURES
PAC-Bayesian framework (McAllester 1999) allows us to study flatness of a solution and connect it generalization. Given prior P is chosen before observing the training set and posterior Q which is distribution on the solutions of the learning algorithm (and hence depends on the training set), we can bound the expected generalization error of solutions generated from Q with high probability based on the DKL divergence of P and Q. The next theorem states a simplified version of MAC-Bayesian bounds.
Theorem I.2 For any 8 > O, distribution D. prior P, with probability 1 over the training set, for any posterior Q the following bound holds:
Ev~q [L(fv]] Ew~Q [E(Fv)] +V Dkr.(Q]IP) + log
(75)
2(m-1)
P and Q are Gaussian distributions with P = NNup, amd Q = -N(uq 2g)) then the DKL-term can be written as follows:
(Na,, Eo)lWW(P Ep)) = 1 tr (EET) + (uq mP) Ep' (nQ - -pp) k + In( det Ep det Eq
2017), the DKL. term will However, since a belongs to prior, if we search to find a value for a, we need adjust the bound reflect that. Since search over less than 20000 predefined values of a in our experiments, we can use the union bound which changes the logarithmic term to (og(2000mm// and we get the following bound:
Setting Q = [[w, o²1) and P P=W(w", similar to Neyshabur et al.
ww 20
be simply
ww-w"
log
+ 10
(u--N(u,oP)) [T([w+u)] S oun [B(Swtu)]
+
(76)
m
Based on the above bound, we define the following measures using the origin as reference tensors:
||w]
m 10
Mpac-bayes-orig (fw)
+ logí
+
(77)
40
where a is chosen to be the largest number such that 2() [f(w--u]
< 0.1.
To understand the importance of the flatness parameters a, we also define the following measure:
1
hpae- flatnes (Sw) d
(78)
where a is computed as explained above.
I.3
SHARPNESS-BASEE MEASURES
SAM
Foret al. 2020 proposed a generalization bound under weight perturbation:
Theorem 1.3 For any p > 0 and any distribution D, with probability 1 over the choice of the training set S ~ D,
A koog
1+
1
log(n)
+ 4 log 2 +((
Ln(w) < max Ls(w + +) + [cl/SS
(79)
where = 1S}, k is the number of parameters and we assumed Ln(w) < Ee !nw + +)]]
Thus, the sharpness of SAM is defined
max Ls(w w + - (s()
(80)
32

---

Published as a conference paper at ICLR 2026
if we minimize max||e|l; Ls c), E(w) = sign g
the solution via first-order approximation will be
1 -
g VLs(w),
(81)
|g
P
9
Especially, p = 2
<(w) |1g12
g = VLs(w).
(82)
and if p- 00
El (w) = p (g),
- VLs(w)
(83)
ASAM Kwon et al. (2021) proposed new adaptive sharpness which is reparamcterization invari- with normalization operator:
Definition I.4 (Normalization operator) Let (Tw, w E {R² be a family of invertible linear oper- ators on TR. Given a weight w, T A = Tw] for any invertible scaling operator A on TR.h which does not chunge the loss function, we say Tw2 is normalization operator of w.
Using the normalization operator, we define adaptive sharpness as follows.
Definition 1.5 (Adaptive sharpness) If Tw is the normalization operator of w in Definition L.4 adaptive sharpness of w is defined by
max Ls(w 6) - -Ls(w)
(84)
| c1
where 1 S cC.
They also demonstrated generalization bound for adaptive sharpness:
Theorem Let Twi be the normalization operator on RR If LD() S ELnwoor
for some O, then with probability 8,
Lp(w)=
< max Ls(w + +) +
(85)
172p2
||T ellaSp
where h IR+ > RR+ is a strictly increasing function, n = |S| and vko(1 + /log//))/
For minimax problem
min
max Ls(w E) |w T Sp
(86)
The solution under first-order approximation for adaptive sharpness is Tw V L. (w) w
pT sign sign((Ls(ww)
(87)
[ V Ls (w)
Especially, i 2,
L T V L w
(88)
and if p 00,
E = pl sign V Ls (w)).
(89)
I.4 IMPLEMENTATION
The measures, including Fisher-Rao norm (Eq. 74). Parameter norm (Eq. 73). Trace of the Hessian
matrix, Pac- Bay-Baaes from origin (Eq 77). and the Pac-Bayes flatness (Eq. 78) are computed through
the repository by Dziugaite al. 2020)
The measures, including Le sharpness (Eq. 82 Lo sharpness (Eq, 83 Le adaptive sharpness (Eq. 88 and Loo sharpness (Eq. 83) compute the corresponding sharpness directly from the solution of
the minimax problem.
For the SAM(ASAM) sharpness, conducted grid search over {10-9 x 10 10-6. 10-5, 3 x 10 10-5 5 10-4 4, 3x 10 10-3, 3 10-3, 3, 10-2, 3x 10-2 22 10-1, 0.3, 1} and select the with highest correlation coefficient for each task.
33

---

Published as a conference paper at ICLR 2026
J EXPERIMENTAL DETAILS
In this section, we describe the datasets, models, hyperparameter choices, and eigenspectrum adjust- ment used our experiments. All our experiments are run using PyTorch on Nvidia GTX1080ti, RTX3090s, RTX4090s, and RTX5090s.
J.1 DATASET
CIFAR-10. CIFAR-10 consists of 60,000 color images, with each image belonging to one of ten different classes with size 32 32. The classes include common objects such airplanes, automo- biles, birds, cats, deer, dogs, frogs, horses, ships, and trucks. The CIFAR- 10 dataset is divided into two subsets: training set and a test set. The training set contains 50,000 images, while the test set contains 10,000 images (Krizhevsky & Hinton 2009). For data processing, we follow the standard augmentation: normalize channel-wise, randomly horizontally flip, and random cropping.
CIFAR-100. The CIFAR-100 dataset consists of 60,000 color images, with each image belong- ing to one of 100 different fine-grained classes (Krizhevsky & Hinton] (209)). These classes are organized into 20 superclasses, each containing 5 fine-grained classes. Similar to CIFAR-10, 1-, the CIFAR-I00 100 dataset is split into a training set and test set. The training set contains 50,000 images, and the test set contains 10,000 images. Each image is of size 32x32 pixels and is labeled with its corresponding fine- aiee-graied class. Augmentation includes normalize Channel-wise, randomly hori- zontally flip, and random cropping.
Tiny ImageNe.. TinyImageNet comprises 100,000 images distributed across 200 classes, with each class consisting of 500 images (Le & Yang 2015). These images have been resized 64 x 64 pixels and are in full color. Each class encompasses 500 training images, 50 validation images, and 50 test images. Data augmentation techniques encompass normalization, random rotation, and random flipping. The dataset includes distinct train, validation, and test sets for experimentation.
J.2 MODEL
In all experiments, the neural networks are initialized by the default initialization provided by Py-
torch.
ResNet18, ResNet20, ResNet34 and ResNet50 (He et al. 2016). We use the standard ResNet architecture for TinyImageNet and tune for the CIFAR dataset on the correlation validation tasks. The detailed network architecture parameters are shown in Table 3 and Table ResNer18 18, ResNet20, ResNet34, and ResNet56 are trained on CIFAR-I00 100. The standard ResNet18 trained on TinylmageNe for efficient computing and tuned ResNet18 is trained TinylmageNet for sharpness-awant aware minimization.
WideResNet (Zagoruyko & Komodakis, 2016). The Wide ResNet implementation uses the wrn28 10 model from the horuma (Hataya 2018) library. Architecture details can be found in
Table 4
Vision Transformer. We use the ViT architecture from the vit -pytorchh library, which a modification of the standard VIT (Dosovitskiy et al. (2020) with a fixed positional embedding and global average pooling instead of the CLS embedding.
J.3 TRAINING
SETUP
J.3.1 CORRELATION EXPERIMENTS
We train models for 200 epochs, and cosine learning rate decay is adopted after a linear warm-up for the first 10 epochs. For the task on CIFARIO/CIFARION we vary the initial learning rate (0.001, 0.03, 0. I, batch size (128, 384, 1280), and weight decay {0.00001, 0.00005, 0.0001, 0.0003, 0.0005} } for SGD with momentum and the initial learning rate (0.00001, 0.0003, 0.001), ). batch size (128, 384, 1280). and weight decay (0.00001, 0.00005, 0.0001, 0.0003, 0.0005) for Adam. For
34

---

Published as a conference paper at ICLR 2026
Table 3: ResNet architecture used correlation experiments.
Layer
ResNet188IFAR
ResNet34
ResNeet88 Tinybaggrea
3x3, 64
3x3, 64 padding I stride
7x 64 padding 3
Conv
padding I stride
stride 2
Max Pool, ks 3, str pad
Layer stack
3x3, 3x3, 64
3x3, 64 3x3,, 64
xx3, 64 3x3, 64
X2
x3
x2
Layer stack
3x3, 128 3x3, 128
3x3, 128 3x3, 128
3x3, 128
3x3, 128
x2
Layer stack
3x33, 256
3x3, 256
3x3, 256
3x3, 256
3x3, 256
3x3, 256
x2
Layer stack
3x3,, 512 3x3, 512
3x3, 512
3x3, 512
x2
3x3, 512
3x3, 512
x2
Adaptive Ave Pool, output size (1,
FC
512 N_ CLLASSSSS
512 CLASSES
512 N_ CLASSES
Table ResNet architecture used in sharpness-aware minimization experiments.
Layer
ResNet-20
ResNet-56
ResNet-50
WideResNet-28-10
3x3, padding stride I
3x3, packing stride
3x 64 padding stride
3x3, 16 padding stride I
Conv 1
Layer stack
16 3x3, 16
3x33, 3x3, 16
1x 64
3x3, 160 3x3, 160
x3
x9
3x3, 64
x3
x4
1x 256
Layer stack
3x3, 32 3x3, 32
3x3,, 32
1x1, 128
3x3, 320
x3
3x3,, 32
x9
3x3, 128
x4
3x3, 320
x4
1x1, 512
Layer stack 3
3x33, 64 3x3, 64
x3
3x33, 64 3x3,
x9
3x3, 256
1x 256
x6
3x3, 640
3x3, 640
Ix1. 1024
Layer stack 4
Ix 512
3x.3, 512
x3
1x 2048
2048
Avg Pool, kernel size 64 x CLASSES
Avg. Pool, kernel size 64 N_CLASSES
Adaptive Avg Pool, output (1,
Avg Pool, kernel size 640 N_CLAASES
FC
2048 N_CLASSES
the task on TinylmageNee we vary the initial learning rate (0.001, 0.03. 0.1), batch size (128, 384, 1280), and weight decay (0.000003, 0.00001, 0.00003, 0.00005, 0.0001, 0.0003) for SGD with momentum and the initial learning rate (0.00001, 0.0003, 0.001), batch size {128, 384, 1280), and weight decay (0.000003, 0,00001, 0.00003, 0.00005, 0.0001, 0.0003) for Adam.
Different from Diang et al. 2019), we pick the data augmentation in the training scheme, which is common setting in modern deep learning, but we still compute the sharpness measure without data augmentation, as from theoretical perspective, data augmentation also challenging to analyze since the training samples generated from the procedure no longer identical and independently distributed.
To investigate the relationship between sharpness and generalization under common training strate- gies, we pick the stopping criterion based on the number of iterations or the number of epochs. To avoid differences in optimization speed across hyperparameter settings, follow the linear scaling rule recommendated by Goyal et al.] (2017) and scale the learning rate and batch size in tandem, which yields comparable convergence after the same number of epochs.
35

---

Published as a conference paper at ICLR 2026
J.3.2 SHARPNESS-AWAR MINIMIZATION EXPERIMENTS
Firstly, we will introduce the Rényi Sharpness-Awar Minimization algorithm as follows:
Algorithm 2 Rényi Sharpness- Aware Minimization (RSAM) Algorithm
Loss function f, training dataset S == - ((x,,))) mini-batch size h, radius Rényi order o, plain SGD epoch 61, RSAM epoch eg, weight decay coefficient X, scheduled learning rate B, initial weight Wo- Output: Trained weight w. Initialize weight w f- wo
Input:
for i 1,...,,, E₁ do
1). Sample mini-batch B of size b from S ((VLB(W) Aw)
2).
end for
for j = 1,,.., e2 do
4). Sample mini-batch B of size b from S
5). e + p sign(11 - a)- Σ, (VL 1
(,3 (L) Ju VLB(w)
6). w - (VVLB(W++
+ +) Aw)
end for Return: w
We first train the neural network with vanilla SGD for en epochs, without applying the Rényi regu- larizer. The intuition is that the gradient-magnitue more accurate only after the model has achieved reasonable training loss/accuracy, the Rényi term at the very beginning of training is unnecessary and may even be harmful. Once the model reaches this warm-up stage, we activate the Rényi regularizer. For each mini-batch B, we compute the loss LB(w) (w) and its gradient L (ww). We then construct the perturbation e according Eq. 13 and form the perturbed parameters w + e. Next, we evaluate the gradient the perturbed point, V L <n + E). and perform gradient-descent step on the original parameters w using this gradient. This procedure structurally identical to SAM (Foret al., 2020) and ASAM (Kwon) et al. 202T); the only difference lies in how the perturbation is computed, which in our case is defined by the Rényí sharpness objective in Eq. [3
approximation underlying our method becomes
penalizing
We set for SAM and Eigen-SAM as 0.05 for CIFARIO and 0.1 for CIFARIOO, and p for ASAM as 0.5 for CIFARIO and 1.0 for CIFAR100. 7 for ASAM is set to 0.01. p and a for RSAM describled in Table. and Table. 6 The mini-batch size is set to 128. The number of epochs is set to 200 for SGD, SAM, ASAM, Eigen-SAM. and RSAM. Although prior work recommends training SGD for 400 epochs assess improvements under matched compute budget, RSAM introduces the regularizer only after a warm-up period, so compute parity no longer holds. Moreover, those studies have already shown performance superior 400-epoch SGD. Consequently, our experiments are not strictly designed under equal-compute conditions. Momentum and weight decay coefficient are set to 0.9 and 0.0005, respectively. Cosine learning rate decay is adopted with an initial learning rate of 1. Also, random cropping, padding by four pixels, normalization and random horizontal flip are applied for data augmentation. As label smoothing is not adopted in Eigen- SAM, all experiments are conducted without label smoothing.
For the evaluations SAM, and RSAM on TinylmageNet. We apply p 0.05 for SAM and Eigen-SAM and p 1.0 for ASAM. p for RSAM is set to The number of training epochs are all set to 100. We use a mini- batch size of 128, an initial learning rate of 0.2, and SGD optimizer with weight decay coefficient of 0.0001. Other hyperparameters are the same as those of CIFAR- 10/100 tests.
larger scale, we compare the performance of SGD, SAM, ASAM, Eigen-
All the hyper-parameter, are summarized in Table Table 6
and Table 7
J.4 RÉNYI ENTROPY COMPUTATION SETUP
The Rényi entropy is computed on the subset of the training dataset. For the CIFARIO and CI- FAR100 datasets. we randomly sample 2000 samples to compute Rényi entropy (1000 for VIT on CIFAR10), and for the TinylmageNet dataset, we randomly sample 1000 samples. Batch size is set to 128. 1 100 and = 15 are set for the Rényi entropy estimation algorithm. The Rényi order is
36

---

Published as a conference paper at ICLR 2026
Table Hyper-parameters of Sharpness-aware Minimization on CIFARIO
Momen
SGD Epochs
SAM Epochs
Batch Size
Weight
Algorithm
Model
-tum
LR
Decay
17
ResNet20
0.9
0.1
200
128
0.0005
0
0
0
SGD
ResNet56
0.9
0.1
200
128
0.0005
0
0
0
WideResNet-28-10
0,9
0.1
200
128
0.0005
0
0
ResNet20
0,9
0.1
0
200
128
0.0005
0.05
0
0
SAM
ResNet56
0.9
0.1
0
200
128
0.0005
0.05
0
WidcResNet-28-10
0.9
0.1
0
200
128
0.0005
0.05
0
ResNer20
0.9
0
200
128
0.0005
0.5
0.01
ASAM
ResNet56
0.9
0.1
200
128
0.0005
0.5
0.01
WideResNet-28-10
0.9
0.1
0
200
128
0.0005
0.5
0.01
ResNer20
0.9
128
0.0005
0.05
0
0.2
Eigen-SAM
ResNe156
0.9
128
0.0005
0.05
0
0.2
WideResNett-8-100
200
128
0.0005
0.05
0.2
ResNet20
0.9
0.1
0
200
128
0.0005
0.1
1.0
ESAM
ResNet56
0.9
128
0.0005
0.1
1.0
WidcResNet-28-10
0.9
200
128
0.0005
0.1
1.0
ResNet20
0.9
0.1
0
200
128
0.0005
0.2
0.0
0
SSAM
ResNet56
0.9
0.1
0
200
128
0.0005
0.2
0.0
0
WidcResNet-28-10
0.9
0.1
0
200
128
0.0005
0.1
0.0
0
ResNer20
0.9
0.1
5
195
128
0.0005
0.65
0
1.2
RSAM
ResNet56
0.9
0.1
5
195
128
0.0005
0.8
0
1.2
11deResNet-28-10
0.9
0.1
5
195
128
0.0005
0.3
0
1.05
Table 6: Hyper-parameters of Sharpness-aware Minimization on CIFAR100
Momen -tum
SGD Epochs
SAM Epochs
Batch Size
Weight Decay
Algorithm
Model
LR
1]
ResNet20
0.9
200
128
0.0005
0
SGD
ResNet56
0,9
200
128
0.0005
0
WidcResNet-28-10
0.9
0.1
200
128
0.0005
0
0
ResNer20
0.9
128
0.0005
0.1
0
SAM
ResNet56
128
0.1
WideResNNt-28-10
128
0.1
ResNet20
0.9
0.1
0
200
128
0.0005
1.0
0.01
0
ASAM
ResNet56
0.9
0.1
0
200
128
0.0005
1.0
0.01
0
WideResNee-28-10
0.9
200
128
0.0005
1.0
0.01
ResNer20
0.9
0.1
0
200
128
0.0005
0.1
0
0.2
Eigen-SAM
ResNet56
0.9
0.1
0
200
128
0.0005
0.1
0
0.2
WideResNet-28-10
0.9
0.1
0
200
128
0.0005
0.1
0
0.2
ResNet20
0.9
0.1
0
200
128
0.0005
0.1
1.0
0
ESAM
ResNet56
0.9
0.1
0
200
128
0.0005
0.1
1.0
0
WideResNet-28-10
0.9
0.1
0
200
128
0.0005
0.1
1.0
0
ResNet20
0.9
0.1
0
200
128
0.0005
0.5
0.0
0
SSAM
ResNet56
0.9
0.1
0
200
128
0.0005
0.5
0.0
WidcResNet-28-10
0.9
0.1
0
200
128
0.0005
0.2
0.0
0
ResNer20
0.9
128
0.0005
0.76
0
1.1
RSAM
ResNet56
0.9
195
128
0.0005
0.9
0
1.1
WideResNet-28-10
0.9
0.1
5
195
128
0.0005
0.7
0
1.05
chosen from (0.0001, 0.01, 0.03, 0.06, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.99, 0.999, 1.001, 1.01, 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 2.0, 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 2.8, 2.9, 3). Due to the fact that training cannot guarantee convergence exactly to a strict local minimum, negative eigenvalues are inevitable, which can cause numerical pathologies for the Rényi entropy Therefore, when assessing how a affects the correlation between Rényi entropy and generalization, restrict to (0. 0.9) and (1.2, 3.0]. Within these ranges, computing the Rényi entropy is stable and free anomalies. During our analysis of the sharpness-gent generalization correlation, we vary a and plot the sharpness that attains the highest correlation coefficient.
1.
37

---

Published as a conference paper at ICLR 2026
Table plyper-parameters of Sharpness-aware Minimization Tinylmagen
Momen -tum
SGD Epochs
SAM Epochs
Batch Size
Weight Decay
Algorithm
Model
LR
7
a
SGD
ResNet50
0.9
0.2
100
0
128
0.0001
0
0
0
SAM
ResNer50
0.9
0.2
0
100
128
0.0001
0.05
0
0
ASAM
ResNer50
0.9
0.2
0
100
128
0.0001
1.0
0.01
0
FSAM
ResNet50
0.2
100
128
0.0001
0.5
0.1
0
RSAM
ResNet50
0.9
0.2
20
80
128
0.0001
1.25
0
1.1
Note. In practice, train with SGD until the validation Top-1 accuracy exceeds 30%, then switch to RSAM; this
typically uround epoch 20.
Table 8: Hyper-parametce of Sharpness-aware Minimization on ViT-B/16 Finetuning
Momen -tum
SGD Epochs
SAM Epochs
Batch Size
Weight Decay
Algorithm
Dataset
LR
CIFARIO
0.9
0.01
20
0
128
0.0005
0
0
0
SGD
CIFAR100
0.9
0.01
20
0
128
0.0005
0
CIFARIO
0.9
0.01
20
128
0.0005
0.05
0
0
SAM
CIFAR100
0.9
0.01
20
128
0.0005
0.1
0
0
CIFARIO
0.9
0.01
20
128
0.0005
0.5
0.01
0
ASAM
CIFARI00
0.9
0.01
20
128
0.0005
1.0
0.01
0
CIFARIO
0.9
0.01
20
128
0.0005
0.1
1.0
0
FSAM
CIFARI00
0.9
0.01
20
128
0.0005
0.1
1.0
0
CIFARIO
0.9
0.01
18
128
0.0005
0.8
1.3
RSAM
CIFAR100
0.9
0.01
18
128
0.0005
0.6
0
1.1
J.5 COMPUTATION OF OTHER SHARPNESS MEASURES
We detail how the remaining sharpness measures are computed. Following the public implementa- tion of Dziugaite et al. (2020), we compute the PAC-Bayes-based measure and estimate the Hessian trace via Hutchinson's 's trick (Eq. 99. The Fisher-Rao norm is computed as in Petzka et al. (021 For SAM and ASAM, we sweep E
(10-5, 3x 10 10-3 3x10 1x 30-
3x 10-2 3x10- 10-1 0.3,
and report the sharpness at the value of p that yields the highest correlation with generalization. Because SAM/ASAM sharpness is defined with respect to the entire dataset, we evaluate on subsample of 1,000 training examples using single batch of size 1,000 (rather than mini-batches). Data augmentation is disabled during these computations. We evaluate sharpness only for pertur- bations that do not induce a large increase in the loss. Once the loss rise becomes substantial, the perturbed point should no longer be regarded as residing in the neighborhood of the minimum. For instance, when the unperturbed loss approximately 0.001 (accuracy approximately 100%) but rises to 5.2 after perturbation (accuracy dropping to 20% or lower), the perturbation has evidently moved the parameters outside the minimum's 's basin. Notably, the formulations of SAM and ASAM presuppose that weight perturbations remain within the local neighborhood of the minimum.
38

---

Published as a conference paper at ICLR 2026
K FULL RESULTS
In this section, we report all the results the tasks in the main body.
K. HESSIAN SPECTRUM
In this section, we provide some spectra of the trained models the correlation validation experi- ments, including ResNet18 8 and ResNet34 on CIFAR1O and ResNet18 8 and ResNet34 on CIFAR 100.
M
M
Figure 4: Spectrum of ResNet18 CIFARIO.
Ae Dew
Oge man slaee
0201
Figure 5: Spectrum of ResNet34 on CIFAR10.
39

---

Published as a conference paper at ICLR 2026
alayer
Corviced
Figure 6: Spectrum of 8 on CIFARI00.
2.01
630
020
459 ado
1
62
Niw Mia
Py 24
X1
E1
21
X1 Sich
M0,
Figure 7: Spectrum of ResNet34 on CIFAR100.
40

---

Published as a conference paper at ICLR 2026
K.2 CORRELATION BETWEEN RÉNYI SHARPNESS AND GENERALIZATION
In this section, we provide the figures about the correlation between generalization and multiple sharpness measures. We can find that Rényi sharpness is strongly correlated with generalization than the other measures.
30182
45
3764
1779
12433
Betweringg
Figure 8: ResNet18 on CIFARIO, The layer to all layer subplots correspond to the Rényi sharpness measure.
BERS
26467
1010
02515
1791
(417
12244
1.0 2.03
Figure 9: ResNet34 on CIFAR1O, The layer 1 to all layer subplots correspond to the Rényi sharpness measure.
Tisk Conrelation cocticcien
CIFAR 16 0,2092
CIFAR Ne134
CIFARIONE 0.19554
CIPARIGIVR=NNH118
CIFAR C4FAR
Tisy TryyImageNee//ooNeelt -0.5053
0,2966
-0.3149
0.5310
Table 9: Correlation coefficient between log det H and generalization gap across tasks. H is the Hessian matrix of the training loss with respect to the whole weights in the model.
41

---

Published as a conference paper at ICLR 2026
WFT
04333
L- 0432
L- 023H
1- 64380
L- EOKE
+08 -15
04822
03977
+15 1E
Figure 10: VIT on CIFARIO, The layer to all layer subplots correspond to the Rényi sharpness measure.
1100
3:30
115
Her
1-66
170
113
T- 08507
20542
0000000
112
1280
posence
Y- 0-0055
T- C000
rod 447
44440
Too 485L
Figure 11: ResNet18 on CIFAR100, The layer all layer subplots correspond the Rényi sharp- ness measure.
42

---

Published as a conference paper at ICLR 2026
CALLS
THERS
315
1/
131
1
111
110
Spr2
11
0.091
T-1
T-3807
0.99
C-
115
1/5
131
175
185
1:11
123
110
1:0
21
2:00
12
110
10
03733
1343
14-43241
115
13:
1/5
175
185
110
110
640
Figure 12: ResNet34 on CIFAR100, The layer to all layer subplots correspond to the Rényi sharp- ness measure.
13
kpw
Mer1 cw-17
20000
1000
Figure 13: ResNet18 on TinylmageNet, The layer to all layer subplots correspond the Rényi sharpness measure.
43

---

Published as a conference paper at ICLR 2026
K.3 MORE RESULTS FOLLOWING AADRIUSHCHENES ET AL. 2023
K.3.1 MORE TRAINING RECIPE FOR RESNET18 8 ON 10
In previous work, Andriushchenko et al. 2023) showed that their sharpness measure correlates strongly with generalization only within certain hyperparameter subsets or sub-groups. To perform similar test, we extend our standard ResNet- Saae--////FRRRRR setup by introducing two additional
hyperparameter dimensions: with/without mixup (a = 0.5) (Zhang et al. 2017) and with/without standard augmentations combined with RandAugment (Cubuk et al. 2020 We then compute the Rényi sharpness of the last two layers and compare it with other sharpness measures. The results in Fig. 14 indicate that, even under these richer hyperparameter combinations, Rényi sharpness still exhibits a strong and consistent correlation with generalization.
E330
33372
Figure 14: ResNet18 on CIFAR10 with more training configurations. The learning rate, batch size, optimizer, and weight decay are varied following standard Caneee-----o--IIA---- setups, and we further introduce variants with mixup (a 0.5) (Zhang et al. 2017) and standard augmentations combined with RandAugment (Cubuk al. 2020 resulting four times as many models as in the standard setting.
K.3.2 PRETRAINING VIT-B/16 ON IMAGENET- 1K
Following Andriushchenko et al. 2023 we evaluate VIT models from Steiner et al. (2021). l), using ViT-B/16-224 weights. Those were trained from scratch on ImageNet- Ik for 300 epochs with dif- ferent hyperparameter settings, and subsequently fine tuned the same dataset for 20.000 steps with 2 different learning rates. The different hyperparameters include augmentations, weight decay, and stochastic depth dropout, leading to a rich pool of 56 models. As shown in Figure 15 Rényi sharpness still exhibits a strong and consistent correlation with generalization, while others not.
T BRISS
Tw 02724
0 0.6967
T -00277
Tw 055S
1.0
1.3
30
200093 a0c033 SXCRE XX00e
0.00
1.33
375
2003
4078
ECES
-34335
4.5009
To 35545
1.0
2.5
2.5
5.54
0.00
0.55
644
2.5
3.0
Figure 15: ViT-B/16 trained from scratch on ImageNet-Ik. We show for 56 models from Steiner al. 202T) the generalization gap vs. various sharpness measures. Overall, Rényí sharpness is still strongly correlated with generalization than the other measures.
K.3.3 FINE-TUNING ON IMAGENET- 1K FROM CLIP
We also follow the experiments that investigate fine-tuning from CLIP Radford et al. (202 l). We study the pool of classifiers obtained by Wortsman et al. (022)), who fine-tuned a CLIP ViT-B/32 model on ImageNet multiple times by randomly selecting training hyperparameters, including learn- ing rate, number of epochs, weight decay, label smoothing, and augmentations. We compute the Rényi sharpness of the last layer within the VIT-B/32 model, and compare it with other sharpness
44

---

Published as a conference paper at ICLR 2026
measures. One confirm from Fig. 16 that Rényi sharpness still exhibits strong and consistent correlation with generalization, compared to the other measures.
leyer
Lok
tieer
eachers Vat
2011
2262
10
65
3.5
34
112
6.04
0.30
0.00
933
112
8.0 6.0
664
0.0
Figure 16: Fine-tuning CLIP ViT-B/32 on ImageNet-Ik. We show for 72 models from Wortsman et al. (2022) the generalization gap ImageNet vs multiple sharpness measures.
45

---

Published as a conference paper at ICLR 2026
K.4 CORRELATION COEFFICIENT AND RÉNYI ORDER
In this section, we report statistics of Kendall's 7 under different Rényi orders. The order a is varied following the guidelines in Section 4.1 We compute Kendall's for each layer and report the average correlation of all layers. The heatmap in Fig. shows that a 0.5 for 0 < a < 1 and 1.5 for are consistently robust across tasks.
0.6
0.6
T10 highherr
0.3
0.4
0.3
8
0.4
0.2
0.2
0.2
-.2
0.1
0.1
0.0
0.0
high
high
(a) CIFAR1O ResNet18
(b) CIFAR1O ResNet34
1.0
1.0
0.8
0.8
0.6
0.6
0.5
0.6
0.5
11..0.5
0.5
low
0.4
r10 Nithee
0.3
0.4
0.3
100
6.2
0.2
0.2
0.2
6.1
6.1
0.0
0.0
high
high
(c) CIFARIO Vision Transformer
(d) CIFARI00 ResNec18
0.8
1.0
0.8
1.0
6.7
0.7
0.8
0.8
0.6
0.5
6.5
0.6
6.5
0.5
700 higherrs
higheris
0.4
0.4
6.3
0.4
0.3
0,4
0.2
0.
0.2
0.2
6.1
0.1
0.0
0.0
1.6
2.0
2.2
2.6
1.6
2.2
2.6
2.0
high
high
(c) CIFAR 100 ResNet34
(f) TiyyImagNeet ResNet18
Figure 17: Correlation Coefficient and Rényi Order a
46

---

Published as a conference paper at ICLR 2026
K.5 SCATTER PLOT OF CORRELATION COEFFICIENT AND RÉNYI ORDER
In this section, we provide all the correlation coefficient T under different a across multiple tasks:
609
0
100
Figure 18: R on CIFARIO, 10, we plot the correlation coefficient T VS Rényi order a.
0.50
1091a
1ps12
Uree b
58
435
082
350
6:88
10
113
131
63
Xtt
Eds
1:4
Figure 19: ResNet34 on CIFAR10, we plot the correlation coefficient T vs Rényi order
413
-14
km/hh
20
kmihk
15
15
Figure 20: ViT on CIFAR1O, we plot the correlation coefficient 7 vs Rényi order a.
47

---

Published as a conference paper at ICLR 2026
CIE
0.0
CTR
X75
lorla
300
10e
615
13: 130 6.
22: 132 10
03) AJ 11
0:55
Figure 21: R8 on CIFARIO0, we plot the correlation coefficient vs Rényi order a.
Mork
$15
125
5.0
674
Se
Figure 22: ResNet34 on CIFARIO0, we plot the correlation coefficient vs Rényi order
63B
AN
6114
4:22
477
Figure 23: R8 on TinylmageNet, we plot the correlation coefficient T vs Rényi order a.
48

---

Published as a conference paper at ICLR 2026
L
LIMITATION
The generalization bounds in our work relies on homogeneity of the activation function, which holds for ReLU networks and approximately holds for GELU networks. Extending the analysis for other activations a both interesting and important direction.
Our proposed RSAM algorithm uses an approximation to Rényi sharpness for simplicity, a tighter approximation surrogate may further improve generalization.
M BROADER IMPACTS
Our work aims to advance the theoretical understanding of network generalization, with the antici- pation that theoretical insights can guide future designs of network optimization methods. There are ethically related issues negative societal consequences in our work.
49


Reviews:
[{"review_id": "FuYURZ4eeg", "reviewer": "Reviewer_2BS3", "summary": "The paper introduces a novel measure using the equation for Renyi entropy. The normalized eigenvalues of the loss Hessian are treated as a distribution, and the Renyi entropy of this distribution is used as a measure. A generalization bound is derived in terms of this entropy, and the experimental results using Renyi entropy regularizer showed small improvements on accuracy on benchmark data.", "questions": "Please address the weaknesses to increase score.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The presented method is novel. Experiments show improvement in accuracy.", "weaknesses": "The presented method is novel, but it is difficult to recommend the paper for acceptance for several reasons. \n\nThe most problematic aspect is that the entropy is defined for a probability distribution, whereas eigenvalue spectrum is not a probability distribution. It is unclear what probability model the authors assume and whether it is valid. In the derived bound, the authors use a posterior distribution Q, but its definition and justification are not clearly explained.  It is very hard to recognize the eigenvalue spectrum as a distribution because no corresponding random variable is defined. Does the largest eigenvalue represent \\the probability of the first random variable? The overall setup and explanation are too vague to assess whether the application of Renyi entropy is conceptually meaningful. For example, I have seen using the information-theoretic measures for the output of the network because the output can be naturally considered as a probability distribution over predicted classes. However, in the case of Hessian eigenvalues, such a probabilistic interpretation is not justifiable.\n\nThe explanation in lines 176-184 is unclear What are the functions h(.), g(.) and other perturbation constants A and rho? \n\nHessian should be a very large matrix, making its computation and eigenvalue calculation at each optimization step computationally expensive. The authors used the square of gradient to approximate the hessian eigenvalues in the experiment without sufficient explanation. Is the eigenvalue distribution related to the square of gradient components in each dimension? During optimization, the gradient should approach zero, while the hessian does not.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YJ2ZjpGmNF", "reviewer": "Reviewer_3Nf3", "summary": "In this paper, the authors study a new sharpness measure motivated by Rényi divergences in statistics. \nThe authors argue that uniformity in eigenvalues of Hessian (for a constant sum of eigenvalues) is desirable to promote generalization, as opposed to the original SAM, which aims to only minimize the largest eigenvalue of Hessian, being independent of the uniformity. They show that Rényi entropy, when applied to the normalized eigenvalues of the Hessian, can characterize the spread of the spectrum. Moreover, they prove generalization bounds from Rényi sharpness using two ideas: (1) Rényi sharpness, according to its definition, is a normalized function of Hessian eigenvalues, thus it is invariant to rescaling of parameters in homogeneous neural networks such as ReLU nets. (2) Multiplicative weight perturbations, according to orthogonal transformations. Furthermore, they experimentally show how generalization correlates with Rényi sharpness (Section 5) and propose Rényi-SAM, achieved via a new loss function. The method is validated over various datasets. \n \n\n \n\n\n\n\nHere is a detailed summary of their main contributions: Rényi sharpness is introduced, and it is shown that it is invariant to rescaling. Its correlation to generalization is shown. In Proposition 3.1, they show that perturbation via orthogonal matrices allows us to upper bound the population loss (Equation 3). This leads to a PAC-bayesian generalization bound in Theorem 3.2 and Theorem 3.3 that relates the population loss to the empirical risk plus some term involving Rényi sharpness.\n\n\nThey use PyHessian to estimate the Hessian of the neural network, which leads to Section 4.1. This includes a comprehensive discussion of how to choose the parameter alpha involving Rényi sharpness. \n\nTo estimate Rényi sharpness, they propose a method based on writing the quantity in terms of the trace of powers of the Hessian, and they use the Hutchinson method along with other quadrature methods used to approximate integrals/expectations. It is given in Algorithm 1.", "questions": "This is an interesting paper on SAM, relating Rényi sharpness to generalization. I think the paper is making good contributions, spanning from theory to algorithms and insights about generalization. Here are some comments/questions:\n\n\nDefinition 2.1: What happens if $H$ is non-positive definite? How do you define Rényi sharpness then? Please clarify this in the paper\n\n\nReparametrization invariance in Proposition 2.2: Why in Equation 1, alpha has to be different from one? Couldn't we take the limit and conclude that the identity also holds for that case?\n\n\n\nProposition 2.2: At first glance, I thought the invariance holds for the Hessian of the whole network, but looking at the proof, it looks like it holds only for an arbitrary fixed later. I ask the authors to clarify this in the text. The invariance follows from the fact that in the definition of Rényi sharpness, the authors introduced a normalization factor (the trace). \n\n\n\nPlease include some explanations about Algorithm 1 in the next version of the paper. Currently, it is really difficult to follow what it means.  \n\n\n\nEq. 9: If the gradients are vectors, then what do you mean by taking squares? You mean $hh^T$?\n\n\nHow do you optimize the objective in Eq. 11? Do you compute the gradients of it? Do you have something like a 'base optimizer' similar to the original SAM? If you compute the gradient of Eq. 11, then you have second-order derivative terms (prohibited). Do you use the same approximations as the original SAM to resolve this? I believe a rigorous explanation of what the Rényi SAM algorithm really is is required for the next version of the paper. Since this is one of the main contributions of this paper, you need to explain this in detail, probably having a clear algorithm dedicated to it. \n\n\nThere are some other notions of sharpness similar to Rényi in recent works that have never been discussed in the paper: For instance, I found 'Tilted SAM' in [1] and 'Frobenius SAM' in [2] on the web. There might also exist others, but at least these two are pretty similar. At least one expects some discussion and comparison in the paper. Please do search for more because I believe there might be other papers. \n\n\n[1] Li, Tian, Tianyi Zhou, and Jeff Bilmes. \"Tilted sharpness-aware minimization.\" ICML 2025\n\n[2] Tahmasebi, Behrooz, Ashkan Soleymani, Dara Bahri, Stefanie Jegelka, and Patrick Jaillet. \"A universal class of sharpness-aware minimization algorithms.\" ICML 2024", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- A new notion of sharpness, which is theoretically and practically correlated to generalization, is of potential interest to the community\n\n- The paper is an interesting mix of theory and practice", "weaknesses": "- The algorithm (Rényi sharpness) is poorly explained in the paper", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4bZhkMZ3LF", "reviewer": "Reviewer_VHgD", "summary": "The authors propose a novel sharpness measure, Renyi sharpness, to consider the spread of the Hessian eigenvalues.\nIt shows a better correlation with generalization. They also propose a regularization method, RSAM, which outperforms other SAM variants.", "questions": "See weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- strong empirical results\n    - strong performance of RSAM\n    - strong  correlation between Renyi sharpness and generalization\n- scaling invariance of Renyi sharpness (see weaknesses)", "weaknesses": "-  **Lack of motivation**. Why do we need to consider the **spread** of Hessian eigenvalues? \n    - The authors said \"uniform eigenvalue is the most desirable to ensure good generalization, since if there exists no particularly large eigen direction, small perturbation of data would just incur small loss change\"\n    - This seems like a wrong statement.\n    - If all eigenvalues are large but similar (small spread), e.g., isotropic, then there surely exists large eigen direction, but Renyi sharpness is small as it only consider the spread, not the magnitude.\n    - $H_\\alpha(cI)=1/(1-\\alpha)\\log \\sum (1/n)^\\alpha=\\log n$ does not depends on the value of $c>0$.\n\n- The concept in Prop 2.2 is **not \"reparameterization\" invariance, but \"scaling\" invariance**. Renyi sharpness may not be (nonlinear) reparameterization invariant.\n\n- It seems like $A$ in the bound in (4) is not constant and depends on $\\theta$. \n\n- There is no definition for capital $N$.\n\n- Not a fair comparison. \"We vary $\\alpha$ and plot the sharpness that attains the highest correlation coefficient\". At least, you should report the best $\\alpha$ for each layer. It would be better to draw a scatter plot for $(\\alpha, \\tau)$-pairs (compared to the other baselines, e.g., trace) for each layer. If you pick the best measure (or $\\alpha$) after observing the gap, it is not an useful generalization measure.\n\n- use $\\langle w_j', v_j\\rangle$ or $w_j^{'\\top}v_j$ to improve readability.\n\n- It would be better to write $|g_j|^{2\\alpha}$ or $(g_j^2)^\\alpha$ instead of $g_j^{2\\alpha}$ in (10) to avoid a confusion for the case of $\\alpha=0.5$.\n\n- Can you elaborate more on the meaning of the **layerwise** Renyi shaprness? Is it important because Prop 3.1 can be applied to a single layer?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qHPWFWK6p0", "reviewer": "Reviewer_NtAE", "summary": "The authors propose a novel sharpness measure to predict generalization performance: Renyi sharpness. They provide generalization bounds based on this measure, and experimental studies on CIFAR and TinyImagenet. Further, they also introduce a variant of the sharpness-aware minimization (SAM) algorithm that is based on an approximation of the Renyi sharpness, with experiments on Cifar and TinyImageNet.", "questions": "I do not have specific questions where an answer would change my opinion on the paper - I think significant changes, addressing the weaknesses outlined above, are necessary to convincingly argue in favour of Renyi sharpness:\n- a much extended experimental setup, like in Andriushchenko et al\n- more and better tuned baselines \n- a practical way of choosing alpha, without the necessity of tuning it\n- convincing arguments why the generalization bounds argue in favour of Renyi sharpness instead of log-det (H)\n- convincing evidence that RSAM improves robustly for fair comparison (tuned rho) over baselines", "limitations": "", "rating": 2, "confidence": 5, "soundness": 1, "presentation": 3, "contribution": 1, "strengths": "The paper aims to address a fundamental problem in deep learning: the poor correlation between many generalization measures, and the empirically observed generalization performance. Using the Renyi sharpness as generalization measure because it captures uniformity is novel (to the best of my knowledge). In the provided experiments, the Renyi sharpnes shows good correlation (albeit doubts remain, see weaknesses below).", "weaknesses": "I have strong doubts on the conclusions that can be drawn from the experiments provided by the authors. In particular, I have concerns regarding the considered sharpness setup, the baselines that were used, the insightsfulness of the generalization bounds, and the effectiveness of the provided RSAM algorithm. \n\n1. Sharpness setup:  \nThe present study on the connection between Renyi sharpness and generalization is much less extensive than other work investigating the relationship between sharpness measures and generalization (e.g. Andriushchenko et al [1]). In [1], the authors also found setups - similar to the ones considered in this work - where sharpness _could_ predict generalization. However, Andriushchenko et al [1] showed that this might only be true for certain subgroups of the training parameters. In particular, when considering their “modern” setup (ViTs, ImageNet-scale, varied pretraining+finetuning schemes, OOD generalization and transfer learning, Language and Vision tasks,  …), the correlation disappeared. To show the effectiveness of the Renyi sharpness, experimental evidence on the scale of [1] would be necessary. \n\n\n\n2. Sharpness baselines and tuning of alpha:  \nThe provided results are for extensively tuned alpha values, whereas the baselines are apparently not tuned (e.g. the $\\rho$ values for SAM and ASAM). Further, many sharpness variants (e.g. the ones from [1]) are omitted from the study, and there are no details on how exactly the baseline measures are computed or what they mean.\n\n\n\n3. Generalization bounds:  \nThe generalization bound in theorem 3.2 is based on upper-bounding a generalization bound from [2] by upper-bounding the log-determinant of the Hessian with the Renyi sharpness. Using the log-determinant of the hessian (like done in [2]) would thus be better motivated by the bound. Same for theorem 3.3, where the log-determinant is also upper-bounded by the Renyi sharpness. There is thus a disconnect between the generalization bounds and the measure used. \n\n\n4. RSAM:  \nThe provided RSAM algorithm seems to be brittle (as admitted by the authors: warmup required, length depends on task) and barely brings improvements beyond error bars over ASAM. Further, from Tables 4 and 5 it seems that $\\rho$ has been tuned for RSAM, but not for the other SAM variants. Finally, there exists a plethora of SAM variants, that are all ignored in the comparison, and the study is limited in terms of dataset size (no ImageNet scale) and models (no ViTs). \n\n\n5. More comments:\n\n- reparametrization-invariant sharpness measures (w.r.t. layerwise rescaling) have been investigated before (see e.g. [1]), and I assume that the ASAM measure used in the provided study is one variant of the measures in [1]. So reparametrization invariance alone cannot be the reason for the (potential) success of a novel sharpness measure. I recommend discussing this when arguing about reparametrization invariance. \n- the authors claim “in our opinion, what matters the most for characterizing the generalization\nis the extent of the spread of the spectrum” when introducing the Renyi sharpness as generalization measure, but it is unclear to me where this opinion stems from. \n- The authors describe how alpha might be chosen, depending on the spectrum, but do not elaborate how this choice might look like in practice, and then tune alpha extensively in their experimental section. Is there an intuition on when to chose low or high alpha? Is it necessary to manually inspect the spectrum and decide on multi-cluster vs uniform? \n\n\n\nTypos:\n\nseveral times (e.g. Line 147): denote vs donate\n\nLine 215 Ler vs Let\n\n\n[1]  Maksym Andriushchenko, Francesco Croce, Maximilian Müller, Matthias Hein, and Nicolas Flammarion. A modern look at the relationship between sharpness and generalization. \n\n[2] Zhiwei Jia and Hao Su, Information-Theoretic Local Minima Characterization and Regularization, ICML 2020", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)